Tag: Pandas
-
Pandas na_values: Treating custom strings as missing data
A file you’re loading into Python Pandas uses weird strings for missing values? Use the na_values parameter in read_* methods:
-
Pandas loc vs iloc slicing: Inclusive vs exclusive endpoints
When retrieving a slice with loc in Python Pandas, the end point is included, highly unusual in Python!
-
pandas read_csv: Speed up loading with the PyArrow engine
Reading a CSV file into a Python Pandas data frame? Speed things up by specifying the PyArrow engine. Data storage isn’t affected. With a 2.2GB file, it took 4s vs. 55s — more than 10x faster!
-
Pandas category dtype: Reduce string column memory by 90%
Does your Python Pandas column contain repeated strings? Turn it into a ‘category’ series (like an enum) to save memory: That’s right — 90% less memory!
-
Python str.split and rsplit: Splitting strings with maxsplit
Break a Python string into a list of strings with str.split: Limit splits with maxsplit: Or split from the right, with a limit:
-
Pandas columns overlap error: Fixing join with lsuffix and rsuffix
Joining Python Pandas DataFrames with overlapping columns? You’ll hit: ValueError: columns overlap but no suffix specified Fix with lsuffix and rsuffix: Now the columns are: x_left, x_right, y_left, y_right Problem solved!
-
Pandas join: Why you need a one-column DataFrame, not a Series
Python Pandas gotcha: You can’t join a single column! Why? join() is a data frame method. A single column is a Series. Solution: Use [[ ]] to get a one-column data frame:
-
Pandas currency strings: Strip USD, then astype(float) and sum
I used Python Pandas to total AWS S3 payments. Data was in a CSV file. But numbers in the “Amount” column were written as “USD 12.34”. Here’s what I did:
-
Pandas 3.0 pd.col: Replacing lambda in loc and assign
Python Pandas 3.0 introduces pd.col, which shortens queries: Also:
-
What’s new in Pandas 3?
🎉 Pandas 3 is out! As of last week, saying “pip install pandas” or “uv add pandas” gives you the latest version. What’s new? What has changed? I’ve got a whole YouTube playlist, explaining what you need to know: https://www.youtube.com/playlist?list=PLbFHh-ZjYFwFWHVT0qeg9Jz1TBD0TlJJT