Category: Python
-
PyArrow dtypes in Pandas aren’t always faster than NumPy
Using PyArrow dtypes in Python Pandas isn’t always faster: %timeit s_pyarr.mean() # 21.5ms%timeit s_np.mean() # 47.3ms %timeit s_pyarr.nlargest(10) # 667ms%timeit s_np.nlargest(10) # 663ms
-
PyArrow strings in pandas: Faster and smaller than object dtype
PyArrow strings in Python Pandas are smaller than Python strings. But they’re also far faster: %timeit s_pyarr.str.len() # 106 µs%timeit s_py.str.len() # 1.6 ms In many examples, PyArrow was far faster.
-
PyArrow dtypes in pandas: Reading CSVs with dtype_backend
Want PyArrow dtypes in your Python Pandas data frame? The dtypes are double[pyarrow], int64[pyarrow], and string[pyarrow], not the normal NumPy ones. Note: This is still experimental… but it’s also the future.
-
Pandas 3 string dtype: PyArrow strings cut memory use
Using Python Pandas 3? Strings use PyArrow, not Pandas 2’s Python strings (dtype “object”):
-
PyArrow dtypes in Pandas 3: Missing values without float conversion
In Python Pandas 3, you can use PyArrow dtypes — which are nullable: What is s? 0 101 202 <NA> # pd.NA, not np.nan3 40dtype: int64[pyarrow] # see? Not float!
-
Pandas missing values: Why assigning pd.NA gives you float64
Assign either np.nan or pd.NA to a Python Pandas series with a NumPy dtype, and it’ll be np.nan, a float. Which means the entire series has a dtype of float: Result: 0 10.01 20.02 NaN3 40.0dtype: float64
-
Pandas int8 overflow: Why small dtypes wrap around silently
If your dtype is too small, operations on your Python Pandas series will fail: Returns: 0 1101 1202 -126 # 🤯dtype: int8 This does give an error: s + 500OverflowError: Python integer 500 out of bounds for int8
-
Pandas read_csv dtype: Setting column types to save memory
Reading a CSV into a Python Pandas data frame? Use the “dtype” keyword arg and a dict to specify dtypes, and avoid the int64/float64/str defaults:
-
Changing a Pandas column dtype: Use astype, not assignment
Want to change the dtype of a Python Pandas series? You can’t — at least, not by assigning: Instead, use “astype” to get a new series, and replace the old one:
-
Pandas dtypes: Checking column types with dtype and value_counts
Get the dtype of a Python Pandas series with dtype: Get the dtypes of all columns in a data frame with “dtypes”, which returns a series: df.dtypes How many of each dtype? Use value_counts: