Tag: Python
-
PyArrow dtypes in pandas: Reading CSVs with dtype_backend
Want PyArrow dtypes in your Python Pandas data frame? The dtypes are double[pyarrow], int64[pyarrow], and string[pyarrow], not the normal NumPy ones. Note: This is still experimental… but it’s also the future.
-
Pandas 3 string dtype: PyArrow strings cut memory use
Using Python Pandas 3? Strings use PyArrow, not Pandas 2’s Python strings (dtype “object”):
-
PyArrow dtypes in Pandas 3: Missing values without float conversion
In Python Pandas 3, you can use PyArrow dtypes — which are nullable: What is s? 0 101 202 <NA> # pd.NA, not np.nan3 40dtype: int64[pyarrow] # see? Not float!
-
Pandas missing values: Why assigning pd.NA gives you float64
Assign either np.nan or pd.NA to a Python Pandas series with a NumPy dtype, and it’ll be np.nan, a float. Which means the entire series has a dtype of float: Result: 0 10.01 20.02 NaN3 40.0dtype: float64
-
Pandas int8 overflow: Why small dtypes wrap around silently
If your dtype is too small, operations on your Python Pandas series will fail: Returns: 0 1101 1202 -126 # 🤯dtype: int8 This does give an error: s + 500OverflowError: Python integer 500 out of bounds for int8
-
Pandas read_csv dtype: Setting column types to save memory
Reading a CSV into a Python Pandas data frame? Use the “dtype” keyword arg and a dict to specify dtypes, and avoid the int64/float64/str defaults:
-
Changing a Pandas column dtype: Use astype, not assignment
Want to change the dtype of a Python Pandas series? You can’t — at least, not by assigning: Instead, use “astype” to get a new series, and replace the old one:
-
Pandas dtypes: Checking column types with dtype and value_counts
Get the dtype of a Python Pandas series with dtype: Get the dtypes of all columns in a data frame with “dtypes”, which returns a series: df.dtypes How many of each dtype? Use value_counts:
-
Pandas data frame preview: df.head and df.sample
Want to get a quick look at a Python Pandas data frame you just created? Option 1, use df.head(n) to look at the first n rows: Option 2, use df.sample(n) to look at n randomly selected rows:
-
Pandas read_csv reads zipped CSV files without unzipping
Want to read a zipped CSV file into a Python Pandas data frame: Just pass the filename to read_csv: The file can contain a single CSV file, with an extension of .gz, .bz2, .zip, .xz, .zst, .tar, .tar.gz, .tar.xz, or .tar.bz2.