Use sep=r'\s*,\s*' to parse a file where the columns may have some number of spaces preceding or following the delimiter (e.g. , ):
transactions = pd.read_csv('transactions.csv', sep=r'\s*,\s*',
header=0, encoding='ascii', engine='python')
Prove:
print(transactions.columns)
Output:
Index(['product_id', 'customer_id', 'store_id', 'promotion_id', 'month_of_year', 'quarter', 'the_year', 'store_sales', 'store_cost', 'unit_sales', 'fact_count'], dtype='object')
Alternatively, remove unquoted spaces in the CSV file, and use your command (unchanged).
Answer from MaxU - stand with Ukraine on Stack OverflowUse sep=r'\s*,\s*' to parse a file where the columns may have some number of spaces preceding or following the delimiter (e.g. , ):
transactions = pd.read_csv('transactions.csv', sep=r'\s*,\s*',
header=0, encoding='ascii', engine='python')
Prove:
print(transactions.columns)
Output:
Index(['product_id', 'customer_id', 'store_id', 'promotion_id', 'month_of_year', 'quarter', 'the_year', 'store_sales', 'store_cost', 'unit_sales', 'fact_count'], dtype='object')
Alternatively, remove unquoted spaces in the CSV file, and use your command (unchanged).
There are extra initial white spaces in your CSV file, e.g. , customer_id, store_id, promotion_id, ...
To prove this, try print(list(df.columns)) and look at the names of the columns:
>>> print(list(df.columns))
['product_id', ' customer_id', ' store_id', ' promotion_id', ...]
Solution
The direct way to solve this is to add the parameter skipinitialspace in pd.read_csv(), for example:
pd.read_csv('transactions.csv',
sep=',',
skipinitialspace=True)
Reference: TheGrimmScientist's answer on "How can I remove extra whitespace from strings when parsing a csv file in Pandas?"
python 3.x - Pandas raise KeyError(key) from err KeyError:, even though the key exists - Stack Overflow
raise KeyError(key) from err from pandas(python) - Stack Overflow
KeyError raised when using pandas DataFrame in SelectFromModel.fit()
BUG: `KeyError` when assigning to Series values after pop from DataFrame
The title explains the full problem I have been facing. I am just going to leave the full code and the full error in hopes someone much smarter and experienced than myself can tell me where my f*ck up is
Thank you in advance!
import pandas as pd
import numpy as np
FilterThese = ['there is a list of about 500 emails', 'that goes here', 'but removed for privacy reasons']
df = pd.read_csv("full path to csv file here", sep=',')
## IF NOT IN LIST, CHANGE EMAIL TO NULL
datafilters = df.loc[~df['work_email'].isin(FilterThese), 'work_email'] = ''
print(df[datafilters])the full error I am getting
PS C:\Users\myusername> & C:/Users/myusername/AppData/Local/Programs/Python/Python310/python.exe z:/Data/awesomedata/pandas-filter.py
Traceback (most recent call last):
File "C:\Users\myusername\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\core\indexes\base.py", line 3621, in get_loc
return self._engine.get_loc(casted_key)
File "pandas\_libs\index.pyx", line 136, in pandas._libs.index.IndexEngine.get_loc
File "pandas\_libs\index.pyx", line 163, in pandas._libs.index.IndexEngine.get_loc
File "pandas\_libs\hashtable_class_helper.pxi", line 5198, in pandas._libs.hashtable.PyObjectHashTable.get_item
File "pandas\_libs\hashtable_class_helper.pxi", line 5206, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: ''
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "z:\Data\awesomedata\pandas-filter.py", line 26, in <module>
print(df[datafilters])
File "C:\Users\myusername\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\core\frame.py", line 3505, in __getitem__
indexer = self.columns.get_loc(key)
File "C:\Users\myusername\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\core\indexes\base.py", line 3623, in get_loc
raise KeyError(key) from err
KeyError: ''I have created an aggregate pandas data frame using groupby() and count() in AWS Sagemaker Studio Lab.
aggregated_result_set = sets_df[['year','name']].groupby(by=['year']).count()
The aggregated_result_set is successfully created; if I'm using tail(), I can see data in it.
Sample data:
| name | |
|---|---|
| year | |
| 2017 | 786 |
| 2018 | 816 |
| 2019 | 840 |
| 2020 | 674 |
| 2021 | 3 |
When I'm trying to filter the data of aggregated_result_set, on the basis of the year column, I'm getting an error.
Code for filtering:
aggregated_result_set[aggregated_result_set['year'].isin([1955, 2019])]
Error Message:
KeyError Traceback (most recent call last) File ~/.conda/envs/dev_env/lib/python3.10/site-packages/pandas/core/indexes/base.py:3803, in Index.get_loc(self, key, method, tolerance) 3802 try: -> 3803 return self._engine.get_loc(casted_key) 3804 except KeyError as err: File ~/.conda/envs/dev_env/lib/python3.10/site-packages/pandas/_libs/index.pyx:138, in pandas._libs.index.IndexEngine.get_loc() File ~/.conda/envs/dev_env/lib/python3.10/site-packages/pandas/_libs/index.pyx:165, in pandas._libs.index.IndexEngine.get_loc() File pandas/_libs/hashtable_class_helper.pxi:5745, in pandas._libs.hashtable.PyObjectHashTable.get_item() File pandas/_libs/hashtable_class_helper.pxi:5753, in pandas._libs.hashtable.PyObjectHashTable.get_item() KeyError: 'year' The above exception was the direct cause of the following exception: KeyError Traceback (most recent call last) Cell In [86], line 1 ----> 1 aggregated_result_set[aggregated_result_set['year'].isin([1955, 2019])] File ~/.conda/envs/dev_env/lib/python3.10/site-packages/pandas/core/frame.py:3805, in DataFrame.__getitem__(self, key) 3803 if self.columns.nlevels > 1: 3804 return self._getitem_multilevel(key) -> 3805 indexer = self.columns.get_loc(key) 3806 if is_integer(indexer): 3807 indexer = [indexer] File ~/.conda/envs/dev_env/lib/python3.10/site-packages/pandas/core/indexes/base.py:3805, in Index.get_loc(self, key, method, tolerance) 3803 return self._engine.get_loc(casted_key) 3804 except KeyError as err: -> 3805 raise KeyError(key) from err 3806 except TypeError: 3807 # If we have a listlike key, _check_indexing_error will raise 3808 # InvalidIndexError. Otherwise we fall through and re-raise 3809 # the TypeError. 3810 self._check_indexing_error(key) KeyError: 'year'
Can anyone please let me know if the year column is in the aggregated_result_set, why I'm getting KeyError, and how I can resolve it?