You can use applymap:
df = pd.DataFrame({'nearby_subway_station':['yes','no'], 'Station':['no','yes']})
print (df)
Station nearby_subway_station
0 no yes
1 yes no
dict_map_yn_bool={'yes':True, 'no':False}
df = df.applymap(dict_map_yn_bool.get)
print (df)
Station nearby_subway_station
0 False True
1 True False
Another solution:
for x in df:
df[x] = df[x].map(dict_map_yn_bool)
print (df)
Station nearby_subway_station
0 False True
1 True False
Thanks Jon Clements for very nice idea - using replace:
df = df.replace({'yes': True, 'no': False})
print (df)
Station nearby_subway_station
0 False True
1 True False
Some differences if data are no in dict:
df = pd.DataFrame({'nearby_subway_station':['yes','no','a'], 'Station':['no','yes','no']})
print (df)
Station nearby_subway_station
0 no yes
1 yes no
2 no a
applymap create None for boolean, strings, for numeric NaN.
df = df.applymap(dict_map_yn_bool.get)
print (df)
Station nearby_subway_station
0 False True
1 True False
2 False None
map create NaN:
for x in df:
df[x] = df[x].map(dict_map_yn_bool)
print (df)
Station nearby_subway_station
0 False True
1 True False
2 False NaN
replace dont create NaN or None, but original data are untouched:
df = df.replace(dict_map_yn_bool)
print (df)
Station nearby_subway_station
0 False True
1 True False
2 False a
Answer from jezrael on Stack OverflowYou can use applymap:
df = pd.DataFrame({'nearby_subway_station':['yes','no'], 'Station':['no','yes']})
print (df)
Station nearby_subway_station
0 no yes
1 yes no
dict_map_yn_bool={'yes':True, 'no':False}
df = df.applymap(dict_map_yn_bool.get)
print (df)
Station nearby_subway_station
0 False True
1 True False
Another solution:
for x in df:
df[x] = df[x].map(dict_map_yn_bool)
print (df)
Station nearby_subway_station
0 False True
1 True False
Thanks Jon Clements for very nice idea - using replace:
df = df.replace({'yes': True, 'no': False})
print (df)
Station nearby_subway_station
0 False True
1 True False
Some differences if data are no in dict:
df = pd.DataFrame({'nearby_subway_station':['yes','no','a'], 'Station':['no','yes','no']})
print (df)
Station nearby_subway_station
0 no yes
1 yes no
2 no a
applymap create None for boolean, strings, for numeric NaN.
df = df.applymap(dict_map_yn_bool.get)
print (df)
Station nearby_subway_station
0 False True
1 True False
2 False None
map create NaN:
for x in df:
df[x] = df[x].map(dict_map_yn_bool)
print (df)
Station nearby_subway_station
0 False True
1 True False
2 False NaN
replace dont create NaN or None, but original data are untouched:
df = df.replace(dict_map_yn_bool)
print (df)
Station nearby_subway_station
0 False True
1 True False
2 False a
You could use a stack/unstack idiom
df.stack().map(dict_map_yn_bool).unstack()
Using @jezrael's setup
df = pd.DataFrame({'nearby_subway_station':['yes','no'], 'Station':['no','yes']})
dict_map_yn_bool={'yes':True, 'no':False}
Then
df.stack().map(dict_map_yn_bool).unstack()
Station nearby_subway_station
0 False True
1 True False
timing
small data

bigger data

Hi y'all,
i have a list of dictionaries (multidict), which holds a lot of different values. I also have an empty dataframe with set columns which do not coincide with keys of the dictionaries.
I need to append each dictionary to the DataFrame and map each key of the dictionary to the right (and fixed) column. How do i pass the values for each dictionary to the right column and row?
Here is my code so far with exemplary data:
multidict = [{'website': 'http://www.example.de/', 'time': '2020-01-05 17:33:53.973205', 'norating': 112, 'name': 'Name', 'rating': 4.4, 'phone': '123456789', 'adress': 'adress', 'typus': "['restaurant', 'bowling_alley', 'lodging', 'food', 'point_of_interest', 'establishment']"},{'website': 'http://www.exmpl.de/', 'time': '2020-02-05 17:33:53.973205', 'norating': 12, 'name': 'Name2', 'rating': 4.3, 'phone': '987654321', 'adress': 'address', 'typus': "['whatever', 'point_of_interest', 'establishment']"}]
df = pd.DataFrame()
df.columns = ['timestamp', 'name', 'adress', 'phone', 'website', 'rating', 'number_of_ratings', 'type']
for i in multidict:
#How do I remap the keys of the dictionary here to the columns of the dataframe???Keep this paradigm in your head: dataframes are meant to be instantiated from a data source, they aren't meant to be created as a blank sheet and 'filled in' later on. So also in this case: don't build a dataframe upfront, instead create the dataframe using the dict as the basis for its data. From Create a Pandas DataFrame from List of Dicts:
cols = ['timestamp', 'name', 'adress', 'phone', 'website', 'rating', 'number_of_ratings', 'type']
df = pd.DataFrame(multidict, columns=cols)
OP, the first dictionary will determine the naming and order of all columns, unless subsequent dictionaries introduce new keys.
All you have to do is pass your list of dictionaries to pandas and it will create the columns based on the dictionary keys automatically.
md = [
{'c':3,'a':1,'b':2},
{'d':44,'a':11,'b':22},
{'b':222}]
df=pd.DataFrame(md)
print(df.head())
c a b d
0 3.0 1.0 2 NaN
1 NaN 11.0 22 44.0
2 NaN NaN 222 NaN
python - Pandas: map column using a dictionary on multiple columns - Stack Overflow
python - How to map to multiple values in a dictionary in pandas - Stack Overflow
pandas - Mapping Python dictionary with multiple keys into dataframe with multiple columns matching keys - Stack Overflow
python - Fastest way to map a dict on a df (multiple columns) - Stack Overflow
df['Gender'] = df.Name.map(lambda x: d[x][0])
df['Age'] = df.Name.map(lambda x: d[x][1])
Take all the values of the dictionary
d = {'Jack':['Male','22'],'Alex':['Male','26'],'Jackie':['Female','28'],'Susan':['Female','30']}
value_list = list(d.values())
df = pd.DataFrame(value_list, columns =['Gender', 'Age'])
print(df)
You can create a MultiIndex from two series and then map. Data from @ALollz.
df['CountyType'] = df.set_index(['County', 'State']).index.map(dct.get)
print(df)
County State CountyType
0 A 1 One
1 A 2 None
2 B 1 None
3 B 2 Two
4 B 3 Three
If you have the following dictionary with tuples as keys and a DataFrame with columns corresponding to the tuple values
import pandas as pd
dct = {('A', 1): 'One', ('B', 2): 'Two', ('B', 3): 'Three'}
df = pd.DataFrame({'County': ['A', 'A', 'B', 'B', 'B'],
'State': [1, 2, 1, 2, 3]})
You can create a Series of the tuples from your df and then just use .map()
df['CountyType'] = pd.Series(list(zip(df.County, df.State))).map(dct)
Results in
County State CountyType
0 A 1 One
1 A 2 NaN
2 B 1 NaN
3 B 2 Two
4 B 3 Three
For improve performance in larger DataFrames is possible use Series with DataFrame.join:
df = pd.DataFrame(df)
df1 = df.join(pd.Series(d, name='new'), on=[0,1,2])
print (df1.head(10))
0 1 2 new
0 EFG DS 321 1
1 EFG DS 900 51
2 EFG DS 900 51
3 EFG Q 900 51
4 EFG DS 1000 41
5 EFG DS 1000 41
6 EFG DS 1000 41
7 ABC DS 444 421
8 EFG DS 900 51
9 EFG DS 900 51
Another idea (similar solution like in question):
res = [d[(a,b,c)] for a,b,c in zip(df[0], df[1], df[2])]
Or:
res = [d[(a,b,c)] for a,b,c in df[[0,1,2]].to_numpy()]
Convert your dataframe as MultiIndex and extract values from your dict (Series):
out = pd.Series(d).loc[pd.MultiIndex.from_frame(df)] \
.rename('values').reset_index()
Output:
>>> out
0 1 2 values
0 EFG DS 321 1
1 EFG DS 900 51
2 EFG DS 900 51
3 EFG Q 900 51
4 EFG DS 1000 41
5 EFG DS 1000 41
6 EFG DS 1000 41
7 ABC DS 444 421
8 EFG DS 900 51
9 EFG DS 900 51
10 EFG DS 321 1
11 EFG DS 900 51
12 EFG DS 1000 41
13 EFG DS 900 51
14 EFG DS 321 1
15 EFG DS 321 1
16 EFG DS 1000 41
17 EFG DS 1000 41
18 EFG DS 1000 41
19 EFG DS 1000 41
20 ABC DS 444 421
21 EFG DS 900 51
22 EFG DAS 12345 123
23 EFG DAS 12345 123
24 EFG DAS 321 31
25 EFG DS 321 1
26 EFG DS 12345 321
27 EFG Q 1000 41
28 EFG DS 900 51
29 EFG DS 321 1
Using .apply
Ex:
df['Rate'] = df.apply(lambda x: rates[x['Currency']][x['Year']], axis=1)
# OR
df['Rate'] = df.apply(lambda x: rates.get(x['Currency'], dict()).get(x['Year'], 1), axis=1)
print(df)
Output:
Item Currency Year Rate
0 1 USD 2019 1
1 2 USD 2020 2
2 3 CAD 2021 6
3 4 CAD 2019 4
4 5 GBP 2020 1
Here is one way, create a multiindex series from rates with stack, that you can reindex with the values from df to get the wanted rate per row.
df['rate'] = (
pd.DataFrame(rates)
.stack()
.reindex(pd.MultiIndex.from_frame(df[['Year','Currency']].astype(str)),
fill_value=1)
.to_numpy()
)
print(df)
Item Currency Year rate
0 1 USD 2019 1
1 2 USD 2020 2
2 3 CAD 2021 6
3 4 CAD 2019 4
4 5 GBP 2020 1
You can use .replace. For example:
>>> df = pd.DataFrame({'col2': {0: 'a', 1: 2, 2: np.nan}, 'col1': {0: 'w', 1: 1, 2: 2}})
>>> di = {1: "A", 2: "B"}
>>> df
col1 col2
0 w a
1 1 2
2 2 NaN
>>> df.replace({"col1": di})
col1 col2
0 w a
1 A 2
2 B NaN
or directly on the Series, i.e. df["col1"].replace(di, inplace=True).
map can be much faster than replace
If your dictionary has more than a couple of keys, using map can be much faster than replace. There are two versions of this approach, depending on whether your dictionary exhaustively maps all possible values (and also whether you want non-matches to keep their values or be converted to NaNs):
Exhaustive Mapping
In this case, the form is very simple:
df['col1'].map(di) # note: if the dictionary does not exhaustively map all
# entries then non-matched entries are changed to NaNs
Although map most commonly takes a function as its argument, it can alternatively take a dictionary or series: Documentation for Pandas.series.map
Non-Exhaustive Mapping
If you have a non-exhaustive mapping and wish to retain the existing variables for non-matches, you can add fillna:
df['col1'].map(di).fillna(df['col1'])
as in @jpp's answer here: Replace values in a pandas series via dictionary efficiently
Benchmarks
Using the following data with pandas version 0.23.1:
di = {1: "A", 2: "B", 3: "C", 4: "D", 5: "E", 6: "F", 7: "G", 8: "H" }
df = pd.DataFrame({ 'col1': np.random.choice( range(1,9), 100000 ) })
and testing with %timeit, it appears that map is approximately 10x faster than replace.
Note that your speedup with map will vary with your data. The largest speedup appears to be with large dictionaries and exhaustive replaces. See @jpp answer (linked above) for more extensive benchmarks and discussion.