You can use .replace. For example:
>>> df = pd.DataFrame({'col2': {0: 'a', 1: 2, 2: np.nan}, 'col1': {0: 'w', 1: 1, 2: 2}})
>>> di = {1: "A", 2: "B"}
>>> df
col1 col2
0 w a
1 1 2
2 2 NaN
>>> df.replace({"col1": di})
col1 col2
0 w a
1 A 2
2 B NaN
or directly on the Series, i.e. df["col1"].replace(di, inplace=True).
You can use .replace. For example:
>>> df = pd.DataFrame({'col2': {0: 'a', 1: 2, 2: np.nan}, 'col1': {0: 'w', 1: 1, 2: 2}})
>>> di = {1: "A", 2: "B"}
>>> df
col1 col2
0 w a
1 1 2
2 2 NaN
>>> df.replace({"col1": di})
col1 col2
0 w a
1 A 2
2 B NaN
or directly on the Series, i.e. df["col1"].replace(di, inplace=True).
map can be much faster than replace
If your dictionary has more than a couple of keys, using map can be much faster than replace. There are two versions of this approach, depending on whether your dictionary exhaustively maps all possible values (and also whether you want non-matches to keep their values or be converted to NaNs):
Exhaustive Mapping
In this case, the form is very simple:
df['col1'].map(di) # note: if the dictionary does not exhaustively map all
# entries then non-matched entries are changed to NaNs
Although map most commonly takes a function as its argument, it can alternatively take a dictionary or series: Documentation for Pandas.series.map
Non-Exhaustive Mapping
If you have a non-exhaustive mapping and wish to retain the existing variables for non-matches, you can add fillna:
df['col1'].map(di).fillna(df['col1'])
as in @jpp's answer here: Replace values in a pandas series via dictionary efficiently
Benchmarks
Using the following data with pandas version 0.23.1:
di = {1: "A", 2: "B", 3: "C", 4: "D", 5: "E", 6: "F", 7: "G", 8: "H" }
df = pd.DataFrame({ 'col1': np.random.choice( range(1,9), 100000 ) })
and testing with %timeit, it appears that map is approximately 10x faster than replace.
Note that your speedup with map will vary with your data. The largest speedup appears to be with large dictionaries and exhaustive replaces. See @jpp answer (linked above) for more extensive benchmarks and discussion.
How do I map the keys of a dictionary to the columns of a pandas dataframe?
Keep this paradigm in your head: dataframes are meant to be instantiated from a data source, they aren't meant to be created as a blank sheet and 'filled in' later on. So also in this case: don't build a dataframe upfront, instead create the dataframe using the dict as the basis for its data. From Create a Pandas DataFrame from List of Dicts:
cols = ['timestamp', 'name', 'adress', 'phone', 'website', 'rating', 'number_of_ratings', 'type'] df = pd.DataFrame(multidict, columns=cols)More on reddit.com
python - How to map column of lists with values in a dictionary using pandas - Stack Overflow
How to remap values in Pandas column using a dictionary, while also preserving any NaN values that may be present? - Python - Data Science Dojo Discussions
python - How do you map a dictionary to an existing pandas dataframe column? - Stack Overflow
Hi y'all,
i have a list of dictionaries (multidict), which holds a lot of different values. I also have an empty dataframe with set columns which do not coincide with keys of the dictionaries.
I need to append each dictionary to the DataFrame and map each key of the dictionary to the right (and fixed) column. How do i pass the values for each dictionary to the right column and row?
Here is my code so far with exemplary data:
multidict = [{'website': 'http://www.example.de/', 'time': '2020-01-05 17:33:53.973205', 'norating': 112, 'name': 'Name', 'rating': 4.4, 'phone': '123456789', 'adress': 'adress', 'typus': "['restaurant', 'bowling_alley', 'lodging', 'food', 'point_of_interest', 'establishment']"},{'website': 'http://www.exmpl.de/', 'time': '2020-02-05 17:33:53.973205', 'norating': 12, 'name': 'Name2', 'rating': 4.3, 'phone': '987654321', 'adress': 'address', 'typus': "['whatever', 'point_of_interest', 'establishment']"}]
df = pd.DataFrame()
df.columns = ['timestamp', 'name', 'adress', 'phone', 'website', 'rating', 'number_of_ratings', 'type']
for i in multidict:
#How do I remap the keys of the dictionary here to the columns of the dataframe???Keep this paradigm in your head: dataframes are meant to be instantiated from a data source, they aren't meant to be created as a blank sheet and 'filled in' later on. So also in this case: don't build a dataframe upfront, instead create the dataframe using the dict as the basis for its data. From Create a Pandas DataFrame from List of Dicts:
cols = ['timestamp', 'name', 'adress', 'phone', 'website', 'rating', 'number_of_ratings', 'type']
df = pd.DataFrame(multidict, columns=cols)
OP, the first dictionary will determine the naming and order of all columns, unless subsequent dictionaries introduce new keys.
All you have to do is pass your list of dictionaries to pandas and it will create the columns based on the dictionary keys automatically.
md = [
{'c':3,'a':1,'b':2},
{'d':44,'a':11,'b':22},
{'b':222}]
df=pd.DataFrame(md)
print(df.head())
c a b d
0 3.0 1.0 2 NaN
1 NaN 11.0 22 44.0
2 NaN NaN 222 NaN
dict = {'c':'B','z':'4'}
#mask those that are not NaN in `target_col`
m=df.target_col.isna()
df.loc[m,'target_col']=df.key_col.map(dict)

an alternative : changed name from dict to dicts to avoid confusion with the built-in type
df.set_index('key_col').T.fillna(dicts).T
target_col
key_col
w a
c B
z 4
You can use join by Series with MultiIndex:
idx = pd.MultiIndex.from_product([[0,1],[0,1]], names=('q1','q2'))
s = pd.Series(['a','b','c','d'], index=idx, name='val')
print (s)
q1 q2
0 0 a
1 b
1 0 c
1 d
Name: val, dtype: object
df = df.join(s, on=['q1','q2'])
print (df)
q1 q2 val
0 0 1 b
1 0 1 b
2 0 1 b
3 0 1 b
4 0 1 b
5 0 1 b
6 0 1 b
7 0 1 b
8 0 1 b
9 0 1 b
10 1 1 d
11 1 1 d
12 0 1 b
13 0 1 b
14 1 0 c
15 0 0 a
16 0 0 a
17 0 0 a
18 0 0 a
19 0 0 a
20 0 0 a
21 0 0 a
Another method with df.map and df.transform:
In [90]: mapping = {(0, 0) :'a', (0, 1) : 'b', (1, 0): 'c', (1, 1): 'd'}
In [91]: df['val'] = df.transform(lambda x: (x['q1'], x['q2']), axis=1).map(mapping); df
Out[91]:
q1 q2 val
0 0 1 b
1 0 1 b
2 0 1 b
3 0 1 b
4 0 1 b
5 0 1 b
6 0 1 b
7 0 1 b
8 0 1 b
9 0 1 b
10 1 1 d
11 1 1 d
12 0 1 b
13 0 1 b
14 1 0 c
15 0 0 a
16 0 0 a
17 0 0 a
18 0 0 a
19 0 0 a
20 0 0 a
21 0 0 a
You can also use zip to generate columns, apply pd.Series and then do the mapping:
In [119]: df['val'] = pd.Series(list(zip(df.q1, df.q2))).map(mapping); df
Out[119]:
q1 q2 val
0 0 1 b
1 0 1 b
2 0 1 b
3 0 1 b
4 0 1 b
5 0 1 b
6 0 1 b
7 0 1 b
8 0 1 b
9 0 1 b
10 1 1 d
11 1 1 d
12 0 1 b
13 0 1 b
14 1 0 c
15 0 0 a
16 0 0 a
17 0 0 a
18 0 0 a
19 0 0 a
20 0 0 a
21 0 0 a
Performance
jezrael's solution:
In [552]: %%timeit
...: idx = pd.MultiIndex.from_product([[0,1],[0,1]], names=('q1','q2'))
...: s = pd.Series(['a','b','c','d'], index=idx, name='val')
...: df.join(s, on=['q1','q2'])
...:
100 loops, best of 3: 2.84 ms per loop
Proposed in this post:
In [553]: %%timeit
...: mapping = {(0, 0) :'a', (0, 1) : 'b', (1, 0): 'c', (1, 1): 'd'}
...: df.transform(lambda x: (x['q1'], x['q2']), axis=1).map(mapping)
...:
1000 loops, best of 3: 1.7 ms per loop
You can split the issue column first, transform only the first part with the mapping and then add the remaining part back:
splits = df['issue'].str.split('_')
short_issue = splits.str[0].map(issue_short_form_map).fillna(splits.str[0])
df['issue'] = short_issue + '_' + splits.str[1]
df
# date source destination issue
#0 2021-02-06 18:48:18.097962 London New York UP_LOW
#1 2021-02-06 18:48:18.097962 Berlin Tokyo DN_HIGH
#2 2021-02-06 18:47:08.209495 Paris Toronto DROP_LOW
list_to_change = [i for i in issue_short_form_map.keys()]
def check_replace(x):
"""
Check elements in x and compare to change
"""
for element_to_check in list_to_change:
if x.__contains__(element_to_check):
return x.replace(element_to_check,issue_short_form_map[element_to_check])
return x
df["issue"]= df["issue"].map(check_replace)
print(df)
date source destination issue
0 2021-02-06 18:48:18.097962 London New York UP_LOW
1 2021-02-06 18:48:18.097962 Berlin Tokyo DN_HIGH
2 2021-02-06 18:47:08.209495 Paris Toronto DROP_LOW