Constructing a new dict:
dict_you_want = {key: old_dict[key] for key in your_keys}
Uses dictionary comprehension.
If you use a version which lacks them (ie Python 2.6 and earlier), make it dict((key, old_dict[key]) for ...). It's the same, though uglier.
Note that this, unlike jnnnnn's version, has stable performance (depends only on number of your_keys) for old_dicts of any size. Both in terms of speed and memory. Since this is a generator expression, it processes one item at a time, and it doesn't looks through all items of old_dict.
Removing everything in-place:
unwanted = set(old_dict) - set(your_keys)
for unwanted_key in unwanted: del your_dict[unwanted_key]
Answer from user395760 on Stack Overflowpython - Filter dict to contain only certain keys? - Stack Overflow
python filter list of dictionaries based on key value - Stack Overflow
python - Filter a dictionary from a list of keys - Stack Overflow
Python - Filter list of dictionaries based on multiple keys - Stack Overflow
Videos
Constructing a new dict:
dict_you_want = {key: old_dict[key] for key in your_keys}
Uses dictionary comprehension.
If you use a version which lacks them (ie Python 2.6 and earlier), make it dict((key, old_dict[key]) for ...). It's the same, though uglier.
Note that this, unlike jnnnnn's version, has stable performance (depends only on number of your_keys) for old_dicts of any size. Both in terms of speed and memory. Since this is a generator expression, it processes one item at a time, and it doesn't looks through all items of old_dict.
Removing everything in-place:
unwanted = set(old_dict) - set(your_keys)
for unwanted_key in unwanted: del your_dict[unwanted_key]
Slightly more elegant dict comprehension:
foodict = {k: v for k, v in mydict.items() if k.startswith('foo')}
You can try a list comp
>>> exampleSet = [{'type':'type1'},{'type':'type2'},{'type':'type2'}, {'type':'type3'}]
>>> keyValList = ['type2','type3']
>>> expectedResult = [d for d in exampleSet if d['type'] in keyValList]
>>> expectedResult
[{'type': 'type2'}, {'type': 'type2'}, {'type': 'type3'}]
Another way is by using filter
>>> list(filter(lambda d: d['type'] in keyValList, exampleSet))
[{'type': 'type2'}, {'type': 'type2'}, {'type': 'type3'}]
Trying a few answers from this post, I tested the performance of each answer.
As my initial guess, the list comprehension is way faster, the filter and list method is second and the pandas is third, by far.
defined variables:
import pandas as pd
exampleSet = [{'type': 'type' + str(number)} for number in range(0, 1_000_000)]
keyValList = ['type21', 'type950000']
1st - list comprehension
%%timeit
expectedResult = [d for d in exampleSet if d['type'] in keyValList]
60.7 ms ± 188 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
2nd - filter and list
%%timeit
expectedResult = list(filter(lambda d: d['type'] in keyValList, exampleSet))
94 ms ± 328 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
3rd - pandas
%%timeit
df = pd.DataFrame(exampleSet)
expectedResult = df[df['type'].isin(keyValList)].to_dict('records')
336 ms ± 1.84 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
On a side note, using pandas to deal with a dict is not a great idea since the pandas.DataFrame is basically a more memory consuming dict and if you are not going to use a dataframe in the end it is just inefficient.
Instead of iterating through all the keys and then checking if it is present in another list, you could simply walk through the list and pick items from the dict and create a new dict.
new_dict = dict( ((key, d[key]) for key in l) )
if items of your list not in your dictionaries you get error if you use this:
it takes about 5 seconds to complete it
new_dict = dict(((key, d[key]) for key in l))
you can use something like this:
and you can be sure you won't get the error if items of your list not in your dictionaries
keys = set(l).intersection(d)
result = {key:d[key] for key in keys}
but it takes about 17 seconds, and it doesn't good performance
you can use this and it takes about 5 seconds to complete it and you can be sure you won't get any error:
result = {}
for k in set(d1).intersection(l1):
result[k]= d1[k]
With list comprehensions:
myDict = [{'first': 'James', 'middle': 'Smith', 'last': 'Joule'},
{'first': 'James', 'middle': 'Johnson', 'last': 'Watt'},
{'first': 'Christian', 'middle': 'Edward', 'last': 'Doppler'},
{'first': 'Robert', 'last': 'Antonio'}]
keys = {"middle", "last"}
l = [{k:v for k, v in i.items() if k in keys} for i in myDict]
But you can also use map for this:
myDict = [{'first': 'James', 'middle': 'Smith', 'last': 'Joule'},
{'first': 'James', 'middle': 'Johnson', 'last': 'Watt'},
{'first': 'Christian', 'middle': 'Edward', 'last': 'Doppler'},
{'first': 'Robert', 'last': 'Antonio'}]
keys = {"middle", "last"}
l = list(map(lambda x: {k:v for k, v in x.items() if k in keys}, myDict))
print(l)
output:
[{'last': 'Joule', 'middle': 'Smith'}, {'last': 'Watt', 'middle': 'Johnson'}, {'last': 'Doppler', 'middle': 'Edward'}, {'last': 'Antonio'}]
Use neverwalkaloner's answer if you are just doing this once. But, if you find yourself manipulating lists of dictionaries often, I've written a free library called PLOD that streamlines much of this.
>>> from PLOD import PLOD
>>> l = PLOD(myDict).dropKey("middle").returnList()
>>> l
[{'last': 'Joule', 'first': 'James'}, {'last': 'Watt', 'first': 'James'}, {'last': 'Doppler', 'first': 'Christian'}, {'last': 'Antonio', 'first': 'Robert'}]
>>> print(PLOD(l).returnString())
[
{first: 'James' , last: 'Joule' },
{first: 'James' , last: 'Watt' },
{first: 'Christian', last: 'Doppler'},
{first: 'Robert' , last: 'Antonio'}
]
>>>
The library is on PyPi: https://pypi.python.org/pypi/PLOD
To more universally do what you want, I'd need to add a new class method. Perhaps .filterKeys. Perhaps I'll do that in version 1.8. It would then go:
>>> l = PLOD(myDict).filterKeys(['first', 'last']).returnList()
Hmmm...
BTW, the lib supports Python 2.7.x right now. Still working on the 3.5.x release.
Given a list of dictionaries like:
[{'first name': 'John', 'last name': 'Smith', 'age': 20, 'sport': 'basketball', 'level': 'college', 'team': 'Bruins', 'team city': 'Los Angeles'}, {'first name': 'Wayne', 'last name': 'Gretsky', 'age': 29, 'sport': 'ice hockey', 'level': 'professional', 'team': 'Kings', 'team city': 'Los Angeles'}, {'first name': 'Dan', 'last name': 'Marino', 'age': 31, 'sport': 'football', 'level': 'professional', 'team': 'Dolphins', 'team city': 'Miami'}...]Outside of using a SQL query via sqllite module, what would be the best approach of writing a script that returns a smaller list of dictionaries for a different combination of filters (ex. all players with last name of 'Smith', all players under the age of 25, all players with the last name 'Smith' who are under the age of 25, all players in professional teams in Las Vegas, all college baseball players under the age of 21 who are not from Los Angeles, etc.)?
I'm not so much asking for specific code, but what would be your thought process in structuring clean code for such a script?
So let's say I have a dictionary of
dict = { chevy_01 : "Ford", chevy_02: "Tahoe"}
I want to filter through the KEYS of the dict and get the last two characters of the keys while later performing another function. So I imagine something like this
{for keys in vals print (key[-2:]}
But I want the end of the function to return a list of the last two characters in the dict so it will just return [01, 02].
I want to do this in a lambda function I'm bad at dict comprehension. Can someone help me out?
I have a nested dictionary of source words, target words, and their frequency counts. It looks like this:
src_tgt_dict = {"each":{"chaque":3}, "in-front-of":{"devant":4}, "next-to":{"à-côté-de":5}, "for":{"pour":7}, "cauliflower":{"chou-fleur":4}, "on":{"sur":2, "panda-et":2}}I am trying to filter the dictionary so that only key-value pairs that are prepositions remain. To that end, I've written the following:
tgt_preps = set(["devant", "pour", "sur", "à"]) #set of target prepositions
src_tgt_dict = {"each":{"chaque":3}, "in-front-of":{"devant":4}, "next-to":{"à-côté-de":5}, "for":{"pour":7}, "cauliflower":{"chou-fleur":4}, "on":{"sur":2, "panda-et":2}}
new_tgt_preps = [] #list of new target prepositions
for src, d in src_tgt_dict.items(): #loop into the dictionary
for tgt, count in d.items(): #loop into the nested dictionary
check_prep = []
if "-" in tgt: #check to see if hyphen occurs in the target word (this is to capture multi-word prepositions that are not in the original preposition set)
check_prep.append(tgt[0:(tgt.index("-"))]) #if there's a hyphen, append the preceding word to the check_prep list
for t in check_prep:
if t in tgt_preps: # check to see if the token preceding the hyphen is a preposition
new_tgt_preps.append(tgt) #if yes, append the multi-word preposition to the list of new target prepositions
tgt_preps.update(new_tgt_preps) # update the set of prepositions to include the multi-word prepositions
temp_2_src_tgt_dict = {} # create new dict for filtering
for src, d in src_tgt_dict.items(): # loop into the dictionary
for tgt, count in d.items(): # loop into the nested dictionary
if tgt in tgt_preps: # if the target is in the set of target prepositions
temp_2_src_tgt_dict[tgt] = count # add to the new dict with the tgt as the key and the count as the valueWhen I print the new dict, I get the following:
{'devant': 4, 'pour': 7, 'sur': 2, 'à-côté-de': 5}
And it totally makes sense why I get that, because that's what I told the machine to do. But that's not my intention!
What I want is:
{"in-front-of:{"devant":4}, "for":{"pour":7}, "on":{"sur":2}, {"next-to":{"à-côté-de":5}}
I've tried to instantiate the nested dictionary by writing:
temp_2_src_tgt_dict[tgt][src] = count
but that throws up a Key Error.
Can anyone provide any suggestions or advice? Thank you in advance for your help.
You can use filter() function. It will take 2 arguments filter(function, iterable). Only those values will be considered which the function returns.
a=[{u'id': 5650, u'children': [{u'id': 4635}]},
{u'id': 5648, u'children': [{u'id': 67}, {u'id': 77}]},
{u'id': 5649}]
print filter(lambda x: 'children' in x, a)
Output:
[{u'id': 5650, u'children': [{u'id': 4635}]},
{u'id': 5648, u'children': [{u'id': 67}, {u'id': 77}]}]
filter(function, iterable) is equivalent to [item for item in iterable if function(item)]
How's this?
dict_list = [{u'id': 5650, u'children': [{u'id': 4635}]}, {u'id': 5648, u'children': [{u'id': 67}, {u'id': 77}]}, {u'id': 5649}]
filtered = [d for d in dict_list if 'children' in d]
print(filtered)
Output
[{'id': 5650, 'children': [{'id': 4635}]}, {'id': 5648, 'children': [{'id': 67}, {'id': 77}]}]
The way you have written new_list, it looks like you expect it to be a list comprehension, but it is only a list containing a single item, which is a dict comprehension. You probably want to move the iterator over lst_dict out of the dictionary and into the list. It should look like this:
new_list = [{k:v for k,v in i.items() if k in key_lst} for i in lst_dict]
You may use list comprehension with dict comprehension as:
>>> my_list = [{key: sub_dict.get(key) for key in key_lst} for sub_dict in lst_dict]
# using "dict.get('key')" in case key is not present in sub dicts in example
>>> my_list
[{'TABLE_NAME': 'TEST1', 'COLUMN_NAME': 'TEST_COLUMN'}, {'TABLE_NAME': 'TEST1', 'COLUMN_NAME': 'TEST_COLUMN2'}, {'TABLE_NAME': 'TEST2', 'COLUMN_NAME': 'TEST_COLUMN1'}]
where lst_dict and key_lst are holding the values mentioned in the question.
The issue with your attempt is that you are not creating separate dict for the sub dicts in your list.