I would advise against using regular expressions for this task, because the language you want to parse is not regular.
You have a character string of multiple key value pairs. The best way to parse this is not to match patterns on it, but to properly tokenize it.
There is a module in the Python standard library, called shlex, that mimics the parsing done by POSIX shells, and that provides a lexer implementation that can easily be customized to your needs.
from shlex import shlex
def parse_kv_pairs(text, item_sep=",", value_sep="="):
"""Parse key-value pairs from a shell-like text."""
# initialize a lexer, in POSIX mode (to properly handle escaping)
lexer = shlex(text, posix=True)
# set ',' as whitespace for the lexer
# (the lexer will use this character to separate words)
lexer.whitespace = item_sep
# include '=' as a word character
# (this is done so that the lexer returns a list of key-value pairs)
# (if your option key or value contains any unquoted special character, you will need to add it here)
lexer.wordchars += value_sep
# then we separate option keys and values to build the resulting dictionary
# (maxsplit is required to make sure that '=' in value will not be a problem)
return dict(word.split(value_sep, maxsplit=1) for word in lexer)
(split has a maxsplit argument, that is much cleaner to use than splitting/slicing/joining.)
Example run:
parse_kv_pairs(
'key1=value1,key2=\'value2,still_value2,not_key1="not_value1"\''
)
Output:
{'key1': 'value1', 'key2': 'value2,still_value2,not_key1="not_value1"'}
The reason I usually stick with shlex rather than using regular expressions (which are faster in this case) is that it gives you fewer surprises, especially if you need to allow more possible inputs later on. I never found how to properly parse such key-value pairs with regular expressions, there will always be inputs (e.g. A="B=\"1,2,3\"") that will trick the engine.
If you do not care about such inputs, (or, put another way, if you can ensure that your input follows the definition of a regular language), regular expressions are perfectly fine.
Answer from pistache on Stack OverflowI would advise against using regular expressions for this task, because the language you want to parse is not regular.
You have a character string of multiple key value pairs. The best way to parse this is not to match patterns on it, but to properly tokenize it.
There is a module in the Python standard library, called shlex, that mimics the parsing done by POSIX shells, and that provides a lexer implementation that can easily be customized to your needs.
from shlex import shlex
def parse_kv_pairs(text, item_sep=",", value_sep="="):
"""Parse key-value pairs from a shell-like text."""
# initialize a lexer, in POSIX mode (to properly handle escaping)
lexer = shlex(text, posix=True)
# set ',' as whitespace for the lexer
# (the lexer will use this character to separate words)
lexer.whitespace = item_sep
# include '=' as a word character
# (this is done so that the lexer returns a list of key-value pairs)
# (if your option key or value contains any unquoted special character, you will need to add it here)
lexer.wordchars += value_sep
# then we separate option keys and values to build the resulting dictionary
# (maxsplit is required to make sure that '=' in value will not be a problem)
return dict(word.split(value_sep, maxsplit=1) for word in lexer)
(split has a maxsplit argument, that is much cleaner to use than splitting/slicing/joining.)
Example run:
parse_kv_pairs(
'key1=value1,key2=\'value2,still_value2,not_key1="not_value1"\''
)
Output:
{'key1': 'value1', 'key2': 'value2,still_value2,not_key1="not_value1"'}
The reason I usually stick with shlex rather than using regular expressions (which are faster in this case) is that it gives you fewer surprises, especially if you need to allow more possible inputs later on. I never found how to properly parse such key-value pairs with regular expressions, there will always be inputs (e.g. A="B=\"1,2,3\"") that will trick the engine.
If you do not care about such inputs, (or, put another way, if you can ensure that your input follows the definition of a regular language), regular expressions are perfectly fine.
Using some regex magic from Split a string, respect and preserve quotes, we can do:
import re
string = 'key1=value1,key2="value2,still_value2"'
key_value_pairs = re.findall(r'(?:[^\s,"]|"(?:\\.|[^"])*")+', string)
for key_value_pair in key_value_pairs:
key, value = key_value_pair.split("=")
Per BioGeek, my attempt to guess, I mean interpret the regex Janne Karila used: The pattern breaks strings on commas but respects double quoted sections (potentially with commas) in the process. It has two separate options: runs of characters that don't involve quotes; and double quoted runs of characters where a double quote finishes the run unless it's (backslash) escaped:
(?: # parenthesis for alternation (|), not memory
[^\s,"] # any 1 character except white space, comma or quote
| # or
"(?:\\.|[^"])*" # a quoted string containing 0 or more characters
# other than quotes (unless escaped)
)+ # one or more of the above
Don't use a regular expression where string splitting will work:
dict(s.split('=', 1) for s in r.split())
Demo:
>>> r = "key1=value1 key2=value2 request=http://www.pandora.com/json/music/artist/justin-moore?explicit\\=false uri=3DLoiRDsBABCAA9FvE1htRg\\=\\="
>>> dict(s.split('=', 1) for s in r.split())
{'key2': 'value2', 'key1': 'value1', 'request': 'http://www.pandora.com/json/music/artist/justin-moore?explicit\\=false', 'uri': '3DLoiRDsBABCAA9FvE1htRg\\=\\='}
This removes the need to escape = characters.
got it.
regex = re.compile(r"\b(\w+)=(.*?)(?=\s\w+=\s*|$)")
Your string seems JSON array, you can easily parse json in any language if it's valid. To make your string valid add '[' at first and ']' at last of your string then parse the JSON in your language. Such as
JavaScript:
JSON.parse(st)
Python:
import json
dict = json.loads(st)
Regular expression:
if you strictly wish to parse using regular expression use:
/(?:\"|\')(?<key>[\w\d]+)(?:\"|\')(?:\:\s*)(?:\"|\')?(?<value>[\w\s-]*)(?:\"|\')?/gm
Try this:
(?<="firstName":")[^"\r\n]+
(?<="firstName":") go to the point where "firstName":" appeasrs in the string,
[^"\r\n]+ then match one or more character except ", \r and \n. not to cross the second double quote of the firstName value and not to cross any newline.
See regex demo.
See python demo.
You have a few ways to do this. First, since you said your key-value pairs are embedded in a larger email, you need to extract them. You can do that with this regex, which checks for a line starting with a word and an equals sign:
import re
text = " ... Full email text ... "
dataPoints = re.search(r"^\w*=.*$", text, re.MULTILINE).group(0)
Then you need to create your dictionary.
Option 1: Simplest
Use the following regex find:
result = dict(re.findall(r'(\w*)=(\".*?\"|\S*)', dataPoints))
Regex demo
Option 2: Typical split
Follow the typical method for this problem: split the various key-value combinations into a list, and then split each combination into separate keys and values. However, since your key-value pairs are separated by spaces rather than semicolons, ampersands, or something similar, and some of your values have spaces in them, we can't simply split by spaces. That means we need to use a regex lookahead for this to work properly:
regexSplit = dict([i.split("=") for i in re.split(r"\s(?=\w+=)", dataPoints)])
Option 3: No regex
If you want to avoid using regex altogether for whatever reason, you can use the following, which splits on equals signs and then recombines the keys and values into the proper arrangement for creating a dictionary:
allSplits = dataPoints.split("=")
splitList = [allSplits[0]] + [i for a in allSplits[1:-1]
for i in a.rsplit(" ", 1)] + [allSplits[-1]]
splitDict = dict(zip(splitList[::2], splitList[1::2]))
The code above assumes your dictionary will end up with at least 2 items.
Demo for all 3 options
What about adding an OR (|) to your regex, e.g.
(\w*)=(\"[\w\s\+()\.]*\"|[\w\-\:\.]*)
matches the string you gave.
Note
\"[\w\s\+()\.]*\"matches all the values enclosed in""[\w\-\:\.]*matches the ones without
This should do it:
new_dict = dict()
for key, value in orig_dict.items():
if 'Ethernet' in str(key):
new_dict[key] = value
When you use print, python calls the __str__ method on the OctetString object, which returns Ethernet8/6. However, I think pprint defaults to printing the object type.
EDIT:
Stefan Pochmann has rightly pointed out below that if 'Ethernet' in will match any string which contains the word Ethernet. The OP did mention using regex in his post to match Ethernet(\d*)/(\d*), so this answer may not be suitable to anyone else looking to solve a similar problem.
(I'll use the same 'Ethernet' in str(key) test as the accepted answer.)
If you want to keep the original dict and have the filtered version as a separate dictionary, I'd use a comprehension:
newdict = {key: value
for key, value in mydict.items()
if 'Ethernet' in str(key)}
If you don't want to keep the original dict, you can also just remove the entries you don't want:
for key in list(mydict):
if 'Ethernet' in str(key):
del mydict[key]
The reason you get "OctetString('...')" is the same as this one:
>>> 'foo'
'foo'
>>> pprint.pprint('foo')
'foo'
>>> print('foo')
foo
The first two tests show you a representation you can use in source code, that's why there are quotes. It's what the repr function gets you. The third test prints the value for normal pleasure, so doesn't add quotes. The "OctetString('...')" is simply such a representation as well, and you can copy&paste it into source code and get actual OctetString objects again, rather than Python string objects. I guess pprint is mostly intended for developing, where it's more useful to get the full repr version.
Try this, using the dictionary comprehensions available in Python 2.7 or newer:
{ k:v for k,v in my_dict.items() if 'Peter' in k }
Alternatively, if we're certain that the name will always be in the first position, we can do this, which is a bit faster:
{ k:v for k,v in my_dict.items() if k[0] == 'Peter' }
If you're using an older version of Python, we can get an equivalent result using generator expressions and the right parameters for the dict() constructor:
dict((k,v) for k,v in my_dict.items() if k[0] == 'Peter')
Anyway, the result is as expected:
=> {('Peter', 'Cell1', 8): 8, ('Peter', 'Cell1', 6): 0}
for any name
def select(d, name):
xs = {}
for e in d:
if e[0].lower() == name.lower(): xs[e] = d[e]
return xs
d = {('Alice', 'Cell3', 3): 9,
('Bob', 'Cell2', 6): 8,
('Peter', 'Cell1', 6): 0,
('Alice', 'Cell1', 6): 4,
('Bob', 'Cell2', 0): 4,
('Peter', 'Cell1', 8): 8
}
print select(d, 'peter')
>>>{('Peter', 'Cell1', 8): 8, ('Peter', 'Cell1', 6): 0}
>>> r = "name: srek age :24 description: blah blah"
>>> import re
>>> regex = re.compile(r"\b(\w+)\s*:\s*([^:]*)(?=\s+\w+\s*:|$)")
>>> d = dict(regex.findall(r))
>>> d
{'age': '24', 'name': 'srek', 'description': 'blah blah'}
Explanation:
\b # Start at a word boundary
(\w+) # Match and capture a single word (1+ alnum characters)
\s*:\s* # Match a colon, optionally surrounded by whitespace
([^:]*) # Match any number of non-colon characters
(?= # Make sure that we stop when the following can be matched:
\s+\w+\s*: # the next dictionary key
| # or
$ # the end of the string
) # End of lookahead
without re:
r = "name: srek age :24 description: blah blah cat: dog stack:overflow"
lis=r.split(':')
dic={}
try :
for i,x in enumerate(reversed(lis)):
i+=1
slast=lis[-(i+1)]
slast=slast.split()
dic[slast[-1]]=x
lis[-(i+1)]=" ".join(slast[:-1])
except IndexError:pass
print(dic)
{'age': '24', 'description': 'blah blah', 'stack': 'overflow', 'name': 'srek', 'cat': 'dog'}