There is a package called thefuzz. Install via pip:
pip install thefuzz
Simple usage:
>>> from thefuzz import fuzz
>>> fuzz.ratio("this is a test", "this is a test!")
97
The package is built on top of difflib. Why not just use that, you ask? Apart from being a bit simpler, it has a number of different matching methods (like token order insensitivity, partial string matching) which make it more powerful in practice. The process.extract functions are especially useful: find the best matching strings and ratios from a set. From their readme:
Partial Ratio
>>> fuzz.partial_ratio("this is a test", "this is a test!")
100
Token Sort Ratio
>>> fuzz.ratio("fuzzy wuzzy was a bear", "wuzzy fuzzy was a bear")
90
>>> fuzz.token_sort_ratio("fuzzy wuzzy was a bear", "wuzzy fuzzy was a bear")
100
Token Set Ratio
>>> fuzz.token_sort_ratio("fuzzy was a bear", "fuzzy fuzzy was a bear")
84
>>> fuzz.token_set_ratio("fuzzy was a bear", "fuzzy fuzzy was a bear")
100
Process
>>> choices = ["Atlanta Falcons", "New York Jets", "New York Giants", "Dallas Cowboys"]
>>> process.extract("new york jets", choices, limit=2)
[('New York Jets', 100), ('New York Giants', 78)]
>>> process.extractOne("cowboys", choices)
("Dallas Cowboys", 90)
Answer from congusbongus on Stack OverflowFuzzy Matching or Other Alternativies?
Fuzzy Logic Matching in Python
The built-in difflib module has the get_close_matches() method.
More on reddit.comUnsupervised Learning for String Matching in Python - can I have advice on how to go about this?
Fuzzy matching
Go for fuzzywuzzy. It's easy to use and have an active community. ๐
More on reddit.comThere is a package called thefuzz. Install via pip:
pip install thefuzz
Simple usage:
>>> from thefuzz import fuzz
>>> fuzz.ratio("this is a test", "this is a test!")
97
The package is built on top of difflib. Why not just use that, you ask? Apart from being a bit simpler, it has a number of different matching methods (like token order insensitivity, partial string matching) which make it more powerful in practice. The process.extract functions are especially useful: find the best matching strings and ratios from a set. From their readme:
Partial Ratio
>>> fuzz.partial_ratio("this is a test", "this is a test!")
100
Token Sort Ratio
>>> fuzz.ratio("fuzzy wuzzy was a bear", "wuzzy fuzzy was a bear")
90
>>> fuzz.token_sort_ratio("fuzzy wuzzy was a bear", "wuzzy fuzzy was a bear")
100
Token Set Ratio
>>> fuzz.token_sort_ratio("fuzzy was a bear", "fuzzy fuzzy was a bear")
84
>>> fuzz.token_set_ratio("fuzzy was a bear", "fuzzy fuzzy was a bear")
100
Process
>>> choices = ["Atlanta Falcons", "New York Jets", "New York Giants", "Dallas Cowboys"]
>>> process.extract("new york jets", choices, limit=2)
[('New York Jets', 100), ('New York Giants', 78)]
>>> process.extractOne("cowboys", choices)
("Dallas Cowboys", 90)
There is a module in the standard library (called difflib) that can compare strings and return a score based on their similarity. The SequenceMatcher class should do what you want.
Small example from Python prompt:
>>> from difflib import SequenceMatcher as SM
>>> s1 = ' It was a dark and stormy night. I was all alone sitting on a red chair. I was not completely alone as I had three cats.'
>>> s2 = ' It was a murky and stormy night. I was all alone sitting on a crimson chair. I was not completely alone as I had three felines.'
>>> SM(None, s1, s2).ratio()
0.9112903225806451
Hi,
My file contains a lot of garbage for the field name city
I'm trying to correct:
For example,
Correct: Los Angeles
Incorrect: Las Angelas
*Las Angelas is a typo. But, the tool will group them together to the correct one.
Second example:
Correct: Los Angeles@!#@!#
Incorrect: Los Angeles,CA
*I don't need to see ,CA there. Just the name of the city.
Third Example:
Correct: Edmonton
Incorrect: Edmonton Alberta
*I don't need to see Alberta there. Just the name of the city.
There are a lot of typos in my data and just wondering the quickest way to handle this and almost impossible to me to correct one by one. Thank you!
What should I do to handle this? should I use fuzzy matching or is there any other tool that is the best to handle this?
ยป pip install fuzzy-match
ยป pip install fuzzywuzzy