In case you're interested in a quick visual comparison of Levenshtein and Difflib similarity, I calculated both for ~2.3 million book titles:
import codecs, difflib, Levenshtein, distance
with codecs.open("titles.tsv","r","utf-8") as f:
title_list = f.read().split("\n")[:-1]
for row in title_list:
sr = row.lower().split("\t")
diffl = difflib.SequenceMatcher(None, sr[3], sr[4]).ratio()
lev = Levenshtein.ratio(sr[3], sr[4])
sor = 1 - distance.sorensen(sr[3], sr[4])
jac = 1 - distance.jaccard(sr[3], sr[4])
print diffl, lev, sor, jac
I then plotted the results with R:

Strictly for the curious, I also compared the Difflib, Levenshtein, Sørensen, and Jaccard similarity values:
library(ggplot2)
require(GGally)
difflib <- read.table("similarity_measures.txt", sep = " ")
colnames(difflib) <- c("difflib", "levenshtein", "sorensen", "jaccard")
ggpairs(difflib)
Result:

The Difflib / Levenshtein similarity really is quite interesting.
2018 edit: If you're working on identifying similar strings, you could also check out minhashing--there's a great overview here. Minhashing is amazing at finding similarities in large text collections in linear time. My lab put together an app that detects and visualizes text reuse using minhashing here: https://github.com/YaleDHLab/intertext
Answer from duhaime on Stack OverflowIn case you're interested in a quick visual comparison of Levenshtein and Difflib similarity, I calculated both for ~2.3 million book titles:
import codecs, difflib, Levenshtein, distance
with codecs.open("titles.tsv","r","utf-8") as f:
title_list = f.read().split("\n")[:-1]
for row in title_list:
sr = row.lower().split("\t")
diffl = difflib.SequenceMatcher(None, sr[3], sr[4]).ratio()
lev = Levenshtein.ratio(sr[3], sr[4])
sor = 1 - distance.sorensen(sr[3], sr[4])
jac = 1 - distance.jaccard(sr[3], sr[4])
print diffl, lev, sor, jac
I then plotted the results with R:

Strictly for the curious, I also compared the Difflib, Levenshtein, Sørensen, and Jaccard similarity values:
library(ggplot2)
require(GGally)
difflib <- read.table("similarity_measures.txt", sep = " ")
colnames(difflib) <- c("difflib", "levenshtein", "sorensen", "jaccard")
ggpairs(difflib)
Result:

The Difflib / Levenshtein similarity really is quite interesting.
2018 edit: If you're working on identifying similar strings, you could also check out minhashing--there's a great overview here. Minhashing is amazing at finding similarities in large text collections in linear time. My lab put together an app that detects and visualizes text reuse using minhashing here: https://github.com/YaleDHLab/intertext
difflib.SequenceMatcher uses the Ratcliff/Obershelp algorithm it computes the doubled number of matching characters divided by the total number of characters in the two strings.
Levenshtein uses Levenshtein algorithm it computes the minimum number of edits needed to transform one string into the other
Complexity
SequenceMatcher is quadratic time for the worst case and has expected-case behavior dependent in a complicated way on how many elements the sequences have in common. (from here)
Levenshtein is O(m*n), where n and m are the length of the two input strings.
Performance
According to the source code of the Levenshtein module : Levenshtein has a some overlap with difflib (SequenceMatcher). It supports only strings, not arbitrary sequence types, but on the other hand it's much faster.
'Fuzzy' string match
Yes, that is fine. Just return the result though, no need for the if: True else: False since the result will be truthy or falsy
python - Better fuzzy matching performance? - Stack Overflow
Difflib and python-Levenshtein give different ratios in some cases
Fuzzy Match values to list of list python - Stack Overflow
I found difflib.get_close_matches() while searching for a way to 'fuzzy match' strings. The matching algorithm seems good for my data, but I only want to check one string against one other and know if they 'match'. So:
def fuzzy_match(test_string, possible_match):
if difflib.get_close_matches(test_string, [possible_match]):
return True
else:
return FalseDoes that seem like a decent solution? Any suggestions? FWIW, I'd prefer to stick within the standard library.
Edit: updated version per indosauros:
def fuzzy_match(test_string, possible_match):
return difflib.get_close_matches(test_string, [possible_match])Edit 2: Or maybe do it right? (basically a stripped down version of get_close_matches):
def fuzzy_match(test_string, possible_match, cutoff=0.6):
s = difflib.SequenceMatcher()
s.set_seq2(test_string)
s.set_seq1(possible_match)
if s.real_quick_ratio() >= cutoff and \
s.quick_ratio() >= cutoff and \
s.ratio() >= cutoff:
return True
else:
return Falsefuzzyset indexes strings by their bigrams and trigrams so it finds approximate matches in O(log(N)) vs O(N) for difflib. For my fuzzyset of 1M+ words and word-pairs it can compute the index in about 20 seconds and find the closest match in less than a 100 ms.
RapidFuzz
is the super-fast lib for fuzzy string matching. It has the same API as famous fuzzywuzzy, but times faster and MIT licensed.
You can implement fuzzy matching obtaining best match ratio (using max()) returned by difflib.SequenceMatcher().
To implement this we should pass lambda as key argument which will return match ratio. In my example I'd use SequenceMatcher.ratio(), but if performance is important you should also try with SequenceMatcher.quick_ratio() and SequenceMatcher.real_quick_ratio().
from difflib import SequenceMatcher
lst = [('Jimmy', 'Smith'), ('James', 'Wilson'), ('Hugh', 'Laurie')]
first_name = 'Jimm'
last_name = 'Smitn'
matcher = SequenceMatcher(a=first_name + ' ' + last_name)
match_first_name, match_last_name = max(lst,
key=lambda x: matcher.set_seq2(' '.join(x)) or matcher.ratio())
print(first_name, last_name, '-', match_first_name, match_last_name)
Another possible path would be to use set intersections.
names = [('Jimmy', 'Smith'), ('James', 'Wilson'), ('Hugh', 'Laurie')]
first_name = "Jimm"
last_name = "Smitn"
setf = set(first_name)
# {'m', 'i', 'J'}
setl = set(last_name)
# {'t', 'n', 'm', 'i', 'S'}
ranked = [(len(setf & set(f)) + len(setl & set(l)), f, l) for f, l in names]
# [(7, 'Jimmy', 'Smith'), (4, 'James', 'Wilson'), (1, 'Hugh', 'Laurie')]
best_match = max(ranked, key=lambda x: x[0])[1:]
# ('Jimmy', 'Smith')
To eliminate the possibility of low-score matches as a result of case-differences, I'd suggest applying .upper() or .lower() to the columns you're matching. After adjusting the case, you could compile a list of all titles into ThisList and apply the following function (relying, as you suggested, on SequenceMatcher) with a given tolerance.
def fuzzy_group_list_elements(ThisList,Tolerance):
from difflib import SequenceMatcher
Groups = {}
TempList = ThisList.copy()
for Elmt in TempList:
if Elmt not in Groups.keys():
Groups[Elmt] = []
for OtherElmt in TempList:
if SequenceMatcher(None,Elmt,OtherElmt).quick_ratio() > Tolerance:
Groups[Elmt] = Groups[Elmt] + [OtherElmt]
TempList.remove(OtherElmt)
Groups[Elmt] = list(set(Groups[Elmt]))
return dict((v,k) for k in Groups for v in Groups[k])
You can then apply the above function to the dataframe columns containing the movie titles:
Mapping = fuzzy_group_list_elements(ThisList,0.85)
df['Matched Title'] = df['Title'].replace(Mapping)
I have written a Python package which aims to solve this problem. Amongst other things, it addresses the n^2 complexity of the problem (e.g. with two datasets of length 100, your code needs 10,000 comparisons).
You can install it using pip install fuzzymatcher
You can find the repo here and docs here.
Basic usage:
Given two dataframes df_left and df_right, which you want to fuzzy join, you can write the following:
from fuzzymatcher import link_table, left join
# Columns to match on from df_left
left_on = ["fname", "mname", "lname", "dob"]
# Columns to match on from df_right
right_on = ["name", "middlename", "surname", "date"]
# The link table potentially contains several matches for each record
fuzzymatcher.link_table(df_left, df_right, left_on, right_on)
Or if you just want to link on the closest match:
fuzzymatcher.fuzzy_left_join(df_left, df_right, left_on, right_on)