You could try something like this:
import difflib
possibilities = ['Summerdalerise', 'Winterstreamrise']
line = 'I went up to Winterstreamrose.'
newWords = []
for word in line.split():
result = difflib.get_close_matches(word, possibilities, n=1)
newWords.append(result[0] if result else word)
result = ' '.join(newWords)
print(result)
Output:
I went up to Winterstreamrise
Explanation:
- The docs show a first argument named
word, and there is no suggestion thatget_close_matches()has any awareness of sub-words within this argument; rather, it reports on the closeness of a match between this word atomically and the list ofpossibilitiessupplied as the second argument. - We can add the awareness of words within
lineby splitting it into a list of such words which we iterate over, callingget_close_matches()for each word separately and modifying the word in our result only if there is a match.
Using difflib.get_close_matches to replace word in string - Python - Stack Overflow
python 2.7 - difflib.get_close_matches GET SCORE - Stack Overflow
string - How does the python difflib.get_close_matches() function work? - Stack Overflow
python - Is there an alternative to `difflib.get_close_matches()` that returns indexes (list positions) instead of a str list? - Stack Overflow
I recently started programming and I stumbled upon th difflib.get_close_matches function when I tried to come up with a way to give a close match upon entering an invalid statement. Now I implemented the function into my programm however I still don't fully understand how exactly this thing works. I only know that it looks for the most adjacent correct letters in all words and determines the close mathes that way. But how does it compare the letters with one antoher? Does it transform every word into a list and compares it with the input?
I found that difflib.get_close_matches is the simplest way for matching/fuzzy-matching strings. But there are a few other more advanced libraries like fuzzywuzzy as you mentioned in the comments.
But if you want to use difflib, you can use difflib.SequenceMatcher to get the score as follows:
import difflib
my_str = 'apple'
str_list = ['ape' , 'fjsdf', 'aerewtg', 'dgyow', 'paepd']
best_match = difflib.get_close_matches(my_str,str_list,1)[0]
score = difflib.SequenceMatcher(None, my_str, best_match).ratio()
In this example, the best match between 'apple' and the list is 'ape' and the score is 0.75.
You can also loop through the list and compute all the scores to check:
for word in str_list:
print "score for: " + my_str + " vs. " + word + " = " + str(difflib.SequenceMatcher(None, my_str, word).ratio())
For this example, you get the following:
score for: apple vs. ape = 0.75
score for: apple vs. fjsdf = 0.0
score for: apple vs. aerewtg = 0.333333333333
score for: apple vs. dgyow = 0.0
score for: apple vs. paepd = 0.4
Documentation for difflib can be found here: https://docs.python.org/2/library/difflib.html
To answer the question, the usual route would be to obtain the comparative score for a match returned by get_close_matches() individually in this manner:
match_ratio = difflib.SequenceMatcher(None, 'aple', 'apple').ratio()
Here's a way that increases speed in my case by about 10% ...
I'm using get_close_matches() for spellcheck, it runs SequenceMatcher() under the hood but strips the scores returning just a list of matching strings. Normally.
But with a small change in Lib/difflib.py currently around line 736 the return can be a dictionary with scores as values, thus no need to run SequenceMatcher again on each list item to obtain their score ratios. In the examples I've shortened the output float values for clarity (like 0.8888888888888888 to 0.889). Input n=7 says to limit the return items to 7 if there are more than 7, i.e. the highest 7, and that could apply if candidates are many.
Current mere list return
In this example result would normally be like ['apple', 'staple', 'able', 'lapel']
... at the default cutoff of .6 if omitted (as in Ben's answer, no judgement).
The change
in difflib.py is simple (this line to the right shows the original):
return {v: k for (k, v) in result} # hack to return dict with scores instead of list, original was ... [x for score, x in result]
New dictionary return
includes scores like {'apple': 0.889, 'staple': 0.8, 'able': 0.75, 'lapel': 0.667}
>>> to_match = 'aple'
>>> candidates = ['lapel', 'staple', 'zoo', 'able', 'apple', 'appealing']
Increasing minimum score cutoff/threshold from .4 to .8:
>>> difflib.get_close_matches(to_match, candidates, n=7, cutoff=.4)
{'apple': 0.889, 'staple': 0.8, 'able': 0.75, 'lapel': 0.667, 'appealing': 0.461}
>>> difflib.get_close_matches(to_match, candidates, n=7, cutoff=.7)
{'apple': 0.889, 'staple': 0.8, 'able': 0.75}
>>> difflib.get_close_matches(to_match, candidates, n=7, cutoff=.8)
{'apple': 0.889, 'staple': 0.8}
Well, there is this part in the docs explaining your issue:
This does not yield minimal edit sequences, but does tend to yield matches that “look right” to people.
For getting the results you are expecting you could use the Levenshtein_distance.
But for comparing IPs I would suggest to use integer comparison:
>>> parts = [int(s) for s in '198.124.252.130'.split('.')]
>>> parts2 = [int(s) for s in '198.124.252.101'.split('.')]
>>> from operator import sub
>>> diff = sum(d * 10**(3-pos) for pos,d in enumerate(map(sub, parts, parts2)))
>>> diff
29
You can use this style to create a compare function:
from functools import partial
from operator import sub
def compare_ips(base, ip1, ip2):
base = [int(s) for s in base.split('.')]
parts1 = (int(s) for s in ip1.split('.'))
parts2 = (int(s) for s in ip2.split('.'))
test1 = sum(abs(d * 10**(3-pos)) for pos,d in enumerate(map(sub, base, parts1)))
test2 = sum(abs(d * 10**(3-pos)) for pos,d in enumerate(map(sub, base, parts2)))
return cmp(test1, test2)
base = '198.124.252.101'
test_list = ['198.124.252.102','134.55.41.41','134.55.219.121',
'134.55.219.137','134.55.220.45', '198.124.252.130']
sorted(test_list, cmp=partial(compare_ips, base))
# yields:
# ['198.124.252.102', '198.124.252.130', '134.55.219.121', '134.55.219.137',
# '134.55.220.45', '134.55.41.41']
Some hint from difflib:
SequenceMatcher is a flexible class for comparing pairs of sequences of any type, so long as the sequence elements are hashable. The basic algorithm predates, and is a little fancier than, an algorithm published in the late 1980's by Ratcliff and Obershelp under the hyperbolic name "gestalt pattern matching". The basic idea is to find the longest contiguous matching subsequence that contains no "junk" elements (R-O doesn't address junk). The same idea is then applied recursively to the pieces of the sequences to the left and to the right of the matching subsequence. This does not yield minimal edit sequences, but does tend to yield matches that "look right" to people.
Regarding your requirement to compare IPs based on custom logic. You should first validate if the string is proper ip. Then writing comparison logic using simple integer arithmetic should be an easy task to fulfill your requirement. A library is not needed at all.
I took the source code for get_close_matches, and modify it in order to return the indexes instead of the string values.
# mydifflib.py
from difflib import SequenceMatcher
from heapq import nlargest as _nlargest
def get_close_matches_indexes(word, possibilities, n=3, cutoff=0.6):
"""Use SequenceMatcher to return a list of the indexes of the best
"good enough" matches. word is a sequence for which close matches
are desired (typically a string).
possibilities is a list of sequences against which to match word
(typically a list of strings).
Optional arg n (default 3) is the maximum number of close matches to
return. n must be > 0.
Optional arg cutoff (default 0.6) is a float in [0, 1]. Possibilities
that don't score at least that similar to word are ignored.
"""
if not n > 0:
raise ValueError("n must be > 0: %r" % (n,))
if not 0.0 <= cutoff <= 1.0:
raise ValueError("cutoff must be in [0.0, 1.0]: %r" % (cutoff,))
result = []
s = SequenceMatcher()
s.set_seq2(word)
for idx, x in enumerate(possibilities):
s.set_seq1(x)
if s.real_quick_ratio() >= cutoff and \
s.quick_ratio() >= cutoff and \
s.ratio() >= cutoff:
result.append((s.ratio(), idx))
# Move the best scorers to head of list
result = _nlargest(n, result)
# Strip scores for the best n matches
return [x for score, x in result]
Usage
>>> from mydifflib import get_close_matches_indexes
>>> words = ['hello', 'Hallo', 'hi', 'house', 'key', 'screen', 'hallo', 'question', 'format']
>>> get_close_matches_indexes('hello', words)
[0, 1, 6]
Now, I can relate this indexes to associated data of the string without having to search back the strings.
Not an exact answer to your question but I was trying to find a simpler single match index and the syntax is
match_string = difflib.get_close_matches(appx_name_str,names_list,n=1,cutoff=0.1)[0] # get the most similar string
match_index = names_list.index(match_string) # index method on list of strings
I came across the same question and I found that "difflib.get_close_matches" uses as foundation the approach on called "Gestalt pattern matching" described by Ratcliff and Obershelp (link below).
The method "difflib.get_close_matches" is based on the class "SequenceMatcher", which in the source code specify this: "SequenceMatcher is a flexible class for comparing pairs of sequences of any type, so long as the sequence elements are hashable. The basic algorithm predates, and is a little fancier than, an algorithm published in the late 1980's by Ratcliff and Obershelp under the hyperbolic name "gestalt pattern matching". The basic idea is to find the longest contiguous matching subsequence that contains no "junk" elements (R-O doesn't address junk). The same idea is then applied recursively to the pieces of the sequences to the left and to the right of the matching subsequence. This does not yield minimal edit sequences, but does tend to yield matches that "look right" to people."
About the "cutoff". This tells you how close you want to find the match, if "1" then it needs to be exactly the same word, and as going down it's more relax. So for instance, if you choose "0" it will for sure return you the most "similar" work no matter you don't have any similar one, so this would not make much sense on most of the cases. It's then "0.6" the default, as this can give significant results, but its up to any particular solution, you need to test what it works for you based on your vocabulary and specific scenario.
PATTERN MATCHING: THE GESTALT APPROACH http://collaboration.cmc.ec.gc.ca/science/rpn/biblio/ddj/Website/articles/DDJ/1988/8807/8807c/8807c.htm
Hope this helps you to understand "difflib.get_close_matches" better.
From the documentation:
Optional argument
cutoff(default0.6) is afloatin the range[0, 1]. Possibilities that don’t score at least that similar to word are ignored.
Trying the example from the documentation:
In [11]: import difflib
In [12]: difflib.get_close_matches('appel', ['ape', 'apple', 'peach', 'puppy'])
Out[12]: ['apple', 'ape']
In [13]: difflib.get_close_matches('appel', ['ape', 'apple', 'peach', 'puppy'], cutoff=0.1)
Out[13]: ['apple', 'ape', 'puppy']
In [14]: difflib.get_close_matches('appel', ['ape', 'apple', 'peach', 'puppy'], cutoff=0.9)
Out[14]: []
Details about the algorithm are noted in the article "Pattern Matching: The Gestalt Approach".