text = re.sub(r'\b(\w+)( \1\b)+', r'\1', text) #remove duplicated words in row

The \b matches the empty string, but only at the beginning or end of a word.

Since you were working with RegEx, I will ofer a RegEx solution.

I will also show that you need to also take care to first remove punctuation. (I will not go down the rabbit-hole of re-sinserting the punctuation back where it was!)

A RegEx solution:

import re
sentence = 'I need need to learn regex... regex from scratch!'

# remove punctuation
# the unicode flag makes it work for more letter types (non-ascii)
no_punc = re.sub(r'[^\w\s]', '', sentence, re.UNICODE)
print('No punctuation:', no_punc)

# remove duplicates
re_output = re.sub(r'\b(\w+)( \1\b)+', r'\1', no_punc)
print('No duplicates:', re_output)

Returns:

No punctuation: I need need to learn regex regex from scratch
No duplicates: I need to learn regex from scratch

• \b : matches word boundaries
• \w : any word character
• \1 : replaces the matches with the second word found - the group in the second set of parentheses

The parts in parentheses are referred to as groups, and you can do things like name them and refer to them later in a regex. This pattern should recursively catch repeating words, so if there were 10 in a row, they get replaced with just the final occurence.

Have a look here for more detailed definitions of the regex patterns.

The more pythonic (looking) way

It has to be said that the groupby method has a certain python-zen feel about it! Simple, easy to read, beautiful.

Here I just show another way of removing the punctuation, making use of the string module, translating any punctuation characters into None (which removes them):

from itertools import groupby
import string

sentence = 'I need need to learn regex... regex from scratch!'

# Remove punctuation
sent_map = sentence.maketrans(dict.fromkeys(string.punctuation))
sent_clean = sentence.translate(sent_map)
print('Clean sentence:', sent_clean)

no_dupes = ([k for k, v in groupby(sent_clean.split())])
print('No duplicates:', no_dupes)

# Put the list back together into a sentence
groupby_output = ' '.join(no_dupes)
print('Final output:', groupby_output)

# At least for this toy example, the outputs are identical:
print('Identical output:', re_output == groupby_output)

Returns:

Clean sentence: I need need to learn regex regex from scratch
No duplicates: ['I', 'need', 'to', 'learn', 'regex', 'from', 'scratch']
Final output: I need to learn regex from scratch
Identical output: True

Benchmarks

Out of curiosity, I dumped the lines above into functions and ran a simple benchmark:

RegEx:

In [1]: %timeit remove_regex(sentence)
8.17 µs ± 88.6 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

groupby:

In [2]: %timeit remove_groupby(sentence)
5.89 µs ± 527 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

I had read that regex would be faster these days (using Python3.6) - but it seems that sticking to beautiful code pays off in this case!

Disclaimer: the example sentence was very short. This result might not scale to sentences with more/less repeated words and punctuation!

You need to doubly escape the back-reference:

re.sub('(.+) \(\\1\)', '\\1', 'the (the)')
--> the

Or use the r prefix:

When an "r" or "R" prefix is present, a character following a backslash is included in the string without change, and all backslashes are left in the string.

re.sub(r'(.+) \(\1\)', r'\1', 'the (the)')
--> the

you could back reference the word in your search expression:

>>> s = "server_server_dev1_check_1233.zzz"
>>> re.sub(r"(.*_)\1",r"\1",s)
'server_dev1_check_1233.zzz'

and use the "many times" suffix so if there are more than 2 occurrences it still works:

'server_server_server_dev1_check_1233.zzz'
>>> re.sub(r"(.*_)\1{1,}",r"\1",s)
'server_dev1_check_1233.zzz'

getting rid of the suffix is not the hardest part, just capture the rest and discard the end:

>>> re.sub(r"(.*_)\1{1,}(.*)(_\d+\..*)",r"\1\2",s)
'server_dev1_check'

You can use a regular expression to remove consecutive duplicated words in a line, however I don't think it's possible to remove duplicated words which are not consecutive (e.g dangerous, hazardous, dangerous).

Use this regex in the replace window in Notepad++, and don't forget to select "Regular expression" as the Search Mode option below:

This regex will remove all consecutive duplicated words - whether it's 2 duplicated words or 10 duplicated words consecutively: \b(\w+)(?:,\s+\1\b)+.

The exact same no-commas regex would be: \b(\w+)(?:\s+\1\b)+ (might be useful for other users).

If you want a regex specifically for only two duplicated words (doubles), use this regex: (\b\w+\b)\W+\1.

Place this regex in the Replace with box to keep one occurrence of the word (otherwise all repeated words will be removed): ${1}.

These regular expressions will fix a situation like the one you described in your question as an example. The first regex will work for every number of duplicated words (e.g dangerous, dangerous, dangerous, dangerous, hazardous), while the second version will only work for two duplicated words (e.g dangerous, dangerous, hazardous).

Note: The regular expression will only apply to the format described in the question, meaning that formats like two words, two words, anotherword, two-words, two-words, anotherword, three words expression, three words expression, anotherword won't be changed because the regex won't apply to them.

Your regex is almost correct.

• You need to add ? to the capturing group, so it matches as little as it can ("lazy matching" rather than the default "greedy" behavior that matches as much as possible).

• I also used + instead of {1,3} because limiting the repetition to 3 seemed arbitrary.

• You can observe the difference between the two behaviors: greedy vs lazy. Note that:

  1. The greedy behavior sees aaaa as aa * 2 rather than a * 4

  2. The greedy behavior only works for even-lengthed repetitions. aaaaa is seen as

     aa * 2 + a thus the replacement result would be aaa instead of a.

for word in "Thisssssssss isisisis echooooooo stringggg. Replaceaceaceace repeatedededed groupssss of symbolssss".split():
    print(re.sub(r'([a-z]+?)\1+', r'\1', word))

outputs

This
is
echo
string.
Replace
repeated
groups
of
symbols

>>> import re
>>> re.sub(r'([a-z])\1+', r'\1', 'ffffffbbbbbbbqqq')
'fbq'

The () around the [a-z] specify a capture group, and then the \1 (a backreference) in both the pattern and the replacement refer to the contents of the first capture group.

Thus, the regex reads "find a letter, followed by one or more occurrences of that same letter" and then entire found portion is replaced with a single occurrence of the found letter.

On side note...

Your example code for just a is actually buggy:

>>> re.sub('a*', 'a', 'aaabbbccc')
'abababacacaca'

You really would want to use 'a+' for your regex instead of 'a*', since the * operator matches "0 or more" occurrences, and thus will match empty strings in between two non-a characters, whereas the + operator matches "1 or more".

Thanks everyone for your attempts and comments. I have finally found a solution:

s = 'I hate *some* kinds of duplicate. This string has a duplicate phrase, duplicate phrase.'
re.sub(r'((\b\w+\b.{1,2}\w+\b)+).+\1', r'\1', s, flags = re.I)
# 'I hate *some* kinds of duplicate. This string has a duplicate phrase.'

Explanation

The regular expression

r'((\b\w+\b.{1,2}\w+\b)+).+\1'

finds every occurrence of multiple runs of alphanumeric characters separated by one or two [any character] (to cover the case where words are separated not just by a space, but perhaps a period or comma and a space), and then repeated following some run of [any character] of indeterminate length. Then

re.sub(r'((\b\w+\b.{1,2}\w+\b)+).+\1', r'\1', s, flags = re.I)

replaces such occurrences with the first multiple run of alphanumeric characters separated by one or two [any character], being sure to ignore case (since the duplicate phrase could sometimes occur at the beginning of a sentence).