So there are following things which are wrong in your pattern, let's address them first
A-z- It includes all the character from ascii table starting from A to z, which also has non alphabetical characters which we don't want to match, so the correct one should be[A-Z]if we want only uppercase, if we want both upper and lowercase then it should be[A-Za-z]or you can turn oniflag^\s-^means negation only when you use it as first character inside the character class elsewhere it is treated as literal^
So your regex should be
[^A-Za-z\s]
Answer from Code Maniac on Stack OverflowSo there are following things which are wrong in your pattern, let's address them first
A-z- It includes all the character from ascii table starting from A to z, which also has non alphabetical characters which we don't want to match, so the correct one should be[A-Z]if we want only uppercase, if we want both upper and lowercase then it should be[A-Za-z]or you can turn oniflag^\s-^means negation only when you use it as first character inside the character class elsewhere it is treated as literal^
So your regex should be
[^A-Za-z\s]
To match all non-word and non-whitespace characters, you can use [^\w\s] - \w is any letter, number, or underscore, and \s is whitespace. If you'd prefer to only get letters, you can use [^a-zA-Z\s] instead.
(Also, when you're negating a capture group, you only need to put ^ at the very start.)
regex - Python, remove all non-alphabet chars from string - Stack Overflow
Regex match all non letters excluding diacritics (python) - Stack Overflow
python - How to replace all characters except letters, numbers, forward and back slashes - Stack Overflow
string - Python - efficient method to remove all non-letters and replace them with underscores - Stack Overflow
Regex to the rescue!
import re
s = re.sub('[^0-9a-zA-Z]+', '*', s)
Example:
>>> re.sub('[^0-9a-zA-Z]+', '*', 'h^&ell`.,|o w]{+orld')
'h*ell*o*w*orld'
The pythonic way.
print "".join([ c if c.isalnum() else "*" for c in s ])
This doesn't deal with grouping multiple consecutive non-matching characters though, i.e.
"h^&i => "h**i not "h*i" as in the regex solutions.
Use re.sub
import re
regex = re.compile('[^a-zA-Z]')
#First parameter is the replacement, second parameter is your input string
regex.sub('', 'ab3d*E')
#Out: 'abdE'
Alternatively, if you only want to remove a certain set of characters (as an apostrophe might be okay in your input...)
regex = re.compile('[,\.!?]') #etc.
If you prefer not to use regex, you might try
''.join([i for i in s if i.isalpha()])
A fair bit faster and works for Unicode:
full_pattern = re.compile('[^a-zA-Z0-9\\\/]|_')
def re_replace(string):
return re.sub(full_pattern, '', string)
If you want it really fast, this is by far the best (but slightly obscure) method:
def wanted(character):
return character.isalnum() or character in '\\/'
ascii_characters = [chr(ordinal) for ordinal in range(128)]
ascii_code_point_filter = [c if wanted(c) else None for c in ascii_characters]
def fast_replace(string):
# Remove all non-ASCII characters. Heavily optimised.
string = string.encode('ascii', errors='ignore').decode('ascii')
# Remove unwanted ASCII characters
return string.translate(ascii_code_point_filter)
Timings:
SETUP="
busy = ''.join(chr(i) for i in range(512))
import re
full_pattern = re.compile('[^a-zA-Z0-9\\\/]|_')
def in_whitelist(character):
return character.isalnum() or character in '\\/'
def re_replace(string):
return re.sub(full_pattern, '', string)
def wanted(character):
return character.isalnum() or character in '\\/'
ascii_characters = [chr(ordinal) for ordinal in range(128)]
ascii_code_point_filter = [c if wanted(c) else None for c in ascii_characters]
def fast_replace(string):
string = string.encode('ascii', errors='ignore').decode('ascii')
return string.translate(ascii_code_point_filter)
"
python -m timeit -s "$SETUP" "re_replace(busy)"
python -m timeit -s "$SETUP" "''.join(e for e in busy if in_whitelist(e))"
python -m timeit -s "$SETUP" "fast_replace(busy)"
Results:
10000 loops, best of 3: 63 usec per loop
10000 loops, best of 3: 135 usec per loop
100000 loops, best of 3: 4.98 usec per loop
Why can't you do something like:
def in_whitelist(character):
return character.isalnum() or character in ['\\','/']
line2 = ''.join(e for e in line2 if in_whitelist(e))
Edited as per suggestion to condense function.
The faster way to do it is to use str.translate()
This is ~50 times faster than your way
# You only need to do this once
>>> title_trans=''.join(chr(c) if chr(c).isupper() or chr(c).islower() else '_' for c in range(256))
>>> "abcde1234!@%^".translate(title_trans)
'abcde________'
# Using map+lambda
$ python -m timeit '"".join(map(lambda x: x if (x.isupper() or x.islower()) else "_", "abcd1234!@#$".strip()))'
10000 loops, best of 3: 21.9 usec per loop
# Using str.translate
$ python -m timeit -s 'titletrans="".join(chr(c) if chr(c).isupper() or chr(c).islower() else "_" for c in range(256))' '"abcd1234!@#$".translate(titletrans)'
1000000 loops, best of 3: 0.422 usec per loop
# Here is regex for a comparison
$ python -m timeit -s 'import re;transre=re.compile("[\W\d]+")' 'transre.sub("_","abcd1234!@#$")'
100000 loops, best of 3: 3.17 usec per loop
Here is a version for unicode
# coding: UTF-8
def format_title_unicode_translate(title):
return title.translate(title_unicode_trans)
class TitleUnicodeTranslate(dict):
def __missing__(self,item):
uni = unichr(item)
res = u"_"
if uni.isupper() or uni.islower():
res = uni
self[item] = res
return res
title_unicode_trans=TitleUnicodeTranslate()
print format_title_unicode_translate(u"Metallica Μεταλλικα")
Note that the Greek letters count as upper and lower, so they are not substituted. If they are to be substituted, simply change the condition to
if item<256 and (uni.isupper() or uni.islower()):
import re
title = re.sub("[\W\d]", "_", title.strip())
should be faster.
If you want to replace a succession of adjacent non-letters with a single underscore, use
title = re.sub("[\W\d]+", "_", title.strip())
instead which is even faster.
I just ran a time comparison:
C:\>python -m timeit -n 100 -s "data=open('test.txt').read().strip()" "''.join(map(lambda x: x if (x.isupper() or x.islower()) else '_', data))"
100 loops, best of 3: 4.51 msec per loop
C:\>python -m timeit -n 100 -s "import re; regex=re.compile('[\W\d]+'); data=open('test.txt').read().strip()" "title=regex.sub('_',data)"
100 loops, best of 3: 2.35 msec per loop
This will work on Unicode strings, too (under Python 3, \W matches any character which is not a Unicode word character. Under Python 2, you'd have to additionally set the UNICODE flag for this).
So basically I have to remove all instances from a string that isn't a letter, number, or space... So far I'm able to remove everything that isn't a letter or space, however, I'm not sure how I would keep numbers for the way I did it.
Any help would be greatly appreciated. My code is as follows:
def remove_punctuation(word):
new_word = ""
for i in range(len(word)):
for j in range(52):
if word[i] == alpha[j]:
new_word += alpha[j]
return new_word
alpha = [' ', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm',
'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z', 'A', 'B', 'C',
'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S',
'T', 'U', 'V', 'W', 'X', 'Y', 'Z']
print remove_punctuation("He$rs*hey {Bea%rs")Obviously I can't add every number in the list, lol.
FYI I do want to keep the commas between strings in the list. Here is my code right now. I've been working on this for hours and I'm stuck, I can't figure it out. I working on a program that will append every string of at least four characters to a new list. I'm going to use this to run through some ebooks I have in .txt
def parse(s):
listOfWords = []
i = 0
final = s.lower()
final = final.split()
while i < len(final):
if len(final[i]) >= 4:
listOfWords.append(final[i])
i = i + 1
return listOfWordsCurrent Output
['lincoln’s', 'silly,', 'flat', 'dishwatery', 'utterances', 'chicago', 'times,', '1863']
Desired Output
["lincoln", "silly", "flat", "dishwatery", "utterances", "chicago", "times"]
Something like that is perfect for regular expressions. re.sub takes as input a regular expression that defines what to match, a string (or a function) to decide what to replace what is matched with, and a string to do all this matching and replacing on. As an example:
import re
string = "lincoln's silly flat dishwatery utterances chicago times 1863"
print re.sub(r'[^a-zA-Z ]', '', string).split()
# ouputs:
# ['lincolns', 'silly', 'flat', 'dishwatery', 'utterances', 'chicago', 'times']
This will replace all characters that are not between lowercase a-z, uppercase A-Z, or a space, with nothing -- essentially removing them.
Use isAlpha() on each character.
You can specify everything that you need not remove in the negated character clas.
re.sub(r'[^\w'+removelist+']', '',mystring)
Test
>>> import re
>>> removelist = "=."
>>> mystring = "asdf1234=.!@#$"
>>> re.sub(r'[^\w'+removelist+']', '',mystring)
'asdf1234=.'
Here the removelist variable is a string which contains the list of all characters you need to exclude from the removal.
What does negated character class means
When the ^ is moved into the character class it does not acts as an anchor where as it negates the character class.
That is ^ in inside a character class say like [^abc] it negates the meaning of the character class.
For example [abc] will match a b or c where as [^abc] will not match a b or c. Which can also be phrased as anything other than a b or c
re.sub(r'[^a-zA-Z0-9=]', '',mystring)
You can add whatever you want like _ whichever you want to save.
Given
s = '@#24A-09=wes()&8973o**_##me' # contains letters 'Awesome'
You can filter out non-alpha characters with a generator expression:
result = ''.join(c for c in s if c.isalpha())
Or filter with filter:
result = ''.join(filter(str.isalpha, s))
Or you can substitute non-alpha with blanks using re.sub:
import re
result = re.sub(r'[^A-Za-z]', '', s)
A solution using RegExes is quite easy here:
import re
newstring = re.sub(r"[^a-zA-Z]+", "", string)
Where string is your string and newstring is the string without characters that are not alphabetic. What this does is replace every character that is not a letter by an empty string, thereby removing it. Note however that a RegEx may be slightly overkill here.
A more functional approach would be:
newstring = "".join(filter(str.isalpha, string))
Unfortunately you can't just call stron a filterobject to turn it into a string, that would look much nicer...
Going the pythonic way it would be
newstring = "".join(c for c in string if c.isalpha())