Yes, you can use a regular expression for this:
import re
output = re.sub(r'\d+', '', '123hello 456world')
print output # 'hello world'
Answer from Martin Konecny on Stack OverflowYes, you can use a regular expression for this:
import re
output = re.sub(r'\d+', '', '123hello 456world')
print output # 'hello world'
str.translate should be efficient.
In [7]: 'hello467'.translate(None, '0123456789')
Out[7]: 'hello'
To compare str.translate against re.sub:
In [13]: %%timeit r=re.compile(r'\d')
output = r.sub('', my_str)
....:
100000 loops, best of 3: 5.46 µs per loop
In [16]: %%timeit pass
output = my_str.translate(None, '0123456789')
....:
1000000 loops, best of 3: 713 ns per loop
Try something like below:
import string
import re
import nltk
from nltk.tokenize import TweetTokenizer
tweet = "first think another Disney movie, might good, it's kids movie. watch it, can't help enjoy it. ages love movie. first saw movie 10 8 years later still love it! Danny Glover superb could play"
def clean_text(text):
# remove numbers
text_nonum = re.sub(r'\d+', '', text)
# remove punctuations and convert characters to lower case
text_nopunct = "".join([char.lower() for char in text_nonum if char not in string.punctuation])
# substitute multiple whitespace with single whitespace
# Also, removes leading and trailing whitespaces
text_no_doublespace = re.sub('\s+', ' ', text_nopunct).strip()
return text_no_doublespace
cleaned_tweet = clean_text(tweet)
tt = TweetTokenizer()
print(tt.tokenize(cleaned_tweet))
output:
['first', 'think', 'another', 'disney', 'movie', 'might', 'good', 'its', 'kids', 'movie', 'watch', 'it', 'cant', 'help', 'enjoy', 'it', 'ages', 'love', 'movie', 'first', 'saw', 'movie', 'years', 'later', 'still', 'love', 'it', 'danny', 'glover', 'superb', 'could', 'play']
# Function for removing Punctuation from Text and It gives total no.of punctuation removed also
# Input: Function takes Existing fie name and New file name as string i.e 'existingFileName.txt' and 'newFileName.txt'
# Return: It returns two things Punctuation Free File opened in read mode and a punctuation count variable.
from nltk.tokenize import word_tokenize
import string
def removePunctuation(tokenizeSampleText, newFileName):
stringPun = list(string.punctuation)
with open(tokenizeSampleText, "r") as existingFile:
tokenize_existingFile = word_tokenize(existingFile.read())
count_pun = 0
words = []
for word in tokenize_existingFile:
if word in stringPun:
count_pun += 1
else:
words.append(word)
with open(newFileName, "w+") as puncRemovedFile:
puncRemovedFile.write(" ".join(words))
with open(newFileName, "r") as file:
yield file.read(), count_pun
punRemoved, punCount = removePunctuation(
"Macbeth.txt", "Macbeth-punctuationRemoved.txt"
)
print(f"Total Punctuation : {punCount}")
print(punRemoved)
Delete digits in Python (Regex) - Stack Overflow
python - Removing numbers from string - Stack Overflow
Removing everything but letters, numbers, and spaces from string
python - Removing punctuation/numbers from text problem - Stack Overflow
Add a space before the \d+.
>>> s = "This must not b3 delet3d, but the number at the end yes 134411"
>>> s = re.sub(" \d+", " ", s)
>>> s
'This must not b3 delet3d, but the number at the end yes '
Edit: After looking at the comments, I decided to form a more complete answer. I think this accounts for all the cases.
s = re.sub("^\d+\s|\s\d+\s|\s\d+$", " ", s)
Try this:
"\b\d+\b"
That'll match only those digits that are not part of another word.
Would this work for your situation?
>>> s = '12abcd405'
>>> result = ''.join([i for i in s if not i.isdigit()])
>>> result
'abcd'
This makes use of a list comprehension, and what is happening here is similar to this structure:
no_digits = []
# Iterate through the string, adding non-numbers to the no_digits list
for i in s:
if not i.isdigit():
no_digits.append(i)
# Now join all elements of the list with '',
# which puts all of the characters together.
result = ''.join(no_digits)
As @AshwiniChaudhary and @KirkStrauser point out, you actually do not need to use the brackets in the one-liner, making the piece inside the parentheses a generator expression (more efficient than a list comprehension). Even if this doesn't fit the requirements for your assignment, it is something you should read about eventually :) :
>>> s = '12abcd405'
>>> result = ''.join(i for i in s if not i.isdigit())
>>> result
'abcd'
And, just to throw it in the mix, is the oft-forgotten str.translate which will work a lot faster than looping/regular expressions:
For Python 2:
from string import digits
s = 'abc123def456ghi789zero0'
res = s.translate(None, digits)
# 'abcdefghizero'
For Python 3:
from string import digits
s = 'abc123def456ghi789zero0'
remove_digits = str.maketrans('', '', digits)
res = s.translate(remove_digits)
# 'abcdefghizero'
So basically I have to remove all instances from a string that isn't a letter, number, or space... So far I'm able to remove everything that isn't a letter or space, however, I'm not sure how I would keep numbers for the way I did it.
Any help would be greatly appreciated. My code is as follows:
def remove_punctuation(word):
new_word = ""
for i in range(len(word)):
for j in range(52):
if word[i] == alpha[j]:
new_word += alpha[j]
return new_word
alpha = [' ', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm',
'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z', 'A', 'B', 'C',
'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S',
'T', 'U', 'V', 'W', 'X', 'Y', 'Z']
print remove_punctuation("He$rs*hey {Bea%rs")Obviously I can't add every number in the list, lol.
Change
for word in word_list:
word = punctuation.sub("", word)
to
word_list = [punctuation.sub("", word) for word in word_list]
Assignment to word in the for-loop above, simply changes the value referenced by this temporary variable. It does not alter word_list.
You're not updating your word list. Try
for i, word in enumerate(word_list):
word_list[i] = punctuation.sub("", word)
Remember that although word starts off as a reference to the string object in the word_list, assignment rebinds the name word to the new string object returned by the sub function. It doesn't change the originally referenced object.
>>> import re
>>> re.sub("[^0-9]", "", "sdkjh987978asd098as0980a98sd")
'987978098098098'
>>> # or
>>> re.sub(r"\D", "", "sdkjh987978asd098as0980a98sd")
'987978098098098'
Not sure if this is the most efficient way, but:
>>> ''.join(c for c in "abc123def456" if c.isdigit())
'123456'
The ''.join part means to combine all the resulting characters together without any characters in between. Then the rest of it is a generator expression, where (as you can probably guess) we only take the parts of the string that match the condition isdigit.
Take a look at the other tokenizing options that nltk provides here. For example, you can define a tokenizer that picks out sequences of alphanumeric characters as tokens and drops everything else:
from nltk.tokenize import RegexpTokenizer
tokenizer = RegexpTokenizer(r'\w+')
tokenizer.tokenize('Eighty-seven miles to go, yet. Onward!')
Output:
['Eighty', 'seven', 'miles', 'to', 'go', 'yet', 'Onward']
You do not really need NLTK to remove punctuation. You can remove it with simple python. For strings:
import string
s = '... some string with punctuation ...'
s = s.translate(None, string.punctuation)
Or for unicode:
import string
translate_table = dict((ord(char), None) for char in string.punctuation)
s.translate(translate_table)
and then use this string in your tokenizer.
P.S. string module have some other sets of elements that can be removed (like digits).