A better question would have been what are those ("()", "'",",") in the ngrams output?

>>> from nltk import ngrams
>>> from nltk import word_tokenize

# Split a sentence into a list of "words"
>>> word_tokenize("This is a foo bar sentence")
['This', 'is', 'a', 'foo', 'bar', 'sentence']
>>> type(word_tokenize("This is a foo bar sentence"))
<class 'list'>

# Extract bigrams.
>>> list(ngrams(word_tokenize("This is a foo bar sentence"), 2))
[('This', 'is'), ('is', 'a'), ('a', 'foo'), ('foo', 'bar'), ('bar', 'sentence')]

# Okay, so the output is a list, no surprise.
>>> type(list(ngrams(word_tokenize("This is a foo bar sentence"), 2)))
<class 'list'>

But what type is ('This', 'is')?

>>> list(ngrams(word_tokenize("This is a foo bar sentence"), 2))[0]
('This', 'is')
>>> first_thing_in_output = list(ngrams(word_tokenize("This is a foo bar sentence"), 2))[0]
>>> type(first_thing_in_output)
<class 'tuple'>

Ah, it's a tuple, see https://realpython.com/python-lists-tuples/

What happens when you print a tuple?

>>> print(first_thing_in_output)
('This', 'is')

What happens if you convert them into a str()?

>>> print(str(first_thing_in_output))
('This', 'is')

But I want the output This is instead of ('This', 'is'), so I will use the str.join() function, see https://www.geeksforgeeks.org/join-function-python/:

>>> print(' '.join((first_thing_in_output)))
This is

Now this is a good point to really go through the tutorial of basic Python types to understand what is happening. Additionally, it'll be good to understand how "container" types work too, e.g. https://github.com/usaarhat/pywarmups/blob/master/session2.md

Going through the original post, there are quite some issues with the code.

I guess the goal of the code is to:

• Tokenize the text and remove stopwords
• Extract ngrams (without stopwords)
• Print out their string forms and their counts

The tricky part is the stopwords.words('english') does not contain punctuation, so you'll end up with strange ngrams that contains punctuations:

from nltk import word_tokenize
from nltk.util import ngrams
from nltk.corpus import stopwords

text = '''The pure amnesia of her face,
newborn. I looked so far into her that, for a while, looked so far into her that, for a while  looked so far into her that, for a while looked so far into her that, for a while the visual 
held no memory. Little by little, I returned to myself, waking to nurse the visual held no  memory. Little by little, I returned to myself, waking to nurse
'''

stoplist = set(stopwords.words('english'))

tokens = [token for token in nltk.word_tokenize(text) if token not in stoplist]

list(ngrams(tokens, 2))

[out]:

[('The', 'pure'),
 ('pure', 'amnesia'),
 ('amnesia', 'face'),
 ('face', ','),
 (',', 'newborn'),
 ('newborn', '.'),
 ('.', 'I'),
 ('I', 'looked'),
 ('looked', 'far'),
 ('far', ','),
 (',', ','), ...]

Perhaps you would like to extend the stoplist with punctuations, e.g.

from string import punctuation
from nltk import word_tokenize
from nltk.util import ngrams
from nltk.corpus import stopwords

text = '''The pure amnesia of her face,
newborn. I looked so far into her that, for a while, looked so far into her that, for a while  looked so far into her that, for a while looked so far into her that, for a while the visual 
held no memory. Little by little, I returned to myself, waking to nurse the visual held no  memory. Little by little, I returned to myself, waking to nurse
'''

stoplist = set(stopwords.words('english') + list(punctuation))

tokens = [token for token in nltk.word_tokenize(text) if token not in stoplist]

list(ngrams(tokens, 2))

[out]:

[('The', 'pure'),
 ('pure', 'amnesia'),
 ('amnesia', 'face'),
 ('face', 'newborn'),
 ('newborn', 'I'),
 ('I', 'looked'),
 ('looked', 'far'),
 ('far', 'looked'),
 ('looked', 'far'), ...]

Then you realized that tokens like I should be a stopword but still exists in your list of ngrams. It's because the list from stopwords.words('english') are lowercased, e.g.

>>> stopwords.words('english')

[out]:

['i',
 'me',
 'my',
 'myself',
 'we',
 'our',
 'ours',
 'ourselves',
 'you',
 "you're", ...]

So when you're checking whether a token is in the stoplist, you should also lowercase the token. (Avoid lowercasing the sentence before word_tokenize because word_tokenize may take cues from capitalization). Thus:

from string import punctuation
from nltk import word_tokenize
from nltk.util import ngrams
from nltk.corpus import stopwords

text = '''The pure amnesia of her face,
newborn. I looked so far into her that, for a while, looked so far into her that, for a while  looked so far into her that, for a while looked so far into her that, for a while the visual 
held no memory. Little by little, I returned to myself, waking to nurse the visual held no  memory. Little by little, I returned to myself, waking to nurse
'''

stoplist = set(stopwords.words('english') + list(punctuation))

tokens = [token for token in nltk.word_tokenize(text) if token.lower() not in stoplist]

list(ngrams(tokens, 2))

[out]:

[('pure', 'amnesia'),
 ('amnesia', 'face'),
 ('face', 'newborn'),
 ('newborn', 'looked'),
 ('looked', 'far'),
 ('far', 'looked'),
 ('looked', 'far'),
 ('far', 'looked'),
 ('looked', 'far'),
 ('far', 'looked'), ...]

Now the ngrams looks like it's achieving the objectives:

• Tokenize the text and remove stopwords
• Extract ngrams (without stopwords)

Then on the last part where you want to print out the ngrams to a file in sorted order, you could actually use the Freqdist.most_common() which will list in descending order, e.g.

from string import punctuation
from nltk import word_tokenize
from nltk.util import ngrams
from nltk.corpus import stopwords
from nltk import FreqDist

text = '''The pure amnesia of her face,
newborn. I looked so far into her that, for a while, looked so far into her that, for a while  looked so far into her that, for a while looked so far into her that, for a while the visual 
held no memory. Little by little, I returned to myself, waking to nurse the visual held no  memory. Little by little, I returned to myself, waking to nurse
'''

stoplist = set(stopwords.words('english') + list(punctuation))

tokens = [token for token in nltk.word_tokenize(text) if token.lower() not in stoplist]

FreqDist(ngrams(tokens, 2)).most_common()

[out]:

[(('looked', 'far'), 4),
 (('far', 'looked'), 3),
 (('visual', 'held'), 2),
 (('held', 'memory'), 2),
 (('memory', 'Little'), 2),
 (('Little', 'little'), 2),
 (('little', 'returned'), 2),
 (('returned', 'waking'), 2),
 (('waking', 'nurse'), 2),
 (('pure', 'amnesia'), 1),
 (('amnesia', 'face'), 1),
 (('face', 'newborn'), 1),
 (('newborn', 'looked'), 1),
 (('far', 'visual'), 1),
 (('nurse', 'visual'), 1)]

(See also: Difference between Python's collections.Counter and nltk.probability.FreqDist)

Final finally, printing it out to file, you should really use a context manager, http://eigenhombre.com/introduction-to-context-managers-in-python.html

with open('bigrams-list.tsv', 'w') as fout:
    for bg, count in FreqDist(ngrams(tokens, 2)).most_common():
        print('\t'.join([' '.join(bg), str(count)]), end='\n', file=fout)

[bigrams-list.tsv]:

looked far  4
far looked  3
visual held 2
held memory 2
memory Little   2
Little little   2
little returned 2
returned waking 2
waking nurse    2
pure amnesia    1
amnesia face    1
face newborn    1
newborn looked  1
far visual  1
nurse visual    1

Food for thought

Now you see this strange bigram Little little, does it make sense?

It's a by-product of removing by from

Little by little

So now, depending on what's the ultimate task for the ngrams you've extracted, you might not really want to remove stopwords from the list.

Take a look at the other tokenizing options that nltk provides here. For example, you can define a tokenizer that picks out sequences of alphanumeric characters as tokens and drops everything else:

from nltk.tokenize import RegexpTokenizer

tokenizer = RegexpTokenizer(r'\w+')
tokenizer.tokenize('Eighty-seven miles to go, yet.  Onward!')

Output:

['Eighty', 'seven', 'miles', 'to', 'go', 'yet', 'Onward']

Regular expressions can be used to create a simple tokenizer and normalizer:

from __future__ import annotations
import re

def tokens(text: str) -> list(str):
    "List all the word tokens in a text."
    return re.findall('[\w]+', text.lower())

assert tokens("To be, or not to be, that is the question:") == ['to', 'be', 'or', 'not', 'to', 'be', 'that', 'is', 'the', 'question']

Otherwise, use an established library like spaCy to generate a list of tokens.

you can use regexp_tokenize from nltk where you can choose a regular expression to define seps

from nltk import regexp_tokenize
title = 'Tesla S&P Debut Comes All at Once'
tokens = regexp_tokenize(title, pattern=r"\s|[\.,;']", gaps=True)

print(tokens)

['Tesla', 'S&P', 'Debut', 'Comes', 'All', 'at', 'Once']

If you're using python2.7, install it from terminal running : pip install nltk Otherwise, for python 3, just run pip3 install nltk

If you meet this kind of error

just run nltk.download('punkt')(or whatever you see in the error, here I got punkt) from our python console

You shouldn't see any importError now

Change the indexing on this line from pure slicing to iloc:

document = re.sub(r'\W', ' ', str(X.iloc[sen]))

Solution 1: Tokenize and strip punctuation off the tokens

>>> from nltk import word_tokenize
>>> import string
>>> punctuations = list(string.punctuation)
>>> punctuations
['!', '"', '#', '$', '%', '&', "'", '(', ')', '*', '+', ',', '-', '.', '/', ':', ';', '<', '=', '>', '?', '@', '[', '\\', ']', '^', '_', '`', '{', '|', '}', '~']
>>> punctuations.append("''")
>>> sent = '''He said,"that's it."'''
>>> word_tokenize(sent)
['He', 'said', ',', "''", 'that', "'s", 'it', '.', "''"]
>>> [i for i in word_tokenize(sent) if i not in punctuations]
['He', 'said', 'that', "'s", 'it']
>>> [i.strip("".join(punctuations)) for i in word_tokenize(sent) if i not in punctuations]
['He', 'said', 'that', 's', 'it']

Solution 2: remove punctuation then tokenize

>>> import string
>>> string.punctuation
'!"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~'
>>> sent = '''He said,"that's it."'''
>>> " ".join("".join([" " if ch in string.punctuation else ch for ch in sent]).split())
'He said that s it'
>>> " ".join("".join([" " if ch in string.punctuation else ch for ch in sent]).split()).split()
['He', 'said', 'that', 's', 'it']

I finally found this to work well:

from unidecode import unidecode

corpus = "".join(words)
sent = []
sent.append(unidecode("".join(corpus)))

I never worked with nltk before. There could be a better solution too. In my code snippet I am simply doing the following:

1. Reading a file that needs to be checked for non-english/english words named as frequencyList.txt to a variable named as lines.

2. Then I am opening a new file named as eng_words_only.txt. This file will contain the english words only. Initially this file will be empty, later after executing the script this file will contain all the English language words present in frequencyList.txt

3. Now for every word in frequencyList.txt I check if it is also present in wordnet. If the word is present then I write this word to the eng_words_only.txt file, else I do nothing. Please see I am using wordnet just for demo purpose. It doesn't contains all the English language words!

Code:

from nltk.corpus import wordnet

fList = open("frequencyList.txt","r")#Read the file
lines = fList.readlines()

eWords = open("eng_words_only.txt", "a")#Open file for writing

for w in lines:
    if not wordnet.synsets(w):#Comparing if word is non-English
        print 'not '+w
    else:#If word is an English word
        print 'yes '+w
        eWords.write(w)#Write to file 
        
eWords.close()#Close the file

Testing: I first created a file named as frequencyList.txt with the following contents:

cat 
meoooow 
mouse

then upon executing the code snippet you'll see the following output in the console:

not cat

not meoooow

yes mouse

Then a file will be created eng_words_only.txt which contains only the words that were supposed to be of the English language. The eng_words_only.txt will contain only mouse word. You may notice that cat is an English word but it is still not in the eng_words_only.txt file. This is the reason why you should use a good source instead of wordnet. Please note: The python script file and the frequencyList.txt should be in the same directory. Also, instead of frequencyList.txt you can use any of your file that you want to check/investigate. In that case don't forget to change the files names in the code snippet too.

Second Solution: Although you didn't ask for it but still there is an other way too to do this English word test.

Here is the code: Here the wordlist-eng.txt is the file which contains the English words. You have to keep

wordlist-eng.txt, frequencyList.txt and the python script in the same directory.

with open("wordlist-eng.txt") as word_file:
    english_words = set(word.strip().lower() for word in word_file)

fList = open("frequencyList.txt","r")
lines = fList.readlines()
fList.close()

eWords = open("eng_words_only.txt", "a")

for w in lines:
    if w.strip().lower() in english_words:
        eWords.write(w)
    else: pass
eWords.close()

After executing the script the eng_words_only.txt will contain all the English words that were present in frequencyList.txt file.

I hope this was helpful.

Based on NLTK documentation here:

A tokenizer that divides a string into substrings by splitting on the specified string (defined in subclasses).

So what it does, it is only dividing a string into substring. If you want to filter words that are not in nltk.corpus.words() you can download the words one time:

import nltk
nltk.download('words')

and then after that:

import nltk
from nltk.corpus import words

sentence = "The word hello is gramatically correct but henlo is not"
sentence_tokenised = nltk.tokenize.word_tokenize(sentence)

output = list(filter(lambda x: x in words.words(), sentence_tokenised))

output:

['The', 'word', 'hello', 'is', 'correct', 'but', 'is', 'not']

The following function performs all things you have mentioned.

import nltk
from nltk.tokenize import RegexpTokenizer
from nltk.stem import WordNetLemmatizer,PorterStemmer
from nltk.corpus import stopwords
import re
lemmatizer = WordNetLemmatizer()
stemmer = PorterStemmer() 

 def preprocess(sentence):
    sentence=str(sentence)
    sentence = sentence.lower()
    sentence=sentence.replace('{html}',"") 
    cleanr = re.compile('<.*?>')
    cleantext = re.sub(cleanr, '', sentence)
    rem_url=re.sub(r'http\S+', '',cleantext)
    rem_num = re.sub('[0-9]+', '', rem_url)
    tokenizer = RegexpTokenizer(r'\w+')
    tokens = tokenizer.tokenize(rem_num)  
    filtered_words = [w for w in tokens if len(w) > 2 if not w in stopwords.words('english')]
    stem_words=[stemmer.stem(w) for w in filtered_words]
    lemma_words=[lemmatizer.lemmatize(w) for w in stem_words]
    return " ".join(filtered_words)


df['cleanText']=df['Text'].map(lambda s:preprocess(s))