The errors you have (besides my other comments) are because you're modifying a list while iterating over it. But you take the length of the list at the start, thus, after you've removed some elements, you cannot access the last positions.

I would do it this way:

words = ['a', 'b', 'a', 'c', 'd']
stopwords = ['a', 'c']
for word in list(words):  # iterating on a copy since removing will mess things up
    if word in stopwords:
        words.remove(word)

An even more pythonic way using list comprehensions:

new_words = [word for word in words if word not in stopwords]

This is one way to do it:

query = 'What is hello'
stopwords = ['what', 'who', 'is', 'a', 'at', 'is', 'he']
querywords = query.split()

resultwords  = [word for word in querywords if word.lower() not in stopwords]
result = ' '.join(resultwords)

print(result)

I noticed that you want to also remove a word if its lower-case variant is in the list, so I've added a call to lower() in the condition check.

You need to split each phrase into words and re-join the words into phrases after filtering out those in stop.

[' '.join(w for w in p.split() if w not in stop) for p in abc]

This outputs:

['issues in performance', 'how are you doing', 'hey my name is abc, what is your name', 'pleased', 'compliance installed']

In natural language processing, the term for excluding common words is called "stop words".

Do you want to preserve the order and count of each word, or do you just want the set of words that appear on the page?

If you just want the set of words that appear on the page, using sets is probably the way to go. Something like the following might work:

# It's probably more common to define your STOP_WORDS in a file and then read
# them into your data structure to keep things simple for large numbers of those
# words.
STOP_WORDS = set([
    'het',
    'om'
])

all_words = set()
for item in g_data:
    all_words |= set(item.text.split())
all_words -= STOP_WORDS
print all_words

If, on the other hand, you care about the order, you could just refrain from adding stop words to your list.

words_in_order = []
for item in g_data:
    words_from_span = item.text.split()
    # You might want to break this out into its own function for modularity.
    for word in words_from_span:
        if word not in STOP_WORDS:
            words_in_order.append(word)
print words_in_order

If you don't care about order but you want frequency, you could create a dict (or defaultdict for convenience) of words to counts.

from collections import defaultdict
word_counts = defaultdict(int)
for item in g_data:
    # You might want to break this out into its own function for modularity.
    for word in item.text.split():
        if word not in STOP_WORDS:
            word_counts[word] += 1
for word, count in word_counts.iteritems():
    print '%s: %d' % (word, count)

Using list comprehension we can easily accomplish this goal. Also using in we can check if a key word is in any elements in the given list.

list1= [ 'one', 'one-test', 'two', 'two-test', 'three', 'three-test']
newList = [elements for elements in list1 if '-test' not in elements]

output

['one', 'two', 'three']

The problem is the for loop of python.

For example: if you do like this:

arr = range(1, 10)
for x in arr:
    print x
    arr.remove(x)

Then you will see that not all item in arr was removed.

In your case, we can do like this:

newDoc = [ word for word in doc if len(word) >= 3 ]

Welcome to python.

You currently have to loop over all exclude words to check if it is equal to the current word. If your exclude was a set, you could just do if j in exclude. This change alone should speed up your code a lot (how much depends on how large your list of exclude words actually is).

In addition, you could simplify getting the exclude words:

from collections import Counter
form itertools import takewhile

THRESHOLD = 25

words = Counter(whatever_generates_your_words())
exclude = set(t[0] for t in takewhile(lambda t: t[1] <= THRESHOLD,
                                      reversed(words.most_common())))

This uses the fact that collections.Counter.most_common is sorted by frequency, so reversing it starts with the least common words. itertools.takewhile stops taking when the condition is no longer true, so you don't need to go through all words in the Counter.

For the filtering you should probably use some list comprehensions. All three of the following functions are doing the same thing:

from itertools import filterfalse

def remove_nonfrequent(note):
    sentences = []
    for sentence in note.split('. '):
        sentence_ = []
        for word in sentence:
            if word not in exclude:
                sentence_.append(word)
        if sentence:
            sentences.append(" ".join(sentence))
    return ". ".join(sentences)

def remove_nonfrequent(note):
    return ". ".join(" ".join(filterfalse(exclude.__contains__,
                              sentence.split(" ")))
                     for sentence in note.split('. '))
                     if sentence)

def remove_nonfrequent(note):        
    return ". ".join(" ".join(word for word in sentence.split(" ")
                              if word not in exclude)
                     for sentence in note.split('. ') if sentence)

I personally prefer the last one, as it is the most readable one to me.

Note that Python has an official style-guide, PEP8, which recommends using 4 spaces as indentation and surrounding operators with spaces.

Since you are using this function with pandas later, a different approach might actually be faster. There is a pd.Series.replace method that can take a dictionary of replacements. You can use it like this:

from itertools import repeat

replacements = dict(zip((fr'\b{word}\b' for word in exclude), repeat("")))
df.NOTES.replace(replacements, regex=True, inplace=True)
df.NOTES.replace({r' +': ' ', r' +\.': '.'}, regex=True, inplace=True)

The first replace does all the word replacements (using \b to ensure only complete words are replaced), while the latter one fixes multiple spaces and spaces before a period.

You can use a regular expression with word boundaries.

pattern = re.compile('|'.join(rf'\b{re.escape(w)}\b' for w in word_list))
def remove_w(text):
    return pattern.sub('', text)

Alternatively, use str.split to separate into words delimited by spaces, remove the words exactly matching one of those in the set, then join it back together.

def remove_w(text):
    return ' '.join(w for w in text.split() if w not in word_list)