Your .filter returns an error because it is the sql filter function (expecting a BooleanType() column) on dataframes not the filter function on RDDs. If you want to use the RDD one, just add .rdd:

small_DF.rdd.filter(lambda x: any(word in x.text for word in test_list))

You don't have to use a UDF, you can use regular expressions in pyspark with .rlike on your column "text":

from pyspark.sql import HiveContext
hc = HiveContext(sc)
import pyspark.sql.functions as psf

words = [x.lower() for x in ['starbucks', 'Nvidia', 'IBM', 'Dell']]
data = [['i love Starbucks'],['dell laptops rocks'],['help me I am stuck!']]
df = hc.createDataFrame(data).toDF('text')
df.filter(psf.lower(df.text).rlike('|'.join(words)))

You are getting the error :

IndexError: list index out of range

because you are trying to access the index 2 of a list (the result of your split) which may not exist if some rows in your dataset only have Date or Date and Label or are empty or may have formatting issues.

In your lambda function you may take advantage of short-circuiting in python to first check if there are at least 3 elements (i.e. an index of 2 is possible using len(x.split(",")) >=3 instead of (x.split(",")[2] != None)) before trying to access this index.

This may be written as:

countC = rdd.filter(lambda x: x.startswith('201') and (len(x.split(",")) >=3) and (len(x.split(",")[2]) > 0))

Let me know if this works for you.

The lambda function is pure python, so something like below would work

table2 = table1.filter(lambda x: "TEXT" in x[12])

You can use filter with a lambda expression to check that the third element of each tuple pair are the same such as:

l = [((111, u'BB', u'A'), (444, u'BB', u'A')),
     ((222, u'BB', u'A'), (888, u'BB', u'A')),
     ((333, u'BB', u'B'), (999, u'BB', u'A'))]

rdd = sc.parallelize(l)
rdd = rdd.filter(lambda x: x[0][2] == x[1][2])
result = rdd.collect()
print result

>>> [((111, u'BB', u'A'), (444, u'BB', u'A')), ((222, u'BB', u'A'), (888, u'BB', u'A'))]

To answer your follow up comment, remember, a lambda is just a function, if you have more complex logic, you can just write it out as a function. You could do something like:

def do_stuff(x):
    if (x[0][2] == 'C') or (x[1][2] == 'C'):
        return x     
    else:
        if x[0][2] == x[1][2]: return x
    return None

rdd = rdd.map(do_stuff).filter(lambda x: x is not None)

res = rdd.collect()