In versions of Pandas > 0.19.0, DataFrame.to_json has a parameter, lines, that will write out JSONL format.

Given that, a more succinct version of your solution might look like this:

import pandas as pd

data = [{'label': 'DRUG', 'pattern': 'aspirin'},
        {'label': 'DRUG', 'pattern': 'trazodone'},
        {'label': 'DRUG', 'pattern': 'citalopram'}]
df = pd.DataFrame(data)

# Wrap pattern column in a dictionary
df["pattern"] = df.pattern.apply(lambda x: {"lower": x})

# Output in JSONL format
print(df.to_json(orient='records', lines=True))

Output:

{"label":"DRUG","pattern":{"lower":"aspirin"}}
{"label":"DRUG","pattern":{"lower":"trazodone"}}
{"label":"DRUG","pattern":{"lower":"citalopram"}}

The pd.read_json() function in the pandas library is used to read JSON data into a DataFrame. When reading a JSON Lines (JSONL) file, where each line represents a separate JSON object, you can use the lines=True parameter to properly parse the file, treating each line in the file as a separate JSON object.

df = pd.read_json("test.jsonl", lines=True)

If the file is large, you can also pass the chunksize to manipulate it in chunks.

To create newline-delimited json from a dataframe df, run the following

df.to_json("path/to/filename.json",
           orient="records",
           lines=True)

Pay close attention to those optional keyword args! The lines option was added in pandas 0.19.0.

In newer versions of pandas (0.20.0+, I believe), this can be done directly:

df.to_json('temp.json', orient='records', lines=True)

Direct compression is also possible:

df.to_json('temp.json.gz', orient='records', lines=True, compression='gzip')

Note: Line separated json is now supported in read_json (since 0.19.0):

In [31]: pd.read_json('{"a":1,"b":2}\n{"a":3,"b":4}', lines=True)
Out[31]:
   a  b
0  1  2
1  3  4

or with a file/filepath rather than a json string:

pd.read_json(json_file, lines=True)

It's going to depend on the size of you DataFrames which is faster, but another option is to use str.join to smash your multi line "JSON" (Note: it's not valid json), into valid json and use read_json:

In [11]: '[%s]' % ','.join(test.splitlines())
Out[11]: '[{"a":1,"b":2},{"a":3,"b":4}]'

For this tiny example this is slower, if around 100 it's the similar, signicant gains if it's larger...

In [21]: %timeit pd.read_json('[%s]' % ','.join(test.splitlines()))
1000 loops, best of 3: 977 µs per loop

In [22]: %timeit l=[ json.loads(l) for l in test.splitlines()]; df = pd.DataFrame(l)
1000 loops, best of 3: 282 µs per loop

In [23]: test_100 = '\n'.join([test] * 100)

In [24]: %timeit pd.read_json('[%s]' % ','.join(test_100.splitlines()))
1000 loops, best of 3: 1.25 ms per loop

In [25]: %timeit l = [json.loads(l) for l in test_100.splitlines()]; df = pd.DataFrame(l)
1000 loops, best of 3: 1.25 ms per loop

In [26]: test_1000 = '\n'.join([test] * 1000)

In [27]: %timeit l = [json.loads(l) for l in test_1000.splitlines()]; df = pd.DataFrame(l)
100 loops, best of 3: 9.78 ms per loop

In [28]: %timeit pd.read_json('[%s]' % ','.join(test_1000.splitlines()))
100 loops, best of 3: 3.36 ms per loop

Note: of that time the join is surprisingly fast.