You will go crazy if you try to parse a json file line by line. The json module has helper methods to read file objects directly or strings i.e. the load and loads methods. load takes a file object (as shown below) for a file that contains json data, while loads takes a string that contains json data.

Option 1: - Preferred

import json
with open('test.json', 'r') as jf:
    weatherData = json.load(jf)
    print weatherData

Option 2:

import json
with open('test.json', 'r') as jf:
    weatherData = json.loads(jf.read())
    print weatherData

If you are looking for higher performance json parsing check out ujson

You have a JSON Lines format text file. You need to parse your file line by line:

import json

data = []
with open('file') as f:
    for line in f:
        data.append(json.loads(line))

Each line contains valid JSON, but as a whole, it is not a valid JSON value as there is no top-level list or object definition.

Note that because the file contains JSON per line, you are saved the headaches of trying to parse it all in one go or to figure out a streaming JSON parser. You can now opt to process each line separately before moving on to the next, saving memory in the process. You probably don't want to append each result to one list and then process everything if your file is really big.

If you have a file containing individual JSON objects with delimiters in-between, use How do I use the 'json' module to read in one JSON object at a time? to parse out individual objects using a buffered method.

Get rid of all of the backslashes and all of the "Pythonic" quoting in the settings file. Works fine if the file is just:

{
  "user":"username",
  "password":"passwd"
}

Note also that JSON strings are quoted with double quotes, not single quotes. See JSON spec here:

http://www.json.org/

def main():
   ... # changeKeyValueCode
   writeFile(data)
 
def writeFile(data):
   with open('new_file.json', 'a') as f:
       json.dump(data, f)

with open('new_file.json', 'a')

open file with (a), it will search the file if found append data to the end, else it will create empty file and then append data.

You are dealing with ndjson(Newline delimited JSON) data format.

You have to read the whole data string, split it by lines and parse each line as a JSON object resulting in a list of JSONs:

def parse_ndjson(data):
    return [json.loads(l) for l in data.splitlines()]


with open('C:\\Users\\test.json', 'r', encoding="utf8") as handle:
    data = handle.read()
    dicts = parse_ndjson(data)

for d in dicts:
    new_d = my_function(d)
    print("New dict", new_d)

Just read each line and construct a json object at this time:

with open(file_path) as f:
    for line in f:
        j_content = json.loads(line)

This way, you load proper complete json object (provided there is no \n in a json value somewhere or in the middle of your json object) and you avoid memory issue as each object is created when needed.

There is also this answer.:

https://stackoverflow.com/a/7795029/671543

Why don't you make it a dictionary and set variables then use the json library to make it into json

import json
json_serial = "123"
my_json = {
    'settings': {
        "serial": json_serial,
        "status": '2',
        "ersion": '3',
    },
    'config': {
        'active': '4',
        'version': '5'
    }
}
print(json.dumps(my_json))

Use the standard library :

json_data = json.loads(your_line)

# Usage exemple
for feature in json_data['features']:
    print feature['id']

Note: Line separated json is now supported in read_json (since 0.19.0):

CopyIn [31]: pd.read_json('{"a":1,"b":2}\n{"a":3,"b":4}', lines=True)
Out[31]:
   a  b
0  1  2
1  3  4

or with a file/filepath rather than a json string:

Copypd.read_json(json_file, lines=True)

It's going to depend on the size of you DataFrames which is faster, but another option is to use str.join to smash your multi line "JSON" (Note: it's not valid json), into valid json and use read_json:

CopyIn [11]: '[%s]' % ','.join(test.splitlines())
Out[11]: '[{"a":1,"b":2},{"a":3,"b":4}]'

For this tiny example this is slower, if around 100 it's the similar, signicant gains if it's larger...

CopyIn [21]: %timeit pd.read_json('[%s]' % ','.join(test.splitlines()))
1000 loops, best of 3: 977 µs per loop

In [22]: %timeit l=[ json.loads(l) for l in test.splitlines()]; df = pd.DataFrame(l)
1000 loops, best of 3: 282 µs per loop

In [23]: test_100 = '\n'.join([test] * 100)

In [24]: %timeit pd.read_json('[%s]' % ','.join(test_100.splitlines()))
1000 loops, best of 3: 1.25 ms per loop

In [25]: %timeit l = [json.loads(l) for l in test_100.splitlines()]; df = pd.DataFrame(l)
1000 loops, best of 3: 1.25 ms per loop

In [26]: test_1000 = '\n'.join([test] * 1000)

In [27]: %timeit l = [json.loads(l) for l in test_1000.splitlines()]; df = pd.DataFrame(l)
100 loops, best of 3: 9.78 ms per loop

In [28]: %timeit pd.read_json('[%s]' % ','.join(test_1000.splitlines()))
100 loops, best of 3: 3.36 ms per loop

Note: of that time the join is surprisingly fast.

There are several problems with the logic of your code.

ss = s.read()

reads the entire file s into a single string. The next line

for line in ss:

iterates over each character in that string, one by one. So on each loop line is a single character. In

    line = ss[7:]

you are getting the entire file contents apart from the first 7 characters (in positions 0 through 6, inclusive) and replacing the previous content of line with that. And then

T.append(json.loads(line))

attempts to convert that to JSON and store the resulting object into the T list.

Here's some code that does what you want. We don't need to read the entire file into a string with .read, or into a list of lines with .readlines, we can simply put the file handle into a for loop and that will iterate over the file line by line.

We use a with statement to open the file, so that it will get closed automatically when we exit the with block, or if there's an IO error.

import json

table = []
with open('simple.json', 'r') as f:
    for line in f:
        table.append(json.loads(line[7:]))

for row in table:
    print(row)

output

{'color': '33ef', 'age': '55', 'gender': 'm'}
{'color': '3444', 'age': '56', 'gender': 'f'}
{'color': '3999', 'age': '70', 'gender': 'm'}

We can make this more compact by building the table list in a list comprehension:

import json

with open('simple.json', 'r') as f:
    table = [json.loads(line[7:]) for line in f]

for row in table:
    print(row)

Your speed-up claim:

avoid loading the whole file at once, instead parsing a file line by line, which seems to be 2x as fast according to testing

seems highly unlikely, and I would like to see proof of this. A compiled-and-tuned built-in JSON parser that operates on an in-memory buffer is nearly certain to outperform a non-compiled, non-built-in, line-by-line parser. The only advantage that your code will likely have is reduced memory occupation for huge files.

More important than speed is correctness, and it seems you've already discovered cases where your parser simply breaks.

It's also worth calling out that your parser is call-recursive. It will be trivially easy to crash your parser by providing a sufficiently-nested JSON file that will blow its stack, and these do exist in the wild in non-malicious situations.

Basically, other than for learning purposes this shouldn't really be done at all. For the 99% of cases where it works, use built-in JSON parsing. For the 1% of cases where memory concerns actually call for iterative parsing, your problem has already been solved multiple times.