Update: I wrote a solution that does not require reading the entire file in one go. It is too big for a stackoverflow answer, but can be found here jsonstream.

You can use json.JSONDecoder.raw_decode to decode arbitarily big strings of "stacked" JSON (so long as they can fit in memory). raw_decode stops once it has a valid object and returns the last position where was not part of the parsed object. It is poorly documented [1] (see footer), but you can pass this position back to raw_decode and it start parsing again from that position. Unfortunately, the Python json module doesn ot accept strings that have prefixing whitespace. So we need to search to find the first non-whitespace part of your document.

from json import JSONDecoder, JSONDecodeError
import re

NOT_WHITESPACE = re.compile(r'\S')

def decode_stacked(document, idx=0, decoder=JSONDecoder()):
    while True:
        match = NOT_WHITESPACE.search(document, idx)
        if not match:
            return
        idx = match.start()
        
        try:
            obj, idx = decoder.raw_decode(document, idx)
        except JSONDecodeError:
            # do something sensible if there's some error
            raise
        yield obj

s = """

{"a": 1}  


   [
1
,   
2
]


"""

for obj in decode_stacked(s):
    print(obj)

prints:

{'a': 1}
[1, 2]

Note About Missing Documentation

The current signature of raw_decode() dates from 2009, when simplejson was ported into the standard library. The documentation for raw_decode() in simplejson mentions an optional idx argument that can be used to start parsing at an offset. Given that the signature of raw_decode() has not changed since 2009, I think it is fair to assume the API is fairly stable. Especially as decode() uses the idx argument of raw_decode() to ignore prefixing whitespace when parsing a string. And this is exactly what this answer is using the idx argument for too. The documentation of raw_decode() in simplejson is:

raw_decode(s[, idx=0])

Decode a JSON document from s (a str or unicode beginning with a JSON document) starting from the index idx and return a 2-tuple of the Python representation and the index in s where the document ended.

This can be used to decode a JSON document from a string that may have extraneous data at the end, or to decode a string that has a series of JSON objects.

JSONDecodeError will be raised if the given JSON document is not valid.

It would be better to assemble all of your data into one dict and then write it all out one time, instead of each time in the loop.

d = {}
for i in hosts_data:
    log.info("Gathering host facts for host: {}".format(i['host']['name']))
    try:
        facts = requests.get(foreman_host+api+"hosts/{}/facts".format(i['host']['id']), auth=(username, password))
        if hosts.status_code != 200:
            log.error("Unable to connect to Foreman! Got retcode '{}' and error message '{}'"
            .format(hosts.status_code, hosts.text))
            sys.exit(1)
    except requests.exceptions.RequestException as e:
        log.error(e)
    facts_data = json.loads(facts.text)
    log.debug(facts_data)
    d.update(facts_data)  #add to dict
# write everything at the end
with open(results_file, 'a') as f:
    f.write(json.dumps(d, sort_keys=True, indent=4))

I think the problem is that you are overwriting the file with fs.writeFileSync().

You should use fs.appendFileSync() to add new data to the end of the file. See the node docs.

https://nodejs.org/api/fs.html#fs_fs_appendfilesync_file_data_options

This is a working example based (and tested) with gson-2.8.0. It accepts an arbitrary sequence of JSON objects on a given input stream. And, of course, it does not impose any restrictions on how you have formatted your input:

       InputStream is = /* whatever */
       Reader r = new InputStreamReader(is, "UTF-8");
       Gson gson = new GsonBuilder().create();
       JsonStreamParser p = new JsonStreamParser(r);

       while (p.hasNext()) {
          JsonElement e = p.next();
          if (e.isJsonObject()) {
              Map m = gson.fromJson(e, Map.class);
              /* do something useful with JSON object .. */
          }
          /* handle other JSON data structures */
       }

You should expect something like one of these:

[ { ... }, { ... }, { ... } ]

{ "items": [ { ... }, { ... }, { ... } ] }

You should not expect something like one of these:

{ ... }, { ... }, { ... }

{ ... }{ ... }{ ... }

These are not valid JSON and you should not accept them if you are in a position to influence an API that is still under construction. If the response content type is going to be application/json, it had better be giving you valid JSON bytes.

The right way to do this is simply to do your GET /some/resource/collection and then inspect the JSON response.

There are several problems with the logic of your code.

ss = s.read()

reads the entire file s into a single string. The next line

for line in ss:

iterates over each character in that string, one by one. So on each loop line is a single character. In

    line = ss[7:]

you are getting the entire file contents apart from the first 7 characters (in positions 0 through 6, inclusive) and replacing the previous content of line with that. And then

T.append(json.loads(line))

attempts to convert that to JSON and store the resulting object into the T list.

Here's some code that does what you want. We don't need to read the entire file into a string with .read, or into a list of lines with .readlines, we can simply put the file handle into a for loop and that will iterate over the file line by line.

We use a with statement to open the file, so that it will get closed automatically when we exit the with block, or if there's an IO error.

import json

table = []
with open('simple.json', 'r') as f:
    for line in f:
        table.append(json.loads(line[7:]))

for row in table:
    print(row)

output

{'color': '33ef', 'age': '55', 'gender': 'm'}
{'color': '3444', 'age': '56', 'gender': 'f'}
{'color': '3999', 'age': '70', 'gender': 'm'}

We can make this more compact by building the table list in a list comprehension:

import json

with open('simple.json', 'r') as f:
    table = [json.loads(line[7:]) for line in f]

for row in table:
    print(row)