Update: I wrote a solution that does not require reading the entire file in one go. It is too big for a stackoverflow answer, but can be found here jsonstream.

You can use json.JSONDecoder.raw_decode to decode arbitarily big strings of "stacked" JSON (so long as they can fit in memory). raw_decode stops once it has a valid object and returns the last position where was not part of the parsed object. It is poorly documented [1] (see footer), but you can pass this position back to raw_decode and it start parsing again from that position. Unfortunately, the Python json module doesn ot accept strings that have prefixing whitespace. So we need to search to find the first non-whitespace part of your document.

from json import JSONDecoder, JSONDecodeError
import re

NOT_WHITESPACE = re.compile(r'\S')

def decode_stacked(document, idx=0, decoder=JSONDecoder()):
    while True:
        match = NOT_WHITESPACE.search(document, idx)
        if not match:
            return
        idx = match.start()
        
        try:
            obj, idx = decoder.raw_decode(document, idx)
        except JSONDecodeError:
            # do something sensible if there's some error
            raise
        yield obj

s = """

{"a": 1}  


   [
1
,   
2
]


"""

for obj in decode_stacked(s):
    print(obj)

prints:

{'a': 1}
[1, 2]

Note About Missing Documentation

The current signature of raw_decode() dates from 2009, when simplejson was ported into the standard library. The documentation for raw_decode() in simplejson mentions an optional idx argument that can be used to start parsing at an offset. Given that the signature of raw_decode() has not changed since 2009, I think it is fair to assume the API is fairly stable. Especially as decode() uses the idx argument of raw_decode() to ignore prefixing whitespace when parsing a string. And this is exactly what this answer is using the idx argument for too. The documentation of raw_decode() in simplejson is:

raw_decode(s[, idx=0])

Decode a JSON document from s (a str or unicode beginning with a JSON document) starting from the index idx and return a 2-tuple of the Python representation and the index in s where the document ended.

This can be used to decode a JSON document from a string that may have extraneous data at the end, or to decode a string that has a series of JSON objects.

JSONDecodeError will be raised if the given JSON document is not valid.

You should expect something like one of these:

[ { ... }, { ... }, { ... } ]

{ "items": [ { ... }, { ... }, { ... } ] }

You should not expect something like one of these:

{ ... }, { ... }, { ... }

{ ... }{ ... }{ ... }

These are not valid JSON and you should not accept them if you are in a position to influence an API that is still under construction. If the response content type is going to be application/json, it had better be giving you valid JSON bytes.

The right way to do this is simply to do your GET /some/resource/collection and then inspect the JSON response.

I think the problem is that you are overwriting the file with fs.writeFileSync().

You should use fs.appendFileSync() to add new data to the end of the file. See the node docs.

https://nodejs.org/api/fs.html#fs_fs_appendfilesync_file_data_options

It would be better to assemble all of your data into one dict and then write it all out one time, instead of each time in the loop.

d = {}
for i in hosts_data:
    log.info("Gathering host facts for host: {}".format(i['host']['name']))
    try:
        facts = requests.get(foreman_host+api+"hosts/{}/facts".format(i['host']['id']), auth=(username, password))
        if hosts.status_code != 200:
            log.error("Unable to connect to Foreman! Got retcode '{}' and error message '{}'"
            .format(hosts.status_code, hosts.text))
            sys.exit(1)
    except requests.exceptions.RequestException as e:
        log.error(e)
    facts_data = json.loads(facts.text)
    log.debug(facts_data)
    d.update(facts_data)  #add to dict
# write everything at the end
with open(results_file, 'a') as f:
    f.write(json.dumps(d, sort_keys=True, indent=4))

That doesn't look much like json, but yes, you can totally have an array of objects in json file. Something like this in your case:

[{"firstName": "John", "lastName": "Smith"},       
{"firstName": "Jane", "lastName": "Doe"}]

A json file may either contain a single object in (which can be complex, with many nested keys) or an array of such objects. It's either curly braces or square brackets on the outside.