Here is a Python solution to your problem.

Don't forget to change the in_file_path to the location of your big JSON file.

import json

in_file_path='path/to/file.json' # Change me!

with open(in_file_path,'r') as in_json_file:

    # Read the file and convert it to a dictionary
    json_obj_list = json.load(in_json_file)

    for json_obj in json_obj_list:
        filename=json_obj['_id']+'.json'

        with open(filename, 'w') as out_json_file:
            # Save each obj to their respective filepath
            # with pretty formatting thanks to `indent=4`
            json.dump(json_obj, out_json_file, indent=4)

Side Note: I ran this in Python3, it should work in Python2 as well

Use this code in linux command prompt

split -b 53750k <your-file>
cat xa* > <your-file>

Refer to this link: https://askubuntu.com/questions/28847/text-editor-to-edit-large-4-3-gb-plain-text-file

You can try:

import json

dct = {
    "client_id": {"0": "abc123", "1": "def456"},
    "client_name": {"0": "companyA", "1": "companyB"},
    "revenue": {"0": "54,786", "1": "62,754"},
    "rate": {"0": "4", "1": "5"},
}

tmp = {}
for k, v in dct.items():
    for kk, vv in v.items():
        tmp.setdefault(kk, {}).update({k: vv})

for i, v in enumerate(tmp.values(), 1):
    with open(f"File{i}.json", "w") as f_out:
        json.dump(v, f_out, indent=4)

This creates two files File1.json, File2.json:

{
    "client_id": "abc123",
    "client_name": "companyA",
    "revenue": "54,786",
    "rate": "4"
}

and

{
    "client_id": "abc123",
    "client_name": "companyA",
    "revenue": "54,786",
    "rate": "4"
}

EDIT: To create output dictionary:

dct = {
    "client_id": {"0": "abc123", "1": "def456"},
    "client_name": {"0": "companyA", "1": "companyB"},
    "revenue": {"0": "54,786", "1": "62,754"},
    "rate": {"0": "4", "1": "5"},
}

tmp = {}
for k, v in dct.items():
    for kk, vv in v.items():
        tmp.setdefault(kk, {}).update({k: vv})

out = {}
for i, v in enumerate(tmp.values(), 1):
    out[f"File{i}"] = v

print(out)

Prints:

{
    "File1": {
        "client_id": "abc123",
        "client_name": "companyA",
        "revenue": "54,786",
        "rate": "4",
    },
    "File2": {
        "client_id": "def456",
        "client_name": "companyB",
        "revenue": "62,754",
        "rate": "5",
    },
}

Use an iteration grouper; the itertools module recipes list includes the following:

from itertools import izip_longest

def grouper(iterable, n, fillvalue=None):
    "Collect data into fixed-length chunks or blocks"
    # grouper('ABCDEFG', 3, 'x') --> ABC DEF Gxx
    args = [iter(iterable)] * n
    return izip_longest(fillvalue=fillvalue, *args)

This lets you iterate over your tweets in groups of 5000:

for i, group in enumerate(grouper(input_tweets, 5000)):
    with open('outputbatch_{}.json'.format(i), 'w') as outputfile:
        json.dump(list(group), outputfile)

Split by keeping the structure.

You have an Array, denoted by the outer square brackets: []

Then, you have objects, denoted but the curly brackets: {}

Split into different files, by creating different arrays:

File A:

[
  {
    "column1": "value1",
    "column2": "value2",
    "column3": "value3"
  },
  {
    "column1": "value4",
    "column2": "value5",
    "column3": "{'something':'something'}"
  }
]

File B:

[
  {
    "column1": "value7",
    "column2": "value8",
    "column3": "value9"
  }
]

Then, you can read each file, and they will be correct.

To split a json with many records into chunks of a desired size I simply use:

jq -c '.[0:1000]' mybig.json

which works like python slicing.

See the docs here: https://stedolan.github.io/jq/manual/

Array/String Slice: .[10:15]

The .[10:15] syntax can be used to return a subarray of an array or substring of a string. The array returned by .[10:15] will be of length 5, containing the elements from index 10 (inclusive) to index 15 (exclusive). Either index may be negative (in which case it counts backwards from the end of the array), or omitted (in which case it refers to the start or end of the array).

The original file is not valid JSON while the json.dump creates a file with valid JSON. My suggestion would be to convert the line items to JSON one at a time when writing to file.

Replace this:

for i in range(total+1):
    json.dump(ll[i * size_of_the_split:(i + 1) * size_of_the_split], open(
        json_file+"\\split50k" + str(i+1) + ".json", 'w',
        encoding='utf8'), ensure_ascii=False, indent=True)

with this:

for i in range(len(ll)):
    if i % size_of_the_split ==0:
        if i != 0:
            file.close()
        file = open(json_file+"\\split50k"+str(i+1)+".json",'w')
    file.write(str(ll[i]))
file.close()