You should use extend instead of append. It will add the items of the passed list to result instead of a new list:
files=['my.json','files.json',...,'name.json']
def merge_JsonFiles(filename):
result = list()
for f1 in filename:
with open(f1, 'r') as infile:
result.extend(json.load(infile))
with open('counseling3.json', 'w') as output_file:
json.dump(result, output_file)
merge_JsonFiles(files)
You should use extend instead of append. It will add the items of the passed list to result instead of a new list:
files=['my.json','files.json',...,'name.json']
def merge_JsonFiles(filename):
result = list()
for f1 in filename:
with open(f1, 'r') as infile:
result.extend(json.load(infile))
with open('counseling3.json', 'w') as output_file:
json.dump(result, output_file)
merge_JsonFiles(files)
import json
import pandas as pd
with open('example1.json') as f1: # open the file
data1 = json.load(f1)
with open('example2.json') as f2: # open the file
data2 = json.load(f2)
df1 = pd.DataFrame([data1]) # Creating DataFrames
df2 = pd.DataFrame([data2]) # Creating DataFrames
MergeJson = pd.concat([df1, df2], axis=1) # Concat DataFrames
MergeJson.to_json("MergeJsonDemo.json") # Writing Json
» pip install jsonmerge
python 3.x - Merge multiple JSON files (more than two) - Stack Overflow
Merging multiple JSON files using Python - Code Review Stack Exchange
What is the best way to merge two JSON file in Python?
Issue with merging multiple JSON files in Python - Stack Overflow
Videos
You're using the json module to convert the JSON file into Python objects, but you're not using the module to convert those Python objects back into JSON. Instead of this at the end
textfile_merged.write(str(all_items))
try this:
json.dump({ "items": all_items }, textfile_merged)
(Note that this is also wrapping the all_items array in a dictionary so that you get the output you expect, otherwise the output will be a JSON array, not an object with an "items" key).
If you just want to merge all json files sequentially,
go to the folder where all json files are, select all and rename the first one as "yourchoice", by doing this all will be in sequential order i.e. yourchoice1,yourchoice2 ...
next go to cmd and type : copy *.json "outputfilename".json
All of your json files are merged sequentially into the "outputfilename".json file
- First off, if you want reusability, turn this into a function. The function should have it's respective arguments.
- Secondly, instead of allocating a variable to store all of the JSON data to write, I'd recommend directly writing the contents of each of the files directly to the merged file. This will help prevent issues with memory.
- Finally, I just have a few nitpicky tips on your variable naming. Preferably,
headshould have a name more along the lines ofmerged_files, and you shouldn't be usingfas an iterator variable. Something likejson_filewould be better.
This is essentially alexwlchan's comment spelled out:
Parsing and serializing JSON doesn't come for free, so you may want to avoid it. I think you can just output "[", the first file, ",", the second file etc., "]" and call it a day. If all inputs are valid JSON, unless I'm terribly mistaken, this should also be valid JSON.
In code, version 1:
def cat_json(outfile, infiles):
file(outfile, "w")\
.write("[%s]" % (",".join([mangle(file(f).read()) for f in infiles])))
def mangle(s):
return s.strip()[1:-1]
Version 2:
def cat_json(output_filename, input_filenames):
with file(output_filename, "w") as outfile:
first = True
for infile_name in input_filenames:
with file(infile_name) as infile:
if first:
outfile.write('[')
first = False
else:
outfile.write(',')
outfile.write(mangle(infile.read()))
outfile.write(']')
The second version has a few advantages: its memory requirements should be something like the size of the longest input file, whereas the first requires twice the sum of all file sizes. The number of simultaneously open file handles is also smaller, so it should work for any number of files.
By using with, it also does deterministic (and immediate!) deallocation of file handles upon leaving each with block, even in python implementations with non-immediate garbage collection (such as pypy and jython etc.).
Hello everyone,
I am trying to merge two JSON files, but I couldn't find any quick package that can do this. One file contains the base policy, while the other includes additional files for excluding special configurations.
My goal is to merge these two JSON files of AntiVirus policy, which contain arrays and numerous elements, without overwriting any data. I was wondering what the best approach would be to accomplish this.
If its element just uses the value of the other files.
If its array just append new elements.
What is best way to achieve this goal?
Thanks all
You can't just concatenate two JSON strings to make valid JSON (or combine them by tacking ',\n' to the end of each).
Instead, you could combine the two (as Python objects) into a Python list, then use json.dump to write it to a file as JSON:
import json
import glob
result = []
for f in glob.glob("*.json"):
with open(f, "rb") as infile:
result.append(json.load(infile))
with open("merged_file.json", "wb") as outfile:
json.dump(result, outfile)
If you wanted to do it without the (unnecesssary) intermediate step of parsing each JSON file, you could merge them into a list like this:
import glob
read_files = glob.glob("*.json")
with open("merged_file.json", "wb") as outfile:
outfile.write('[{}]'.format(
','.join([open(f, "rb").read() for f in read_files])))
There is a module called jsonmerge which merges dictionaries. It can be used very simple by just providing two dictionaries or you can define schema's that described how to merge, like instead of overwriting same key's, automatically create a list and append to it.
base = {
"foo": 1,
"bar": [ "one" ],
}
head = {
"bar": [ "two" ],
"baz": "Hello, world!"
}
from jsonmerge import merge
result = merge(base, head)
print(result)
>>> {'foo': 1, 'bar': ['two'], 'baz': 'Hello, world!'}
More examples with complex rules: https://pypi.org/project/jsonmerge/#description
I have a large number of big json files and I cannot open all of them at the same time. I'm trying to merge them into one json file, but all the solutions I find on SO (e.g., this) require opening the files which causes my machine to crash. I was wondering if there's a way to merge them without opening them
Hi there, I am new to Python and I am trying to merge approx 350 JSON files into one. Any ideas how can I do that? Thanks for any advice!
This is my first venture into anything Python and JSON. I'm receiving about 30 JSON files a day all stored in the same directory. I would like to be able to combine these JSON files into a single file nightly. Once I have the single file I will then flatten out the JSON file and convert to a CSV.
All of my JSON files are structured the same way regarding the objects that are included. I have been reading over many different ways to merge the JSON files with Python but have not had any luck yet.
Here is what I am currently trying to get to work and any help would be appreciated.
import glob
import json
result =[]
for f in glob.glob("*.json"):
with open(f, "r") as infile:
result.append(json.load(infile))
with open("merged_file.json", "w") as outfile:
json.dump(result, outfile)I then would run this code which does convert from JSON to CSV.
import json
import csv
def get_leaves(item, key=None):
if isinstance(item, dict):
leaves = {}
for i in item.keys():
leaves.update(get_leaves(item[i], i))
return leaves
elif isinstance(item, list):
leaves = {}
for i in item:
leaves.update(get_leaves(i, key))
return leaves
else:
return {key: item}
with open('test2.json') as f_input:
json_data = json.load(f_input)
fieldnames = set()
for entry in json_data:
fieldnames.update(get_leaves(entry).keys())
with open('output.csv', 'a', newline='') as f_output:
csv_output = csv.DictWriter(f_output, fieldnames=sorted(fieldnames))
csv_output.writeheader()
csv_output.writerows(get_leaves(entry) for entry in json_data)In a perfect world I would love to be able to have Python automate the process and run at 3 am daily. It would search the directory for any JSON files created over the last 24 hours and combine them into the single JSON file. Then convert the file to CSV.
If anyone cant assist I would appreciate it.