Why json.dump() and .load() are really needed?
What is the difference between json.dump() and ...
JSON dumps format Python - Stack Overflow
python - how to format json dumps - Stack Overflow
Hi, hope everyone is well.
Just nearing the basics end of PCC book, I'm at saving user's data now. What exactly is the reason, when storing simple data, to use json.dump() or load(), instead of just saving and then reading it from simple text file?
I just can't place it in my head why do I really need it and it always makes it more difficult for me to learn if that's the case.
Thank you all in advance.
If you want to dump the JSON into a file/socket or whatever, then you should go with dump(). If you only need it as a string (for printing, parsing or whatever) then use dumps() (dump string)
As mentioned by Antti Haapala in this answer, there are some minor differences on the ensure_ascii behaviour. This is mostly due to how the underlying write() function works, being that it operates on chunks rather than the whole string. Check his answer for more details on that.
json.dump()
Serialize obj as a JSON formatted stream to fp (a .write()-supporting file-like object
If ensure_ascii is False, some chunks written to fp may be unicode instances
json.dumps()
Serialize obj to a JSON formatted str
If ensure_ascii is False, the result may contain non-ASCII characters and the return value may be a unicode instance
In memory usage and speed.
When you call jsonstr = json.dumps(mydata) it first creates a full copy of your data in memory and only then you file.write(jsonstr) it to disk. So this is a faster method but can be a problem if you have a big piece of data to save.
When you call json.dump(mydata, file) -- without 's', new memory is not used, as the data is dumped by chunks. But the whole process is about 2 times slower.
Source: I checked the source code of json.dump() and json.dumps() and also tested both the variants measuring the time with time.time() and watching the memory usage in htop.
There's a parameter called indent. It is None by default, which means "no pretty printing". If you set it to an integer value, it will enable pretty-printing, and use that many spaces to indent nested elements.
In your case it will be something along the lines of:
json.dumps(links, outfile, indent=4)
(or indent=2 if you prefer less spaces). Here's an example from the docs (link), which shows more functionality you might also want as you progress:
>>> import json
>>> print json.dumps({'4': 5, '6': 7}, sort_keys=True,
... indent=4, separators=(',', ': '))
{
"4": 5,
"6": 7
}
The indent parameter allows some degree of pretty printing.
From the documentation:
If indent is a non-negative integer, then JSON array elements and object members will be pretty-printed with that indent level. An indent level of 0, or negative, will only insert newlines. None (the default) selects the most compact representation.
Here's a hack which converts everything to lists and then changes square brackets to curly ones. If your strings might contain square brackets that'll be a problem.
import json
inp = """
{
"aaa": {
"bbb": {
"ccc": {
"ddd": "string1",
"eee": "string2"
}
},
"kkk": "string3"
}
}
"""
inp = json.loads(inp)
def items(d):
if isinstance(d, dict):
return [(k, items(v)) for k, v in d.items()]
return d
inp = items(inp)
print(json.dumps(inp, indent=2).replace("[", "{").replace("]", "}"))
Output:
{
{
"aaa",
{
{
"bbb",
{
{
"ccc",
{
{
"ddd",
"string1"
},
{
"eee",
"string2"
}
}
}
}
},
{
"kkk",
"string3"
}
}
}
}
Note that you are treating dictionary keys as ordered when they aren't, so I made it more explicit with lists.
If it were me, I wouldn't dump to JSON in the first place, I'd serialize the native python data structure straight to C++ initializer list syntax:
myobj = {
"aaa": [
{ "bbb": {
"ccc": [
{"ddd": "string1"},
{"eee": "string2"}
]
}},
{ "kkk": "string3" }
]
}
def pyToCpp(value, key=None):
if key:
return '{{ "{}", {} }}'.format(key, pyToCpp(value))
if type(value) == dict:
for k, v in value.items():
return pyToCpp(v, k)
elif type(value) == list:
l = [pyToCpp(v) for v in value]
return '{{ {} }}'.format(", ".join(l))
else:
return '"{}"'.format(value)
y = pyToCpp(myobj)
print(y)
Output:
{ "aaa", { { "bbb", { "ccc", { { "ddd", "string1" }, { "eee", "string2" } } } }, { "kkk", "string3" } } }
Run it here: https://repl.it/repls/OddFrontUsers
You don't dump a string but to a string.
d = {"foo": "bar"}
j = json.dumps(d)
print(j)
Output:
{"foo": "bar"}
It makes no sense to use a string as the argument of json.dumps which is expecting a dictionary or a list/tuple as the argument.
Serialize
objto a JSON formatted str using this conversion table.
In [17]: d = {"foo": "bar"}
In [18]: j = json.dumps(d)
In [19]: j
Out[19]: '{"foo": "bar"}'
Now if you need your dictionary back instead of a string
In [37]: l=json.loads(j)
In [38]: l
Out[38]: {u'foo': u'bar'}
or alternatively
In [20]: import ast
In [22]: k=ast.literal_eval(j)
In [23]: k
Out[23]: {'foo': 'bar'}
Dumping JSON directly (json.dump) writes the serialized output to the file "on the fly", as it is created. On the other hand, dumping to a string (json.dumps) and then writing the string to a file happens sequentially, so nothing is written to the file until the whole object is serialized in memory.
In practice, this makes very little difference for reasonably sized JSONs. Unless your JSONs are at least several megabytes and you care about the performance, use whatever makes the code cleaner.
For the most part, the two methods are equivalent, however there is one important difference. json.dump can iterate over your data and write it out the file as it iterates, where as json.dumps must return a complete string for you to write to the output:
import json
values = list(range(1000000))
with open("test.json", "w") as f:
# Maximum resident set size (kbytes): 62960
# f.write(json.dumps(values))
# Maximum resident set size (kbytes): 46828
json.dump(values, f)
In some extreme cases, this will cause more memory usage, which could cause problems if you're ever in a resource constrained situation.
Best to avoid the issue unless you have a compelling reason to use json.dumps to output to a file.
dumps takes an object and produces a string:
>>> a = {'foo': 3}
>>> json.dumps(a)
'{"foo": 3}'
load would take a file-like object, read the data from that object, and use that string to create an object:
with open('file.json') as fh:
a = json.load(fh)
Note that dump and load convert between files and objects, while dumps and loads convert between strings and objects. You can think of the s-less functions as wrappers around the s functions:
def dump(obj, fh):
fh.write(dumps(obj))
def load(fh):
return loads(fh.read())
json loads -> returns an object from a string representing a json object.
json dumps -> returns a string representing a json object from an object.
load and dump -> read/write from/to file instead of string