One way to avoid a copy is to reverse the list inplace, eg:
mylist.reverse()
json_string = json.dumps(mylist)
Then mylist.reverse() it back if needs be.
One way to avoid a copy is to reverse the list inplace, eg:
mylist.reverse()
json_string = json.dumps(mylist)
Then mylist.reverse() it back if needs be.
Before we go crazy, see if any of the following meet your performance requirements:
mylist.reverse(); json.dumps(mylist); mylist.reverse()
json.dumps(mylist[::-1])
json.dumps(tuple(reversed(mylist)))
You mentioned defining your own JSONEncoder default function, which is fairly simple to do (example at the very bottom*), but I don't think it works here since the json.JSONEncoder requires the default function to convert the object into one of the following:
None, True, False, str, int, float, list, tuple, dict
Converting an iterator to a list or tuple would create a large object, which is what we're trying to avoid.
You'll either need to modify your json library or monkey-patch it.
Here's the CPython source code of json.encoder. PyPy, Jython, and other Python implementations are probably using the same code for the json module.
https://github.com/python/cpython/blob/master/Lib/json/encoder.py#L204
def _make_iterencode(markers, _default, _encoder, _indent, _floatstr,
_key_separator, _item_separator, _sort_keys, _skipkeys, _one_shot,
## HACK: hand-optimized bytecode; turn globals into locals
ValueError=ValueError,
dict=dict,
float=float,
id=id,
int=int,
isinstance=isinstance,
list=list,
str=str,
tuple=tuple,
_intstr=int.__str__,
...
def _iterencode(o, _current_indent_level):
if isinstance(o, str):
yield _encoder(o)
...
elif isinstance(o, (list, tuple)):
yield from _iterencode_list(o, _current_indent_level)
# Add support for processing iterators
elif isinstance(o, iterator_types):
# Side-effect: this will consume the iterator.
# This is probably why it's not included in the official json module
# We could use itertools.tee to be able to iterate over
# the original iterator while still having an unconsumed iterator
# but this would require updating all references to the original
# iterator with the new unconsumed iterator.
# The side effect may be unavoidable.
yield from _iterencode_list(o, _current_index_level)
For performance reasons, you'll want to define the iterator types outside of the function and bring it in as a local.
str_iterator = type(iter( str() ))
list_iterator = type(iter( list() ))
tuple_iterator = type(iter( tuple() ))
range_iterator = type(iter( range(0) ))
list_reverseiterator = type(reversed( list() ))
reverseiterator = type(reversed( tuple() )) #same as <class 'reversed'>
# Add any other iterator classes that you need here, plus any container data types that json doesn't support (sets, frozensets, bytes, bytearray, array.array, numpy.array)
iterator_types = (str_iterator, list_iterator, tuple_iterator, range_iterator,
list_reverseiterator, reversed)
If you want to go the monkey-patching route, you'll need to redefine the json.encoder._make_iterencode function, replacing all occurrences of isinstance(X, (list, tuple)) with isinstance(X, (list, tuple)+iterator_types)
import json
def _make_iterencode(markers, _default, _encoder, _indent, _floatstr,
_key_separator, _item_separator, _sort_keys, _skipkeys, _one_shot,
iterable_types=_get_iterable_types(),
...
):
...
json.encoder._make_iterencode = _make_iterencode
These changes look something like this: https://github.com/python/cpython/pull/3034/files
*As promised, how to define your own default function, though not useful for dumping iterators without copying the iterator into a list or tuple first.
class JSONEncoderThatSupportsIterators(json.JSONEncoder):
def default(self, o):
try:
iterable = iter(o)
except TypeError:
pass
else:
return list(iterable)
# Let the base class default method raise the TypeError
return json.JSONEncoder.default(self, o)
li = range(10000000) # or xrange if Python 2
dumped = JSONEncoderThatSupportsIterators().encode(reversed(li))
assert dumped.startswith('[999999, 999998, 999997, ')
assert dumped.endswith('6, 5, 4, 3, 2, 1, 0]')
Alternatively, rather than subclassing json.JSONEncoder, you can define the default(self, o) function and pass it as an argument to json.dumps(default=default).
python - Iterating through a JSON object - Stack Overflow
python - How can I loop over entries in JSON? - Stack Overflow
Using Python to loop through JSON dictionary that re-uses the same key?
Reverse loop an array of JSON data
I believe you probably meant:
from __future__ import print_function
for song in json_object:
# now song is a dictionary
for attribute, value in song.items():
print(attribute, value) # example usage
NB: You could use song.iteritems instead of song.items if in Python 2.
Your loading of the JSON data is a little fragile. Instead of:
json_raw= raw.readlines()
json_object = json.loads(json_raw[0])
you should really just do:
json_object = json.load(raw)
You shouldn't think of what you get as a "JSON object". What you have is a list. The list contains two dicts. The dicts contain various key/value pairs, all strings. When you do json_object[0], you're asking for the first dict in the list. When you iterate over that, with for song in json_object[0]:, you iterate over the keys of the dict. Because that's what you get when you iterate over the dict. If you want to access the value associated with the key in that dict, you would use, for example, json_object[0][song].
None of this is specific to JSON. It's just basic Python types, with their basic operations as covered in any tutorial.
Actually, to query the team_name, just add it in brackets to the last line. Apart from that, it seems to work on Python 2.7.3 on command line.
from urllib2 import urlopen
import json
url = 'http://openligadb-json.heroku.com/api/teams_by_league_saison?league_saison=2012&league_shortcut=bl1'
response = urlopen(url)
json_obj = json.load(response)
for i in json_obj['team']:
print i['team_name']
Try this :
import urllib, urllib2, json
url = 'http://openligadb-json.heroku.com/api/teams_by_league_saison?league_saison=2012&league_shortcut=bl1'
request = urllib2.Request(url)
request.add_header('User-Agent','Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)')
request.add_header('Content-Type','application/json')
response = urllib2.urlopen(request)
json_object = json.load(response)
#print json_object['results']
if json_object['team'] == []:
print 'No Data!'
else:
for rows in json_object['team']:
print 'Team ID:' + rows['team_id']
print 'Team Name:' + rows['team_name']
print 'Team URL:' + rows['team_icon_url']
I'm trying to restructure JSON output from a API to the specific format that Ansible wants for it's invetory file. To do this I thought I would loop through the JSON and grab the variables I need and then feed that into the proper structure. However, I noticed my API JSON is using the same $ key for multiple variables. How can I properly reference and differentiate between the variables? Here is an example:
"model-responses": {
"model": [
{
"@mh": "0x11e013",
"attribute": [
{
"@id": "0x1006e",
"$": "switch1"
},
{
"@id": "0x23000e",
"$": "Raritan Computer, Inc.DV"
},
{
"@id": "0x12d7f",
"$": "10.10.10.1"
},
{
"@id": "0x10052",
"$": "PX2 030610"
}
]
},
{
"@mh": "0x115014",
"attribute": [
{
"@id": "0x1006e",
"$": "switch2"
},
{
"@id": "0x23000e",
"$": "DCS-7010T-48"
},
{
"@id": "0x12d7f",
"$": "10.10.10.2"
},
{
"@id": "0x10052",
"$": "Arista Networks EOS version 4.22.3M-INT running on an Arista Networks DCS-7010T-48"
}
]
},I got JSON files which can grow huge, I basically import it as a dictionary. My task is to process them in a certain way - for this purpose let's simplify my goal here:
Iterate through the dictionary. Everytime i find a value, let's print 'Found a value: value'
Everytime i find a list, print the key of that value then go back to step 1.
Continue doing this until the entire dictionary has been navigated through.
Example and the current code I actively use:
iterThis = {
"a": "hello",
"b": [{"name":"Bob"}, {"name":"Kate"}],
"c": "world"
}
def recursive(testDict):
for key, value in testDict.items():
if type(value) == list:
print("Found a list: " + key)
for l in value:
recursive(l)
else:
print("Found a value: " + value
recursive(iterThis)My expected output is the following:
Found a value: hello Found a list: b Found a value: Bob Found a value: Kate Found a value: world
This is a grand oversimplification of the overall task, but this is the most important part.
My problem is that the JSONs I have to process can grow huge (e.g. 20k rows), with lot of lists here and there, and there can be a list within a list within a list etc. This means that until the code reached the 'bottom', everything is pretty much held in memory and my intuition is that there should be a pythonistic approach to this that is efficient.
I have a vague intuition that suggests looking at two things: itertools and generators.
What do you think?
Hi there, I am trying to read through a package.json file which is below. I want to read in specifically the dependencies and add them to key, value variables. I can't figure out how to read in nested object. If anyone can give me any tips?
{
"name": "untitled",
"version": "0.1.0",
"private": true,
"dependencies": {
"react": "^17.0.2",
"react-dom": "^17.0.2",
"react-scripts": "5.0.0",
"express": "https://github.com/IAmAndyIE/expressjs.com.git"
},
}My current code is below.
import json
def test_document():
f = open('../untitled/package.json')
data = json.load(f)
for key, values in data.items():
if key == "dependencies":
print(values)
f.close()
if __name__ == '__main__':
test_document()Thanks
It's because of this line:
data = data['items'][0]
Here you have explicitly accessed that first item in each iteration of the loop.
Try this instead:
data = request.json()
items = data['items']
for item in items:
price = item['salePrice']
name = item['name']
print(name)
Wim is right. Your problem comes from the fact that you are trying to manipulate JSON data when the data is not JSON ready yet.
Below is a cleaner version of your code which calls a fake API that returns a list of 10 users.
The code will only use 2 libraries:
json: this library will make sure our data us JSON ready (for python, I should mention) before we can iterate through itrequests: this library will allow us to perform the http request to get the data back
Using just these two libraries, I can get my data with requests, make it JSON ready using json, loop through it with python using the for ... in ... statement, and print the content any way I want, like so:
# importing the necessary libraries
import json
import requests
# Making the http request
r = requests.get('https://jsonplaceholder.typicode.com/users')
# Transforming the data returned into JSON format
data = r.json()
# Construct various objects and variables to read the data
one_user = {}
first_user = data[0]
first_user_name = data[0]['name']
print('The first user is:', first_user)
print('\n')
print('The first user s name is:', first_user_name)
print('\n')
for user in data:
one_user['name'] = user['name']
one_user['email'] = user['email']
one_user['phone'] = user['phone']
print(one_user)
print('\n')
Data Sample
[
{
"id": 1,
"name": "Leanne Graham",
"username": "Bret",
"email": "[email protected]",
"address": {
"street": "Kulas Light",
"suite": "Apt. 556",
"city": "Gwenborough",
"zipcode": "92998-3874",
"geo": {
"lat": "-37.3159",
"lng": "81.1496"
}
},
"phone": "1-770-736-8031 x56442",
"website": "hildegard.org",
"company": {
"name": "Romaguera-Crona",
"catchPhrase": "Multi-layered client-server neural-net",
"bs": "harness real-time e-markets"
}
},
{...}
]
Screenshot of a working example on jupyter notebook

I hope this helps someone out there trying to solve this issue the right way.