If you load in the entire json as a dict (or list) e.g. using json.load, you can use json_normalize:
In [11]: d = {"response": {"body": {"contact": {"email": "mr@abc.com", "mobile_number": "0123456789"}, "personal": {"last_name": "Muster", "gender": "m", "first_name": "Max", "dob": "1985-12-23", "family_status": "single", "title": "Dr."}, "customer": {"verified": "true", "customer_id": "1234567"}}, "token": "dsfgf", "version": "1.1"}}
In [12]: df = pd.json_normalize(d)
In [13]: df.columns = df.columns.map(lambda x: x.split(".")[-1])
In [14]: df
Out[14]:
email mobile_number customer_id verified dob family_status first_name gender last_name title token version
0 mr@abc.com 0123456789 1234567 true 1985-12-23 single Max m Muster Dr. dsfgf 1.1
Answer from Andy Hayden on Stack OverflowI'm trying to figure out how to extract heavily nested JSON data and convert it to data tables using pandas. I got to where I can go down to one nested level, but I don't know how to phrase the request to go to the next level down.
import pandas URL = 'https://statsapi.mlb.com/api/v1.1/game/642186/feed/live' df = pandas.read_json(URL) df=pandas.json_normalize(df['liveData']) df=pandas.DataFrame(df) print(df) print(df.info())
So this goes to the 'liveData' level. Nested under 'liveData' is 'plays' -- 'allPlays' -- 'result'. Is it possible to write the JSON_normalize code so it gets down to the 'result' level, and the data frame is at the result level (which I would use sqlalchemy to put into a MySQL table)?
pandas - How to convert nested json into python dataframe - Stack Overflow
python - How to convert a nested JSON file into a Pandas dataframe? - Stack Overflow
python - Nested Json in to dataframe (pandas) - Stack Overflow
python - Parsing nested JSON into dataframe - Stack Overflow
If you load in the entire json as a dict (or list) e.g. using json.load, you can use json_normalize:
In [11]: d = {"response": {"body": {"contact": {"email": "mr@abc.com", "mobile_number": "0123456789"}, "personal": {"last_name": "Muster", "gender": "m", "first_name": "Max", "dob": "1985-12-23", "family_status": "single", "title": "Dr."}, "customer": {"verified": "true", "customer_id": "1234567"}}, "token": "dsfgf", "version": "1.1"}}
In [12]: df = pd.json_normalize(d)
In [13]: df.columns = df.columns.map(lambda x: x.split(".")[-1])
In [14]: df
Out[14]:
email mobile_number customer_id verified dob family_status first_name gender last_name title token version
0 mr@abc.com 0123456789 1234567 true 1985-12-23 single Max m Muster Dr. dsfgf 1.1
It's much easier if you deserialize the JSON using the built-in json module first (instead of pd.read_json()) and then flatten it using pd.json_normalize().
# deserialize
with open(r'C:\scoring_model\json.js', 'r') as f:
data = json.load(f)
# flatten
df = pd.json_normalize(d)
If a dictionary is passed to json_normalize(), it's flattened into a single row, but if a list is passed to it, it's flattened into multiple rows. So if the nested structure contains only key-value pairs, pd.json_normalize() with no parameters suffices to flatten it.
However, if the data contains a list (JSON array in the nesting in the file), then passing record_path= argument to let pandas find the path to the records. For example, if the data is like the following (notice how the value under "body" is a list, i.e. a list of records):
data = {
"response":[
{
"version":"1.1",
"customer": {"id": "1234567", "verified":"true"},
"body":[
{"email":"mr@abc.com", "mobile_number":"0123456789"},
{"email":"ms@abc.com", "mobile_number":"9876543210"}
]
},
{
"version":"1.2",
"customer": {"id": "0987654", "verified":"true"},
"body":[
{"email":"master@abc.com", "mobile_number":"9999999999"}
]
}
]
}
then you can pass record_path= to let the program know that the records are under "body" and pass meta= to set the path to the metadata. Note how in "body", "version" and "customer" are in the same level in the data but "id" is nested one level more so you need to pass a list to get the value under "id".
df = pd.json_normalize(data['response'], record_path=['body'], meta=['version', ['customer', 'id']])

This function recursively calls itself to flatten dictionaries and lists.
from collections import OrderedDict
def flatten(json_object, container=None, name=''):
if container is None:
container = OrderedDict()
if isinstance(json_object, dict):
for key in json_object:
flatten(json_object[key], container=container, name=name + key + '_')
elif isinstance(json_object, list):
for n, item in enumerate(json_object, 1):
flatten(item, container=container, name=name + str(n) + '_')
else:
container[str(name[:-1])] = str(json_object)
return container
Examples:
flatten([1, 2, 3])
OrderedDict([('1', '1'), ('2', '2'), ('3', '3')])
flatten([1, 2, 3], name='x')
OrderedDict([('x1', '1'), ('x2', '2'), ('x3', '3')])
flatten({'a': [1, 2, 3], 'b': 4, 'c': {'d': [5, 6], 'e': 7}}, name='x')
OrderedDict([('xa_1', '1'),
('xa_2', '2'),
('xa_3', '3'),
('xc_e', '7'),
('xc_d_1', '5'),
('xc_d_2', '6'),
('xb', '4')])
Response:
# j = json string
>>> pd.DataFrame(flatten(j), index=[0]).T
0
perMinuteLimit 10
distance 10
perMonthCurrent 0
longitude 35.751607
perMonthLimit 2000
latitude -6.162959
perMinuteCurrent 0
networkRank_1_networkId 6402
networkRank_1_type3G_sampleSizeSpeed 29
networkRank_1_type3G_averageRssiAsu 9.5429091136
networkRank_1_type3G_pingTime 320.9600
networkRank_1_type3G_networkType 3
networkRank_1_type3G_averageRssiDb -69.5664329624972
networkRank_1_type3G_networkName Vodacom
networkRank_1_type3G_networkId 6402
networkRank_1_type3G_downloadSpeed 1508.1304
networkRank_1_type3G_uploadSpeed 893.7692
networkRank_1_type3G_reliability 0.804236452826138
networkRank_1_type3G_sampleSizeRSSI 948
networkRank_1_networkName Vodacom
networkRank_2_networkId 6400
networkRank_2_type3G_sampleSizeSpeed 21
networkRank_2_type3G_averageRssiAsu 15.3537142857
networkRank_2_type3G_pingTime 259.0000
networkRank_2_type3G_networkType 3
networkRank_2_type3G_averageRssiDb -61.4563389583101
networkRank_2_type3G_networkName tiGO
networkRank_2_type3G_networkId 6400
networkRank_2_type3G_downloadSpeed 516.0000
networkRank_2_type3G_uploadSpeed 320.4211
networkRank_2_type3G_reliability 0.911904765537807
networkRank_2_type3G_sampleSizeRSSI 935
networkRank_2_networkName tiGO
networkRank_3_networkId 6403
networkRank_3_type3G_sampleSizeSpeed 21
networkRank_3_type3G_averageRssiAsu 13.2729999375
networkRank_3_type3G_pingTime 194.5556
networkRank_3_type3G_networkType 3
networkRank_3_type3G_averageRssiDb -58.1521092977699
networkRank_3_type3G_networkName Airtel
networkRank_3_type3G_networkId 6403
networkRank_3_type3G_downloadSpeed 1080.2500
networkRank_3_type3G_uploadSpeed 572.1579
networkRank_3_type3G_reliability 0.554680264185345
networkRank_3_type3G_sampleSizeRSSI 587
networkRank_3_networkName Airtel
network_type None
apiVersion 2
1) Parse JSON string to python structure
2) Iterete over 'networkRank' list of dictionaries and put each key you want to add inside the hash
for data_row in deserialized_json['networkRank']:
data_row['latitude'] = deserialized_json['latitude']
# etc
3)
yourdataframe = pd.DataFrame( deserialized_json['networkRank'] )