I used the following function (details can be found here):

def flatten_data(y):
    out = {}

    def flatten(x, name=''):
        if type(x) is dict:
            for a in x:
                flatten(x[a], name + a + '_')
        elif type(x) is list:
            i = 0
            for a in x:
                flatten(a, name + str(i) + '_')
                i += 1
        else:
            out[name[:-1]] = x

    flatten(y)
    return out

This unfortunately completely flattens whole JSON, meaning that if you have multi-level JSON (many nested dictionaries), it might flatten everything into single line with tons of columns.

What I used, in the end, was json_normalize() and specified structure that I required. A nice example of how to do it that way can be found here.

Answer from Bostjan on Stack Overflow
Top answer
1 of 10
73

I used the following function (details can be found here):

def flatten_data(y):
    out = {}

    def flatten(x, name=''):
        if type(x) is dict:
            for a in x:
                flatten(x[a], name + a + '_')
        elif type(x) is list:
            i = 0
            for a in x:
                flatten(a, name + str(i) + '_')
                i += 1
        else:
            out[name[:-1]] = x

    flatten(y)
    return out

This unfortunately completely flattens whole JSON, meaning that if you have multi-level JSON (many nested dictionaries), it might flatten everything into single line with tons of columns.

What I used, in the end, was json_normalize() and specified structure that I required. A nice example of how to do it that way can be found here.

2 of 10
20

Cross-posting (but then adapting further) from https://stackoverflow.com/a/62186053/4355695 : In this repo: https://github.com/ScriptSmith/socialreaper/blob/master/socialreaper/tools.py#L8 , I found an implementation of the list-inclusion comment by @roneo to the answer posted by @Imran.

I've added checks to it for catching empty lists and empty dicts. And also added print lines that will help one understand precisely how this function works. You can turn on those print statements by passing crumbs=True in the function's args.

from collections.abc import MutableMapping
def flatten(dictionary, parent_key=False, separator='.', crumbs=False):
    """
    Turn a nested dictionary into a flattened dictionary
    :param dictionary: The dictionary to flatten
    :param parent_key: The string to prepend to dictionary's keys
    :param separator: The string used to separate flattened keys
    :return: A flattened dictionary
    """

    items = []
    for key, value in dictionary.items():
        if crumbs: print('checking:',key)
        new_key = str(parent_key) + separator + key if parent_key else key
        if isinstance(value, MutableMapping):
            if crumbs: print(new_key,': dict found')
            if not value.items():
                if crumbs: print('Adding key-value pair:',new_key,None)
                items.append((new_key,None))
            else:
                items.extend(flatten(value, new_key, separator).items())
        elif isinstance(value, list):
            if crumbs: print(new_key,': list found')
            if len(value):
                for k, v in enumerate(value):
                    items.extend(flatten({str(k): v}, new_key, separator).items())
            else:
                if crumbs: print('Adding key-value pair:',new_key,None)
                items.append((new_key,None))
        else:
            if crumbs: print('Adding key-value pair:',new_key,value)
            items.append((new_key, value))
    return dict(items)

Test it:

ans = flatten({'a': 1, 'c': {'a': 2, 'b': {'x': 5, 'y' : 10}}, 'd': [1, 2, 3], 'e':{'f':[], 'g':{}} })
print('\nflattened:',ans)

Output:

checking: a
Adding key-value pair: a 1
checking: c
c : dict found
checking: a
Adding key-value pair: c.a 2
checking: b
c.b : dict found
checking: x
Adding key-value pair: c.b.x 5
checking: y
Adding key-value pair: c.b.y 10
checking: d
d : list found
checking: 0
Adding key-value pair: d.0 1
checking: 1
Adding key-value pair: d.1 2
checking: 2
Adding key-value pair: d.2 3
checking: e
e : dict found
checking: f
e.f : list found
Adding key-value pair: e.f None
checking: g
e.g : dict found
Adding key-value pair: e.g None

flattened: {'a': 1, 'c.a': 2, 'c.b.x': 5, 'c.b.y': 10, 'd.0': 1, 'd.1': 2, 'd.2': 3, 'e.f': None, 'e.g': None}

Annd that does the job I need done: I throw any complicated json at this and it flattens it out for me. I added a check to the original code to handle empty lists too

Credits to https://github.com/ScriptSmith whose repo I found the intial flatten function in.

Testing OP's sample json, here's the output:

{'count': 13,
 'virtualmachine.0.id': '1082e2ed-ff66-40b1-a41b-26061afd4a0b',
 'virtualmachine.0.name': 'test-2',
 'virtualmachine.0.displayname': 'test-2',
 'virtualmachine.0.securitygroup.0.id': '9e649fbc-3e64-4395-9629-5e1215b34e58',
 'virtualmachine.0.securitygroup.0.name': 'test',
 'virtualmachine.0.securitygroup.0.tags': None,
 'virtualmachine.0.nic.0.id': '79568b14-b377-4d4f-b024-87dc22492b8e',
 'virtualmachine.0.nic.0.networkid': '05c0e278-7ab4-4a6d-aa9c-3158620b6471',
 'virtualmachine.0.nic.1.id': '3d7f2818-1f19-46e7-aa98-956526c5b1ad',
 'virtualmachine.0.nic.1.networkid': 'b4648cfd-0795-43fc-9e50-6ee9ddefc5bd',
 'virtualmachine.0.nic.1.traffictype': 'Guest',
 'virtualmachine.0.hypervisor': 'KVM',
 'virtualmachine.0.affinitygroup': None,
 'virtualmachine.0.isdynamicallyscalable': False}

So you'll see that 'tags' and 'affinitygroup' keys are also handled and added to output. Original code was omitting them.

2021-05-30 : Updated: collections.MutableMapping is changed to collections.abc.MutableMapping

2023-01-11 : edited, added separator arg in second items.extend() call as advised by @MHebes

2024-02-20 : how did that .abc go missing from the import statement?

2025-07-21 : moved crumbs param into function's args

🌐
PyPI
pypi.org › project › flatten-json
flatten-json · PyPI
Flattens JSON objects in Python.
      » pip install flatten-json
    
Published   Oct 27, 2023
Version   0.1.14
Discussions

How to Flatten Nested Json Files Efficiently?
I would absolutely do this in SQL first if you have the option. SQL was designed to be lightning fast at tasks like this (look up lateral flatten if you are unfamiliar with how to do this). More on reddit.com
🌐 r/datascience
19
41
January 15, 2024
Help with flattening really deeply nested json.
Here's a version that enumerates list items and throws out lists with only 1 element in them: def flatten(old_data, new_data=None, parent_key='', sep='_', width=2): ''' Json-style nested dictionary / list flattener :old_data: the original data :new_data: the result dictionary :parent_key: all keys will have this prefix :sep: the separator between the keys :width: width of the field when converting list indexes ''' if new_data is None: new_data = {} if isinstance(old_data, dict): for k, v in old_data.items(): new_key = parent_key + sep + k if parent_key else k flatten(v, new_data, new_key, sep, width) elif isinstance(old_data, list): if len(old_data) == 1: flatten(old_data[0], new_data, parent_key, sep, width) else: for i, elem in enumerate(old_data): new_key = "{}{}{:0>{width}}".format(parent_key, sep if parent_key else '', i, width=width) flatten(elem, new_data, new_key, sep, width) else: if parent_key not in new_data: new_data[parent_key] = old_data else: raise AttributeError("key {} is already used".format(parent_key)) return new_data More on reddit.com
🌐 r/learnpython
9
2
July 10, 2017
How do you flatten nested json/xml?
It depends… Unnesting structs is fine, it’s when you start exploding arrays that you have to think about what your data is representing and what you are trying to achieve with your data. Imagine having an order with information from the order header and an array of order lines, if you explode this array and keep it in the same table the data from your header becomes harder to use for some use cases. In this case I would create 2 tables: order_header (drop the array and unnest everything) order_line (explode the array and drop all header data except for order_id) In many cases I end up flattening data with multiple arrays into multiple tables. In most cases just unnesting everything and exploding all arrays doesn’t make sense. If the array always only contains 1 item m you can also do column[0] to get the first (and only) element before unnesting. More on reddit.com
🌐 r/dataengineering
28
22
January 24, 2024
Dealing with big JSON objects - flatten into tabular or find a way to query JSON efficiently?
It really depends on how you plan to use your data. Months ago I tried treating the JSON directly after the request and the storage is really simple; however, whenever my client needed more fields added, I had to re-extract all the data which included making hundreds of thousands of requests. If you believe you are going to need extra fields in the future, I strongly recommend saving the data as JSON and then treating it. As to how you’re going to save it/your pipeline, I don’t know how to help you since I mainly use python and parquet files with local storage + airflow More on reddit.com
🌐 r/dataengineering
34
42
November 22, 2023
🌐
GeeksforGeeks
geeksforgeeks.org › python › flattening-json-objects-in-python
Flattening JSON objects in Python - GeeksforGeeks
July 12, 2025 - The json-flatten library provides functions for flattening a JSON object to a single key-value pairs, and unflattening that dictionary back to a JSON object. Installing library In order to use the flatten_json library, we need to install this ...
🌐
Simon Willison
simonwillison.net › 2024 › Sep › 7 › json-flatten
json-flatten, now with format documentation
September 7, 2024 - [[[cog example = { "fruits": ["apple", "banana", "cherry"] } cog.out("```json\n") cog.out(str(example)) cog.out("\n```\n") cog.out("Flattened:\n```\n") for key, value in flatten(example).items(): cog.out(f"{key}: {value}\n") cog.out("```\n") ]]] [[[end]]]
🌐
Medium
medium.com › @rganesh0203 › flatten-json-format-different-methods-using-python-1d4357e446cd
Flatten JSON format different methods using Python! | by R. Ganesh | Medium
July 1, 2024 - Display Schema: Use printSchema() to display the nested structure of the JSON data. Flatten DataFrame: Use the explode function to flatten the nested array of phones.
🌐
GitHub
github.com › amirziai › flatten
GitHub - amirziai/flatten: Flatten JSON in Python
Flattens JSON objects in Python.
Starred by 552 users
Forked by 98 users
Languages   Python 100.0% | Python 100.0%
🌐
GeeksforGeeks
geeksforgeeks.org › python › python-pandas-flatten-nested-json
Python Pandas - Flatten nested JSON - GeeksforGeeks
December 10, 2025 - Converting JSON data into a Pandas DataFrame makes it easier to analyze, manipulate, and visualize. Pandas provides a built-in function- json_normalize(), which efficiently flattens simple to moderately nested JSON data into a flat tabular format.
Find elsewhere
🌐
Towards Data Science
towardsdatascience.com › home › latest › how to flatten deeply nested json objects in non-recursive elegant python
How to Flatten Deeply Nested JSON Objects in Non-Recursive Elegant Python | Towards Data Science
March 5, 2025 - The following function is an example of flattening JSON recursively. Code at line 16 and 20 calls function "flatten" to keep unpacking items in JSON object until all values are atomic elements (no dictionary or list).
🌐
Kaggle
kaggle.com › code › jboysen › quick-tutorial-flatten-nested-json-in-pandas
Quick Tutorial: Flatten Nested JSON in Pandas
Checking your browser before accessing www.kaggle.com · Click here if you are not automatically redirected after 5 seconds
🌐
Medium
selectfrom.dev › flatten-nested-json-python-scala-8f3b1833b8c7
How to Flatten Nested JSON Using Python and Scala | by Siddharth Ghosh | SelectFrom
May 17, 2025 - We shall now explore the wrapper method, which takes a JSON string to be flattened and a bool variable “need_parent” as a parameter that adds the parent key as a prefix. We can modify the function to accept a custom separator as a parameter for prefixing the parent key.
🌐
Reddit
reddit.com › r/datascience › how to flatten nested json files efficiently?
r/datascience on Reddit: How to Flatten Nested Json Files Efficiently?
January 15, 2024 -

I am working with extremely nested json data and need to flatten out the structure. I have been using pandas json_normalize, but I have only been working with a fraction of the data and need to start flattening out all of the data. With only a few GB of data, Json_normalize is taking me around 3 hours to complete. I need it to run much faster in order to complete my analysis on all of the data. How do I make this more efficient? Is there a better route to go with this function? My team is thinking about transferring our work to pyspark but I am hesitant as the rest of the ETL processing doesn't take long at all, and it is really this part of the process that takes forever. I also saw people online recommend to use pandas json_normalize to do this procedure rather than using pyspark. I would appreciate any insight, thanks!

🌐
PyPI
pypi.org › project › json-flatten
json-flatten
JavaScript is disabled in your browser. Please enable JavaScript to proceed · A required part of this site couldn’t load. This may be due to a browser extension, network issues, or browser settings. Please check your connection, disable any ad blockers, or try using a different browser
🌐
TutorialsPoint
tutorialspoint.com › flattening-json-objects-in-python
Flattening JSON objects in Python
August 21, 2023 - Python has a built-in JSON module that provides functions to encode and decode JSON data. data= <json-data-here> flattened_data = flatten(data, "_")
🌐
GitHub
github.com › simonw › json-flatten
GitHub - simonw/json-flatten: Python functions for flattening a JSON object to a single dictionary of pairs, and unflattening that dictionary back to a JSON object · GitHub
>>> import json_flatten >>> json_flatten.flatten({"foo": {"bar": [1, True, None]}}) {'foo.bar.[0]$int': '1', 'foo.bar.[1]$bool': 'True', 'foo.bar.[2]$none': 'None'} >>> json_flatten.unflatten(_) {'foo': {'bar': [1, True, None]}}
Starred by 56 users
Forked by 9 users
Languages   Python
🌐
CodeSpeedy
codespeedy.com › home › how to flatten json objects in python
How to flatten JSON objects in Python - CodeSpeedy
December 10, 2020 - In this post, we are going to see how to flatten JSON objects in Python. We will use 2 methods. 1. Using recursion. 2. Using flatten_json library.
🌐
Nexla
nexla.com › home › data automation › how to flatten data
How To Flatten Data | Nexla
February 17, 2023 - If you have a Nexset containing nested JSON that you want to flatten, the first step is to transform that Nexset, add a rule group, and select Transform: Code. Nexla will display a code editor to write low-code transformations in Python or Javascript; in this example we’ll be using Python.
🌐
Medium
medium.com › @data_engineering_0216 › flatten-json-data-file-to-pyspark-dataframe-using-python-a96e7c3816c3
Flatten Json data/file in to PySpark Dataframe using Python function. | by Viral Patel | Medium
February 18, 2024 - The above code provides a function called flatten_json that takes a DataFrame df as input and returns the flattened version of it.
🌐
GitHub
github.com › andrewrgoss › flatten-json
GitHub - andrewrgoss/flatten-json: Small Python script for flattening JSON objects. Unit-tested.
Small Python script for flattening JSON objects. Unit-tested. - andrewrgoss/flatten-json
Author   andrewrgoss