If you are using Django 1.8 you can create your own model field that can store a json. This class will make sure that you have the right JSON format as well.

import json
from django.db import models

class JsonField(models.TextField):
    """
    Stores json-able python objects as json.
    """
    def get_db_prep_value(self, value, connection, prepared=False):
        try:
            return json.dumps(value)
        except TypeError:
            BAD_DATA.error(
                "cannot serialize %s to store in a JsonField", str(value)
            )
            return ""

    def from_db_value(self, value, expression, connection, context):
        if value == "":
            return None
        try:
            return json.loads(value)
        except TypeError:
            BAD_DATA.error("cannot load dictionary field -- type error")
            return None
Answer from nael on Stack Overflow
Top answer
1 of 2
5

If you are using Django 1.8 you can create your own model field that can store a json. This class will make sure that you have the right JSON format as well.

import json
from django.db import models

class JsonField(models.TextField):
    """
    Stores json-able python objects as json.
    """
    def get_db_prep_value(self, value, connection, prepared=False):
        try:
            return json.dumps(value)
        except TypeError:
            BAD_DATA.error(
                "cannot serialize %s to store in a JsonField", str(value)
            )
            return ""

    def from_db_value(self, value, expression, connection, context):
        if value == "":
            return None
        try:
            return json.loads(value)
        except TypeError:
            BAD_DATA.error("cannot load dictionary field -- type error")
            return None
2 of 2
2

I found a way to store JSON data into DB. Since I'm accessing nodes from remote service which returns a list of nodes on every request, I need to build proper json to store/retrieve from db.

Say API returned json text as : '{"cursor": null, "nodes" = [{"name": "Test1", "value: 1}, {"name": "Test2", "value: 2}, ...]}'

So, first we need to access nodes list as:

data = json.loads(api_data)
nodes = data['nodes']

Now for 1st entry into DB column we need to do following:

str_data = json.dumps({"nodes": nodes})

So, str_data would return a valid string/buffer, which we can store into DB with a "nodes" key.

For 2nd or successive entries into DB column, we will do following:

# get data string from DB column and load into json
db_data = json.loads(db_col_data)
# get new/latest 'nodes' data from api as explained above
# append this data to 'db_data' json as
latest_data = db_data["nodes"] + new_api_nodes
# now add this data back to column after json.dumps()
db_col_data = json.dumps(latest_data)
# add to DB col and DB commit

It is a proper way to load/dump data from DB while adding/removing json and keeping proper format.

Thanks!

๐ŸŒ
Stack Overflow
stackoverflow.com โ€บ questions โ€บ 41952552 โ€บ python-how-to-parse-and-save-json-to-mysql-database
Python - How to parse and save JSON to MYSQL database - Stack Overflow
import MySQLdb def dbconnect(): try: db = MySQLdb.connect( host='localhost', user='root', passwd='password', db='nameofdb' ) except Exception as e: sys.exit("Can't connect to database") return db def insertDb(): try: db = dbconnect() cursor = db.cursor() cursor.execute(""" INSERT INTO nameoftable(nameofcolumn) \ VALUES (%s) """, (data)) cursor.close() except Exception as e: print e ... Sign up to request clarification or add additional context in comments. ... If this is merely for storage for processing later, kind of like a cache, a varchar field is enough. If however you need to retrieve some structured jdata, JSON field is what you need.
Discussions

python - How can I use a JSON file such as a database to store new and old objects? - Stack Overflow
I suggest you read the file in one go to get a python dict or list, update it with new information and write the whole thing back again. ... I suggest to use both read and write modes to fulfill this task. First you have to read the current content of the file by using the read state and then store them in a variable. try: with open('data.json... More on stackoverflow.com
๐ŸŒ stackoverflow.com
python - Store nested json repsonses in relational database - Code Review Stack Exchange
The goal is to collect market data from a free rest api. The response is in json and size per response is above 1MB. I want to get at least updated data once per minute which means about 24h * 60mi... More on codereview.stackexchange.com
๐ŸŒ codereview.stackexchange.com
July 28, 2018
Storing JSON into sqlite database in python
SQLite does not have the type JSON, so all your JSON will be stored as string. You could call the SQLite function json on the data to store it as optimized JSON (basically parse it and remove whitespaces). More on reddit.com
๐ŸŒ r/sqlite
6
11
October 28, 2022
python - Keeping JSON in database - Software Engineering Stack Exchange
I'm trying to create web app(flask or django-rest) that would scrape some data and save it to JSON so that it can be viewed in the frontend (VueJS). I'm wondering if it is better to save the scrape... More on softwareengineering.stackexchange.com
๐ŸŒ softwareengineering.stackexchange.com
June 19, 2020
๐ŸŒ
Stack Overflow
stackoverflow.com โ€บ questions โ€บ 62779532 โ€บ store-json-in-database-with-python
Store JSON in database with python - Stack Overflow
Explore Stack Internal ... Save this question. Show activity on this post. I've been having trouble storing JSON to database with non ASCII characters. Here's what I have so far. I'm using python requests library. getJSON = requests.get('https://jsonapi.com/jsonfile') storeDB = requests.post( 'http://databaseAPI/Database/item', data='{"payload": %s}' % (getJSON.text), headers={'Content-Type': 'application/json'} )
๐ŸŒ
JCharisTech
blog.jcharistech.com โ€บ 2020 โ€บ 01 โ€บ 08 โ€บ how-to-convert-json-to-sql-format-in-python
How to Convert JSON to SQL format In Python โ€“ JCharisTech
January 22, 2020 - In this tutorial we will see how to convert JSON โ€“ Javascript Object Notation to SQL data format such as sqlite or db. We will be using Pandas for this. Installation pip install pandas sqlaโ€ฆ
๐ŸŒ
Opensource.com
opensource.com โ€บ article โ€บ 19 โ€บ 7 โ€บ save-and-load-data-python-json
Save and load Python data with JSON | Opensource.com
While you may have previously resorted to custom text configuration files or data formats, JSON offers you structured, recursive storage, and Pythonโ€™s JSON module offers all of the parsing libraries necessary for getting this data in and out of your application. So, you donโ€™t have to write parsing code yourself, and other programmers donโ€™t have to decode a new data format when interacting with your application. For this reason, JSON is easy to use, and ubiquitous.
๐ŸŒ
SQL Shack
sqlshack.com โ€บ working-with-json-data-in-python
Working with JSON data in Python
April 2, 2021 - In order for the machine to understand this string, it needs to be converted into an object which can be then consumed by the interpreter. The process of converting a string JSON into a python object is called Deserialization and the process of converting a python object back to JSON is called Serialization.
Find elsewhere
Top answer
1 of 1
6

You don't use typing properly. Parametrizing generic types should be done using brackets notation (aka __getitem__) not parenthesis (aka instantiation).

You also don't use any feature of dataclasses in your defined classes, so you might as well drop that dependency. Or you could use it properly so that Country(**json_data) will build the whole thing; but:

  • Keys in the JSON data are not valid python identifiers, you would need to convert them;
  • You would need to handle converting children "manually" after the __init__ took place;
  • You will need to swap the order in which you define classes due to scope evaluation.

The closest we would come should be something along the lines of:

from dataclasses import dataclass
from typing import List


def convert_keys(dct):
    return {name.replace('-', '_'): value for name, value in dct.items()}


@dataclass
class Street:
    city_id: int
    name: str
    street_id: int


@dataclass
class City:
    state_id: int
    name: str
    city_id: int
    streets: List[Street]

    def __post_init__(self):
        self.streets = [Street(**convert_keys(street)) for street in self.streets]


@dataclass
class State:
    country_id: int
    name: str
    state_id: int
    cities: List[City]

    def __post_init__(self):
        self.cities = [City(**convert_keys(city)) for city in self.cities]


@dataclass
class Country:
    country_id: int
    name: str
    states: List[State]

    def __port_init__(self):
        self.states = [State(**convert_keys(state)) for state in self.states]

Initiate the call using Country(**convert_keys(json_data)). But this solution doesn't necessary feel cleaner than yours.


Now as regards to using these classes as a mean to store data into a relational database, we need to examine usage.

The DB API 2.0 tells us that you can expect to be able to:

cursor.execute('INSERT INTO Street VALUES (?, ?, ?)', (city_id, name, street_id))

Which means that we need to:

  1. Be able to convert these classes to tuples;
  2. Remove the reverse relationship that we had so much troubles parsing properly Country.states, State.cities, and City.streets).

So here we go, trying to patch our approach. We could use dataclasses.astuple to convert or objects to proper parameters for our query, but we would still need a specific parser to recursively traverse nested JSON layers. Coupling that with the invalid identifiers issue, I don't think storing the data into intermediate classes makes much sense. Instead I would rather write them directly in the database:

def parse_country(cursor, json_data):
    country_id = json_data['country-id']
    name = json_data['name']
    cursor.execute('INSERT INTO Country VALUES (?, ?)', (country_id, name))
    for state in json_data['states']:
        parse_state(cursor, state)


def parse_state(cursor, json_data):
    state_id = json_data['state-id']
    name = json_data['name']
    country_id = json_data['country-id']
    cursor.execute('INSERT INTO State VALUES (?, ?, ?)', (state_id, name, country_id))
    for city in json_data['cities']:
        parse_city(cursor, city)


def parse_city(cursor, json_data):
    city_id = json_data['city-id']
    name = json_data['name']
    state_id = json_data['state-id']
    cursor.execute('INSERT INTO City VALUES (?, ?, ?)', (city_id, name, state_id))
    for street in json_data['streets']:
        parse_street(cursor, street)


def parse_street(cursor, json_data):
    street_id = json_data['street-id']
    name = json_data['name']
    city_id = json_data['city-id']
    cursor.execute('INSERT INTO Street VALUES (?, ?, ?)', (street_id, name, city_id))

And that's pretty much your original code except the data is now in DB and not in memory. Usage being:

conn = # create appropriate DB connection here
with conn:
    parse_country(conn.cursor(), json_data)
๐ŸŒ
Reddit
reddit.com โ€บ r/sqlite โ€บ storing json into sqlite database in python
r/sqlite on Reddit: Storing JSON into sqlite database in python
October 28, 2022 -

Hello! I want insert a json file into my sqlite db. But is there a way to do that without converting json data into a string value?

cmd = โ€œโ€โ€CREATE TABLE IF NOT EXISTS Note(note TEXT)โ€โ€โ€
cursor.execute(cmd)
with open(โ€œnotes.jsonโ€) as f:
     data = str(json.load(f))
cursor.execute(โ€œINSERT INTO Note VALUES (?)โ€, ((data,))

With the code above, Iโ€™m able to insert json data as string, but is there a way, inserting json as json, not string?

๐ŸŒ
DEV Community
dev.to โ€บ ahmed__elboshi โ€บ learn-how-to-use-json-as-a-small-database-for-your-python-projects-by-building-a-hotel-accounting-system-47b4
Learn How to Use JSON as a Small Database for Your Python Projects by Building a Hotel Accounting System - DEV Community
October 3, 2024 - In our Hotel Accounting System, weโ€™ll use JSON to store information like bookings, customer details, and room charges. Pythonโ€™s built-in json module makes it easy to read and write JSON data.
๐ŸŒ
Stack Overflow
stackoverflow.com โ€บ questions โ€บ 46898834 โ€บ save-json-file-into-structured-database-with-python
sql - Save JSON file into structured database with Python - Stack Overflow
October 24, 2017 - ... Save this answer. ... Show activity on this post. You should be able to use json.dumps(json_value) to convert your JSON object into a JSON string that can be put into an sql database.
๐ŸŒ
Quora
quora.com โ€บ How-can-I-keep-JSON-data-to-my-PostgreSQL-database-using-the-Python-language
How to keep JSON data to my PostgreSQL database using the Python language - Quora
Answer: You may make use of the [code ]psycopg2[/code] and [code ]json[/code] modules . Postgres allows you to store jsons as columns having a datatype of [code ]JSON[/code] or [code ]JSONB.[/code] You may convert a simple Python dictionary to json using [code ]json.dumps.[/code] Firstly, letโ€™s...
๐ŸŒ
Reddit
reddit.com โ€บ r/python โ€บ learn how to use json as a small database for your py projects by building a hotel accounting system
r/Python on Reddit: Learn How to Use JSON as a Small Database for Your Py Projects by Building a Hotel Accounting System
October 4, 2024 -

This is the first free tutorial designed to help beginners learn how to use JSON to create a simple database for their projects.

It also prepares developers for the next two tutorials in our "Learn by Build" series, where we'll cover how to use the requests library, build asynchronous code, and work with threads.

and by time we will add extra more depth projects to enhance your pythonic skills

find tutorial in github https://github.com/rankap/learn_by_build/tree/main/tut_1_learn_json

Top answer
1 of 5
15
u/RevolutionaryAd8906 , I appreciate your good intentions in creating a beginner-friendly tutorial like this. JSON is a core technology that anyone working with modern APIs should understand, and it's important to make learning accessible for beginners. That said, your mindset of setting up JSON to be thought of as a "database" is perhaps sub-optimal ... while it's useful to teach data persistence early on, positioning JSON as a database can set beginners up for struggles down the line. I think you know this, and I see where you're going with it, but fundamentally it sets up a paradigm in the heads of new programmers that is misaligned with best practices. JSON files are easy to use and understand, making them great for small projects or single-user applications. Maybe a to-do list or managing simple configuration settings, storing user preferences, tracking small collections (e.g., book or movie lists). JSON works well if the scale is very limited. The progression from using JSON for data storage to interacting with APIs is logical and effective. However the limitations of JSON are very quickly apparent: Scalability: JSON is not designed for large datasets (like managing a hotel............. maybe suggest a different example in your HOW TO?) Performance degrades significantly with thousands of records, as the entire file must be read and written each time. RAM also becomes a limiting factor. Beyond a few thousand records, load and save times will noticeably degrade, especially with detailed records. Concurrency: JSON does not handle concurrent access. When multiple users or processes interact with the data, conflicts and errors are likely. Traditional databases manage this with record locking, transactions, and ACID compliance. Data Integrity and Security: JSON lacks built-in features for enforcing data types, constraints, or relationships. Storing sensitive information in plain-text JSON without encryption is risky (like the hotel guests credit card numbers and addresses...??) Databases like SQLite or even NoSQL options like MongoDB help mitigate these issues. Integrating databases from the beginning with new programming students can be valuable and I would argue worth doing. It seems to me you're kicking a can down the line and instilling a bad notion in their heads that they can use JSON as a flat file database. SQLite, for instance, requires minimal setup and offers skills that are more transferable to complex projects. It introduces concepts like relationships, querying, and data normalization, which can save a lot of headaches later on. I wonder if there's a middle ground where you introduce data persistence with JSON, and then transition to something like SQLite or an ORM in short order? This would help beginners avoid the pitfalls of using JSON beyond its intended use while still learning important concepts in a manageable way. Anyway ... not trying to criticize too harshly, but I think this is a misguided way to teach what JSON is all about.
2 of 5
8
Students would be far better served by studying examples of database centered applications built around SQLite (an SQL standards compliant database system embedded in the Python standard libraries). JSON is not suitable as a format for database management. None of the JSON accessibility APIs provides support for ACID transaction management, record (table nor row level) locking, schema definition and enforcement, normalization and JOINs, no indexing. This is a lot of code you'd have to write into the front end that no sane, modern, production database application would implement. Database management engines implement those features. Students are better served using tools and frameworks which offer features and APIs similar to those used in real world, production applications.
๐ŸŒ
Reddit
reddit.com โ€บ r/python โ€บ storing and querying large json array of data
r/Python on Reddit: Storing and querying large json array of data
October 13, 2022 -

I've written a list below of what I am trying to achieve. I'm just unsure of the best way to store the data, my main considerations are the speed in which I can query the data and RAM usage when running the query.

My Python script queries an API which returns a JSON array containing 1000 entries of data. The script will iterate through each page of the API until there is no more data to be retrieved. This should result in 140 million entries in the end up.

I need to store the JSON somewhere, I've be told I can lump all of it into a JSON file. I've no idea how large that would make the file or what it would mean when it comes to trying to query it, which ill need to do. I could store it in a database, something like MySQL, again not sure what this means in terms of the size of the database, time taken to query and if machine RAM would be a factor, both for MySQL and a JSON file?

Once the JSON is stored, I need to query all 140 million entries to produce a kind of summary report (was planning on writing a python script for this) (regardless of what the data is stored in, a python script will still query the 140 million entries).

After the Python script produces the report, I will store it in a MySQL database where a PHP script will pickup the data and display it on a webpage.

Thanks

๐ŸŒ
Blogger
datavu.blogspot.com โ€บ 2013 โ€บ 10 โ€บ parse-json-using-python-and-store-in.html
Datavu: Parse JSON using Python and store in MySQL
October 3, 2013 - Python has a library called "json" which will helps us to deal with the json data. Also we can use the "pprint" which is "data pretty printer" to display formatted output. Generally we will have three main steps for this kind of task, ... Store ...
๐ŸŒ
DEV Community
dev.to โ€บ kamranakhan โ€บ python-convert-json-to-sqlite-4a5n
Python Convert JSON to SQLite - DEV Community
January 27, 2023 - import sqlite3 import json conn = sqlite3.connect('example.db') conn.execute("CREATE TABLE example_table (field1 text, field2 text, field3 text);") with open('example.json', 'r') as json_file: data = json.load(json_file) for item in data: conn.execute("INSERT INTO example_table (field1, field2, field3) VALUES (?, ?, ?)", (item["field1"], item["field2"], item["field3"])) conn.commit() conn.close() This will create a SQLite database file named 'example.db' in the current directory and insert the data from 'example.json' into a table named 'example_table'.