If you are using Django 1.8 you can create your own model field that can store a json. This class will make sure that you have the right JSON format as well.
import json
from django.db import models
class JsonField(models.TextField):
"""
Stores json-able python objects as json.
"""
def get_db_prep_value(self, value, connection, prepared=False):
try:
return json.dumps(value)
except TypeError:
BAD_DATA.error(
"cannot serialize %s to store in a JsonField", str(value)
)
return ""
def from_db_value(self, value, expression, connection, context):
if value == "":
return None
try:
return json.loads(value)
except TypeError:
BAD_DATA.error("cannot load dictionary field -- type error")
return None
Answer from nael on Stack OverflowIf you are using Django 1.8 you can create your own model field that can store a json. This class will make sure that you have the right JSON format as well.
import json
from django.db import models
class JsonField(models.TextField):
"""
Stores json-able python objects as json.
"""
def get_db_prep_value(self, value, connection, prepared=False):
try:
return json.dumps(value)
except TypeError:
BAD_DATA.error(
"cannot serialize %s to store in a JsonField", str(value)
)
return ""
def from_db_value(self, value, expression, connection, context):
if value == "":
return None
try:
return json.loads(value)
except TypeError:
BAD_DATA.error("cannot load dictionary field -- type error")
return None
I found a way to store JSON data into DB. Since I'm accessing nodes from remote service which returns a list of nodes on every request, I need to build proper json to store/retrieve from db.
Say API returned json text as : '{"cursor": null, "nodes" = [{"name": "Test1", "value: 1}, {"name": "Test2", "value: 2}, ...]}'
So, first we need to access nodes list as:
data = json.loads(api_data)
nodes = data['nodes']
Now for 1st entry into DB column we need to do following:
str_data = json.dumps({"nodes": nodes})
So, str_data would return a valid string/buffer, which we can store into DB with a "nodes" key.
For 2nd or successive entries into DB column, we will do following:
# get data string from DB column and load into json
db_data = json.loads(db_col_data)
# get new/latest 'nodes' data from api as explained above
# append this data to 'db_data' json as
latest_data = db_data["nodes"] + new_api_nodes
# now add this data back to column after json.dumps()
db_col_data = json.dumps(latest_data)
# add to DB col and DB commit
It is a proper way to load/dump data from DB while adding/removing json and keeping proper format.
Thanks!
python - How can I use a JSON file such as a database to store new and old objects? - Stack Overflow
python - Store nested json repsonses in relational database - Code Review Stack Exchange
Storing JSON into sqlite database in python
python - Keeping JSON in database - Software Engineering Stack Exchange
Hello! I want insert a json file into my sqlite db. But is there a way to do that without converting json data into a string value?
cmd = โโโCREATE TABLE IF NOT EXISTS Note(note TEXT)โโโ cursor.execute(cmd)
with open(โnotes.jsonโ) as f:
data = str(json.load(f))cursor.execute(โINSERT INTO Note VALUES (?)โ, ((data,))
With the code above, Iโm able to insert json data as string, but is there a way, inserting json as json, not string?
This is the first free tutorial designed to help beginners learn how to use JSON to create a simple database for their projects.
It also prepares developers for the next two tutorials in our "Learn by Build" series, where we'll cover how to use the requests library, build asynchronous code, and work with threads.
and by time we will add extra more depth projects to enhance your pythonic skills
find tutorial in github https://github.com/rankap/learn_by_build/tree/main/tut_1_learn_json
I've written a list below of what I am trying to achieve. I'm just unsure of the best way to store the data, my main considerations are the speed in which I can query the data and RAM usage when running the query.
My Python script queries an API which returns a JSON array containing 1000 entries of data. The script will iterate through each page of the API until there is no more data to be retrieved. This should result in 140 million entries in the end up.
I need to store the JSON somewhere, I've be told I can lump all of it into a JSON file. I've no idea how large that would make the file or what it would mean when it comes to trying to query it, which ill need to do. I could store it in a database, something like MySQL, again not sure what this means in terms of the size of the database, time taken to query and if machine RAM would be a factor, both for MySQL and a JSON file?
Once the JSON is stored, I need to query all 140 million entries to produce a kind of summary report (was planning on writing a python script for this) (regardless of what the data is stored in, a python script will still query the 140 million entries).
After the Python script produces the report, I will store it in a MySQL database where a PHP script will pickup the data and display it on a webpage.
Thanks
use json.dumps(json_value) to convert your json object(python object) in a json string that you can insert in a text field in mysql
http://docs.python.org/library/json.html
The most straightforward way to insert a python map into a MySQL JSON field...
python_map = { "foo": "bar", [ "baz", "biz" ] }
sql = "INSERT INTO your_table (json_column_name) VALUES (%s)"
cursor.execute( sql, (json.dumps(python_map),) )