You should just be able to use Pandas as follows:
import pandas as pd
with open('PeshVsQuetta.json', encoding='utf-8-sig') as f_input:
df = pd.read_json(f_input)
df.to_csv('PeshVsQuetta.csv', encoding='utf-8', index=False)
This assumes that your JSON file contains a BOM at the start. For the data you have given above, this produces the following CSV file:
date_time,tweet_content,tweet_id,user_id
2018-03-20 18:38:35,RT @PTISPOfficial: پاکستان تحریک انصاف کے وائس چیئرمین شاہ محمود قریشی بغیر کسی پروٹوکول کے پاکستان سپر لیگ کا میچ دیکھنے کے لئے اسٹیڈیم م…,976166125502427136,938118866135343104
2018-03-20 18:38:35,"At last, Pakistan Have Witnessed The Most Thrilling Match Of Cricket In Pakistan, The Home.
#PZvQG
#ABC",976166125535973378,959235642
2018-03-20 18:38:35,"RT @thePSLt20: SIX! 19.4 Liam Dawson to Anwar Ali
Watch ball by ball highlights at (link removed)
#PZvQG #HBLPSL #PSL2018 @_crici…",976166126202839040,395163528
2018-03-20 18:38:35,RT @JeremyMcLellan: Rumor has it Amir Liaquat isn’t allowed to play in #PSL2018 because he keeps switching teams every week.,976166126483902466,3117825702
2018-03-20 18:38:35,RT @daniel86cricket: Peshawar beat Quetta by 1 run in one of the best T20 thrillers. PSL played in front of full house in Lahore Pakistan i…,976166126559354880,3310967346
2018-03-20 18:38:35,"I wanted a super over😭
#PZvQG",976166126836178944,701494826194354179
2018-03-20 18:38:35,RT @hinaparvezbutt: Congratulations Peshawar Zalmi over great win but Quetta Gladiators won our hearts ♥️ #PZvQG,976166126685171713,347132028
2018-03-20 18:38:35,"RT @walterMiitty: It's harder than I thought to tell the truth
It's gonna leave you in pieces
All alone with your demons
And I know that we…",976166126924201986,3461853618
Note: some of your fields contain newlines, so the output might look a bit odd. An application reading this though will handle it correctly (as long as you tell it the encoding is UTF-8 when importing it)
Answer from Martin Evans on Stack OverflowDownload a csv from url and make it a dataframe python pandas - Stack Overflow
How to download .csv file, produced by pandas df, on click of a simple button?
python - Download csv file and convert to JSON - Stack Overflow
python - Using Pandas to convert csv to Json - Stack Overflow
There are multiple ways to get CSV data from URLs. From your example, namely Yahoo Finance, you can copy the Historical data link and call it in Pandas
...
HISTORICAL_URL = "https://query1.finance.yahoo.com/v7/finance/download/GOOG?period1=1582781719&period2=1614404119&interval=1d&events=history&includeAdjustedClose=true"
df = pd.read_csv(HISTORICAL_URL)
A general pattern could involve tools like requests or httpx to make a GET|POST request and then get the contents to io.
import pandas as pd
import requests
import io
url = 'https://query1.finance.yahoo.com/v7/finance/download/GOOG'
params ={'period1':1538761929,
'period2':1541443929,
'interval':'1d',
'events':'history',
'crumb':'v4z6ZpmoP98',
}
r = requests.post(url,data=params)
if r.ok:
data = r.content.decode('utf8')
df = pd.read_csv(io.StringIO(data))
To get the params, I just followed the liked and copied everything after ‘?’. Check that they match ;)
Results:

Update:
If you can see the raw csv contents directly in url, just pass the url in pd.read_csv
Example data directly from url:
data_url ='https://raw.githubusercontent.com/pandas-dev/pandas/master/pandas/tests/data/iris.csv'
df = pd.read_csv(data_url)
I routinely use this procedure
import pandas as pd
import requests
url="<URL TO DOWNLOAD.CSV>"
s=requests.get(url).content
c=pd.read_csv(s)
I am learning web dev with flask. So far I have loved flask with its simplistic framework, Django was taking too much of my time especially since I don't main web development.
Please help me understand why I can't download my dataframe to csv using the view function in flask.
My flask app for creating an output.csv from a simple dataframe - df = pd.DataFrame({"id":[1,2,3],"name":['x','y','z']}).
Issues:
-
Linux has / whilst windows uses . How would I account for this exactly when choosing a download folder?
-
Is there a way to dynamically set a downloads folder based on the client's computer?
-
Must the view function always return a string? I am getting the error
TypeError: The view function for 'show_data' did not return a valid response. The function either returned None or ended without a return statement.
One last confusion on "downloads":
-
I have scoured through several download and upload tutorials on flask. For download, does the file actually need to exist on server, unlike producing one from pandas?
#./df_to_csv.py
from flask import *
import pandas as pd
from tkinter import filedialog as fd
app = Flask(__name__)
@app.route('/')
def open_page():
return render_template('btn.html')
@app.route('/get_data',methods=['GET','POST'])
def show_data():
if request. Method == 'POST':
f = './df_output.csv'
df = pd.DataFrame({"id":[1,2,3],"name":['x','y','z']})
# df.to_csv(f"{f}")
pth = fd.askdirectory() + "/test.csv"
return df.to_csv(pth)
if __name__ == '__main__':
app.run(debug=True)Here is my corresponding simple html form:
<!-- ./templates/btn.html --> <form action="http://localhost:5000/get_data" method="post" enctype="multipart/form-data" > <input type="submit" value="Download"> </form> Many thanks for your guidance.
Try this:
import pandas as pd
def parse_nested_json(json_d):
result = {}
for key in json_d.keys():
if not isinstance(json_d[key], dict):
result[key] = json_d[key]
else:
result.update(parse_nested_json(json_d[key]))
return result
json_data = pd.read_json("my_json_file.json")
json_list = [j[1][0] for j in json_data.iterrows()]
parsed_list = [parse_nested_json(j) for j in json_list]
result = pd.DataFrame(parsed_list)
result.to_csv("my_csv_file.csv", index=False)
Update(12/3/2018):
I read the docs, there is a convenient way:
from pandas.io.json import json_normalize
df = json_normalize(data["profile_set"])
df.to_csv(...)
Update(11/7/2021):
pandas.io.json.json_normalize is deprecated, use pandas.json_normalize instead
from pandas import json_normalize
df = json_normalize(data["profile_set"])
df.to_csv(...)
If you load the json, and then feed the Dataframe the part of the json needed, then you can get it like:
Code:
def json_csv(filename):
with open(filename) as data_file:
data = json.load(data_file)
return pd.DataFrame(data['profile_set'])
Test Code:
print(json_csv('file1'))
ResultsL
created_datetime doc_type first_name key last_name \
0 2018-01-06T12:52:09 PROFILE Robert 123 John
1 2018-01-07T11:32:07 PROFILE Lily 456 Hubert
mem_list middle_name
0 {'mem_num': '10001', 'current_flag': 'Y', 'mem... []
1 {'mem_num': '10002', 'current_flag': 'Y', 'mem... []
I have a script that currently queries json from an api, and store the individual records in a list. Then uses pandas to convert the json into a dataframe so that I can export to a csv. Is this the best way to handle this? Or should I be implementing some other method? Example code below:
json_data = [
{
'column1' : 'value1',
'column2' : 'value2'
},
{
'column1' : 'value1',
'column2' : 'value2'
}
]
df = pd.DataFrame.from_records(json_data)
df.to_csv('my_cool_csv.csv')