There are multiple ways to get CSV data from URLs. From your example, namely Yahoo Finance, you can copy the Historical data link and call it in Pandas
...
HISTORICAL_URL = "https://query1.finance.yahoo.com/v7/finance/download/GOOG?period1=1582781719&period2=1614404119&interval=1d&events=history&includeAdjustedClose=true"
df = pd.read_csv(HISTORICAL_URL)
A general pattern could involve tools like requests or httpx to make a GET|POST request and then get the contents to io.
import pandas as pd
import requests
import io
url = 'https://query1.finance.yahoo.com/v7/finance/download/GOOG'
params ={'period1':1538761929,
'period2':1541443929,
'interval':'1d',
'events':'history',
'crumb':'v4z6ZpmoP98',
}
r = requests.post(url,data=params)
if r.ok:
data = r.content.decode('utf8')
df = pd.read_csv(io.StringIO(data))
To get the params, I just followed the liked and copied everything after ‘?’. Check that they match ;)
Results:

Update:
If you can see the raw csv contents directly in url, just pass the url in pd.read_csv
Example data directly from url:
data_url ='https://raw.githubusercontent.com/pandas-dev/pandas/master/pandas/tests/data/iris.csv'
df = pd.read_csv(data_url)
Answer from Prayson W. Daniel on Stack OverflowThere are multiple ways to get CSV data from URLs. From your example, namely Yahoo Finance, you can copy the Historical data link and call it in Pandas
...
HISTORICAL_URL = "https://query1.finance.yahoo.com/v7/finance/download/GOOG?period1=1582781719&period2=1614404119&interval=1d&events=history&includeAdjustedClose=true"
df = pd.read_csv(HISTORICAL_URL)
A general pattern could involve tools like requests or httpx to make a GET|POST request and then get the contents to io.
import pandas as pd
import requests
import io
url = 'https://query1.finance.yahoo.com/v7/finance/download/GOOG'
params ={'period1':1538761929,
'period2':1541443929,
'interval':'1d',
'events':'history',
'crumb':'v4z6ZpmoP98',
}
r = requests.post(url,data=params)
if r.ok:
data = r.content.decode('utf8')
df = pd.read_csv(io.StringIO(data))
To get the params, I just followed the liked and copied everything after ‘?’. Check that they match ;)
Results:

Update:
If you can see the raw csv contents directly in url, just pass the url in pd.read_csv
Example data directly from url:
data_url ='https://raw.githubusercontent.com/pandas-dev/pandas/master/pandas/tests/data/iris.csv'
df = pd.read_csv(data_url)
I routinely use this procedure
import pandas as pd
import requests
url="<URL TO DOWNLOAD.CSV>"
s=requests.get(url).content
c=pd.read_csv(s)
pandas - Saving a downloaded CSV file using Python - Stack Overflow
pandas - How to download a csv file in Python - Stack Overflow
How to download .csv file, produced by pandas df, on click of a simple button?
Fastest way to export a large dataframe to a csv file?
If you're trying to write this data to a CSV file, you can first download it using requests.get, then save each line to a CSV file.
import csv
import requests
url = 'https://www.alphavantage.co/query?function=TIME_SERIES_DAILY_ADJUSTED&symbol=MSFT&apikey=demo&datatype=csv'
response = requests.get(url)
with open('out.csv', 'w') as f:
writer = csv.writer(f)
for line in response.iter_lines():
writer.writerow(line.decode('utf-8').split(','))
Alternatively, if you have pandas installed (pip install --user pandas), you can load data by passing a URL directly.
import pandas as pd
df = pd.read_csv(url)
df.head()
timestamp open high low close adjusted_close volume dividend_amount split_coefficient
0 2019-06-19 135.00 135.93 133.81 135.69 135.69 17946556 0.0 1.0
1 2019-06-18 134.19 135.24 133.57 135.16 135.16 25908534 0.0 1.0
2 2019-06-17 132.63 133.73 132.53 132.85 132.85 14517785 0.0 1.0
3 2019-06-14 132.26 133.79 131.64 132.45 132.45 17821703 0.0 1.0
4 2019-06-13 131.98 132.67 131.56 132.32 132.32 17200848 0.0 1.0
df.to_csv('out.csv')
You can achieve it via requests as
import os
import requests
def download_file(url, filename):
''' Downloads file from the url and save it as filename '''
# check if file already exists
if not os.path.isfile(filename):
print('Downloading File')
response = requests.get(url)
# Check if the response is ok (200)
if response.status_code == 200:
# Open file and write the content
with open(filename, 'wb') as file:
# A chunk of 128 bytes
for chunk in response:
file.write(chunk)
else:
print('File exists')
You can call the function with your url and filename that you want. In your case it would be:
url = 'https://www.alphavantage.co/query?function=TIME_SERIES_DAILY_ADJUSTED&symbol=MSFT&apikey=demo&datatype=csv'
filename = 'MSFT.csv'
download_file(url, filename)
Hope this helps.
Try:
import requests
url = "https://qubeshub.org/publications/1220/serve/1/3861?el=1&download=1"
r = requests.get(url)
filename = r.headers["Content-Disposition"].split('"')[1]
with open(filename, "wb") as f_out:
print(f"Downloading {filename}")
f_out.write(r.content)
Prints:
Downloading Elephant Morphometrics and Tusk Size-originaldata-3861.csv
and saves the file.
This should download the file and parse the rows and columns into a csv file
import requests
import csv
url = "https://qubeshub.org/publications/1220/serve/1/3861?el=1&download=1"
req=requests.get(url)
rows = req.content.decode('utf-8').split("\r\n")
rows.pop()
csv_local_filename = "test.csv"
with open(csv_local_filename, 'w') as fs:
writer = csv.writer(fs, delimiter = ',')
for row in rows:
entries = row.split(',')
b=writer.writerow(entries)
You'll likely want to convert those columns into the desired types before you start working with them. The example code above leaves everything as a string.
After I run the above code I see:
>tail test.csv
2005-13,88,m,32.5,290,162.3,40
2005-13,51,m,37.5,270,113.2,40
2005-13,86,m,37.5,310,175.3,38
and
>head test.csv
Years of sample collection,Elephant ID,Sex,Estimated Age (years),shoulder Height in cm,Tusk Length in cm,Tusk Circumference in cm
1966-68,12,f,0.08,102,,
1966-68,34,f,0.08,89,,
1966-68,162,f,0.083,89,,
1966-68,292,f,0.083,92,,
I am learning web dev with flask. So far I have loved flask with its simplistic framework, Django was taking too much of my time especially since I don't main web development.
Please help me understand why I can't download my dataframe to csv using the view function in flask.
My flask app for creating an output.csv from a simple dataframe - df = pd.DataFrame({"id":[1,2,3],"name":['x','y','z']}).
Issues:
-
Linux has / whilst windows uses . How would I account for this exactly when choosing a download folder?
-
Is there a way to dynamically set a downloads folder based on the client's computer?
-
Must the view function always return a string? I am getting the error
TypeError: The view function for 'show_data' did not return a valid response. The function either returned None or ended without a return statement.
One last confusion on "downloads":
-
I have scoured through several download and upload tutorials on flask. For download, does the file actually need to exist on server, unlike producing one from pandas?
#./df_to_csv.py
from flask import *
import pandas as pd
from tkinter import filedialog as fd
app = Flask(__name__)
@app.route('/')
def open_page():
return render_template('btn.html')
@app.route('/get_data',methods=['GET','POST'])
def show_data():
if request. Method == 'POST':
f = './df_output.csv'
df = pd.DataFrame({"id":[1,2,3],"name":['x','y','z']})
# df.to_csv(f"{f}")
pth = fd.askdirectory() + "/test.csv"
return df.to_csv(pth)
if __name__ == '__main__':
app.run(debug=True)Here is my corresponding simple html form:
<!-- ./templates/btn.html --> <form action="http://localhost:5000/get_data" method="post" enctype="multipart/form-data" > <input type="submit" value="Download"> </form> Many thanks for your guidance.