If you're trying to write this data to a CSV file, you can first download it using requests.get, then save each line to a CSV file.
import csv
import requests
url = 'https://www.alphavantage.co/query?function=TIME_SERIES_DAILY_ADJUSTED&symbol=MSFT&apikey=demo&datatype=csv'
response = requests.get(url)
with open('out.csv', 'w') as f:
writer = csv.writer(f)
for line in response.iter_lines():
writer.writerow(line.decode('utf-8').split(','))
Alternatively, if you have pandas installed (pip install --user pandas), you can load data by passing a URL directly.
import pandas as pd
df = pd.read_csv(url)
df.head()
timestamp open high low close adjusted_close volume dividend_amount split_coefficient
0 2019-06-19 135.00 135.93 133.81 135.69 135.69 17946556 0.0 1.0
1 2019-06-18 134.19 135.24 133.57 135.16 135.16 25908534 0.0 1.0
2 2019-06-17 132.63 133.73 132.53 132.85 132.85 14517785 0.0 1.0
3 2019-06-14 132.26 133.79 131.64 132.45 132.45 17821703 0.0 1.0
4 2019-06-13 131.98 132.67 131.56 132.32 132.32 17200848 0.0 1.0
df.to_csv('out.csv')
I am trying to write a script which downloads and saves a csv file in my local machine. Normally I would just do this
data = requests.get(URL) for this but for this case I'm not sure what goes there. I have linked the website below.
URL
How to read a CSV file from a URL with Python? - Stack Overflow
urllib - Download csv file through python (url) - Stack Overflow
Automatically download CSV from website (Python)
Use python requests to download CSV - Stack Overflow
If you're trying to write this data to a CSV file, you can first download it using requests.get, then save each line to a CSV file.
import csv
import requests
url = 'https://www.alphavantage.co/query?function=TIME_SERIES_DAILY_ADJUSTED&symbol=MSFT&apikey=demo&datatype=csv'
response = requests.get(url)
with open('out.csv', 'w') as f:
writer = csv.writer(f)
for line in response.iter_lines():
writer.writerow(line.decode('utf-8').split(','))
Alternatively, if you have pandas installed (pip install --user pandas), you can load data by passing a URL directly.
import pandas as pd
df = pd.read_csv(url)
df.head()
timestamp open high low close adjusted_close volume dividend_amount split_coefficient
0 2019-06-19 135.00 135.93 133.81 135.69 135.69 17946556 0.0 1.0
1 2019-06-18 134.19 135.24 133.57 135.16 135.16 25908534 0.0 1.0
2 2019-06-17 132.63 133.73 132.53 132.85 132.85 14517785 0.0 1.0
3 2019-06-14 132.26 133.79 131.64 132.45 132.45 17821703 0.0 1.0
4 2019-06-13 131.98 132.67 131.56 132.32 132.32 17200848 0.0 1.0
df.to_csv('out.csv')
You can achieve it via requests as
import os
import requests
def download_file(url, filename):
''' Downloads file from the url and save it as filename '''
# check if file already exists
if not os.path.isfile(filename):
print('Downloading File')
response = requests.get(url)
# Check if the response is ok (200)
if response.status_code == 200:
# Open file and write the content
with open(filename, 'wb') as file:
# A chunk of 128 bytes
for chunk in response:
file.write(chunk)
else:
print('File exists')
You can call the function with your url and filename that you want. In your case it would be:
url = 'https://www.alphavantage.co/query?function=TIME_SERIES_DAILY_ADJUSTED&symbol=MSFT&apikey=demo&datatype=csv'
filename = 'MSFT.csv'
download_file(url, filename)
Hope this helps.
Using pandas it is very simple to read a csv file directly from a url
import pandas as pd
data = pd.read_csv('https://example.com/passkey=wedsmdjsjmdd')
This will read your data in tabular format, which will be very easy to process
You need to replace open with urllib.urlopen or urllib2.urlopen.
e.g.
import csv
import urllib2
url = 'http://winterolympicsmedals.com/medals.csv'
response = urllib2.urlopen(url)
cr = csv.reader(response)
for row in cr:
print row
This would output the following
Year,City,Sport,Discipline,NOC,Event,Event gender,Medal
1924,Chamonix,Skating,Figure skating,AUT,individual,M,Silver
1924,Chamonix,Skating,Figure skating,AUT,individual,W,Gold
...
The original question is tagged "python-2.x", but for a Python 3 implementation (which requires only minor changes) see below.
Try this. Change "folder" to a folder on your machine
import os
import requests
url='https://data.toulouse-metropole.fr/api/records/1.0/download/?dataset=dechets-menagers-et-assimiles-collectes'
response = requests.get(url)
with open(os.path.join("folder", "file"), 'wb') as f:
f.write(response.content)
You can adapt an example from the docs
import urllib.request
url='https://data.toulouse-metropole.fr/api/records/1.0/download/?dataset=dechets-menagers-et-assimiles-collectes'
with urllib.request.urlopen(url) as testfile, open('dataset.csv', 'w') as f:
f.write(testfile.read().decode())
I need to be able to download a file each day from the following site:
https://fundcentres.lgim.com/uk/en/fund-centre/PMC/
There is a link that says 'Download Data' which then presents a pop-up with 'All Fund Data'.
Normally I use Python requests but this is more complicated than I've dealt with. I've searched different way to try to replicated the POST request but can't quite understand what I'm missing.
Any help would be appreciated!
This should help:
import csv
import requests
CSV_URL = 'http://samplecsvs.s3.amazonaws.com/Sacramentorealestatetransactions.csv'
with requests.Session() as s:
download = s.get(CSV_URL)
decoded_content = download.content.decode('utf-8')
cr = csv.reader(decoded_content.splitlines(), delimiter=',')
my_list = list(cr)
for row in my_list:
print(row)
Ouput sample:
['street', 'city', 'zip', 'state', 'beds', 'baths', 'sq__ft', 'type', 'sale_date', 'price', 'latitude', 'longitude']
['3526 HIGH ST', 'SACRAMENTO', '95838', 'CA', '2', '1', '836', 'Residential', 'Wed May 21 00:00:00 EDT 2008', '59222', '38.631913', '-121.434879']
['51 OMAHA CT', 'SACRAMENTO', '95823', 'CA', '3', '1', '1167', 'Residential', 'Wed May 21 00:00:00 EDT 2008', '68212', '38.478902', '-121.431028']
['2796 BRANCH ST', 'SACRAMENTO', '95815', 'CA', '2', '1', '796', 'Residential', 'Wed May 21 00:00:00 EDT 2008', '68880', '38.618305', '-121.443839']
['2805 JANETTE WAY', 'SACRAMENTO', '95815', 'CA', '2', '1', '852', 'Residential', 'Wed May 21 00:00:00 EDT 2008', '69307', '38.616835', '-121.439146']
[...]
Related question with answer: https://stackoverflow.com/a/33079644/295246
Edit: Other answers are useful if you need to download large files (i.e. stream=True).
To simplify these answers, and increase performance when downloading a large file, the below may work a bit more efficiently.
import requests
from contextlib import closing
import csv
from codecs import iterdecode
url = "http://download-and-process-csv-efficiently/python.csv"
with closing(requests.get(url, stream=True)) as r:
reader = iterdecode(csv.reader(r.iter_lines(), 'utf-8'),
delimiter=',',
quotechar='"')
for row in reader:
print(row)
By setting stream=True in the GET request, when we pass r.iter_lines() to csv.reader(), we are passing a generator to csv.reader(). By doing so, we enable csv.reader() to lazily iterate over each line in the response with for row in reader.
This avoids loading the entire file into memory before we start processing it, drastically reducing memory overhead for large files.
If you have a direct link to the URL you can just enter the URL directly in pandas to get the file:
import pandas as pd
import io
import requests
url="https://raw.githubusercontent.com/cs109/2014_data/master/countries.csv"
s=requests.get(url).content
c=pd.read_csv(io.StringIO(s.decode('utf-8')))
Code is from this question: Pandas read_csv from url
Have a look at that as well maybe :)
import pandas as pd
data = pd.read_csv("https://abc/xyz.csv")
For more, check the documentation:
https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html
You can use
csv.writerows()
to write the csv file. Check this site for more. https://www.programiz.com/python-programming/writing-csv-files
def main():
import requests
url = "https://coronavirus.data.gov.uk/downloads/csv/coronavirus-deaths_latest.csv"
with requests.get(url, stream=True) as response:
response.raise_for_status()
with open("deaths_latest.csv", "wb") as file:
for chunk in response.iter_content(chunk_size=8192):
file.write(chunk)
file.flush()
print("Done")
return 0
if __name__ == "__main__":
import sys
sys.exit(main())
So I am working on a pretty large project at work where I want to automate the entire process of checking through thousands of lines of transactions every month. I'd like to also download and save the files that we need for it within the program.
The website is a secure financial reporting site that requires a username and password to enter. I'm currently using the requests library. When I do a simple get request on the URL i get back a lot of what I believe are javascript objects. Perhaps I am mistaken, but I believe i need the specific web address of the csv download file.
I watched and read a number of tutorials and most started by "copy link address" . However, this only gives me the address of the image of the CSV icon.
Basically, Im pretty lost here. The website is dynamic, you can add or subtract columns, filter, add excel type formulas and then export whatever your results are.
So I have 3 questions, how do I get the appropriate link to download the file? am I just completely out of my depth here on how this whole thing works? And if so are there any good tutorials to start learning enough to get this to work.