Use python requests to download CSV - Stack Overflow
Download .csv file using python - Stack Overflow
Script to download a csv file from a website
Automatically download CSV from website (Python)
This should help:
import csv
import requests
CSV_URL = 'http://samplecsvs.s3.amazonaws.com/Sacramentorealestatetransactions.csv'
with requests.Session() as s:
download = s.get(CSV_URL)
decoded_content = download.content.decode('utf-8')
cr = csv.reader(decoded_content.splitlines(), delimiter=',')
my_list = list(cr)
for row in my_list:
print(row)
Ouput sample:
['street', 'city', 'zip', 'state', 'beds', 'baths', 'sq__ft', 'type', 'sale_date', 'price', 'latitude', 'longitude']
['3526 HIGH ST', 'SACRAMENTO', '95838', 'CA', '2', '1', '836', 'Residential', 'Wed May 21 00:00:00 EDT 2008', '59222', '38.631913', '-121.434879']
['51 OMAHA CT', 'SACRAMENTO', '95823', 'CA', '3', '1', '1167', 'Residential', 'Wed May 21 00:00:00 EDT 2008', '68212', '38.478902', '-121.431028']
['2796 BRANCH ST', 'SACRAMENTO', '95815', 'CA', '2', '1', '796', 'Residential', 'Wed May 21 00:00:00 EDT 2008', '68880', '38.618305', '-121.443839']
['2805 JANETTE WAY', 'SACRAMENTO', '95815', 'CA', '2', '1', '852', 'Residential', 'Wed May 21 00:00:00 EDT 2008', '69307', '38.616835', '-121.439146']
[...]
Related question with answer: https://stackoverflow.com/a/33079644/295246
Edit: Other answers are useful if you need to download large files (i.e. stream=True).
To simplify these answers, and increase performance when downloading a large file, the below may work a bit more efficiently.
import requests
from contextlib import closing
import csv
from codecs import iterdecode
url = "http://download-and-process-csv-efficiently/python.csv"
with closing(requests.get(url, stream=True)) as r:
reader = iterdecode(csv.reader(r.iter_lines(), 'utf-8'),
delimiter=',',
quotechar='"')
for row in reader:
print(row)
By setting stream=True in the GET request, when we pass r.iter_lines() to csv.reader(), we are passing a generator to csv.reader(). By doing so, we enable csv.reader() to lazily iterate over each line in the response with for row in reader.
This avoids loading the entire file into memory before we start processing it, drastically reducing memory overhead for large files.
The file you're trying to download is in UTF-16 format, and the CSV module is not designed for that. You need to decode it from UTF-16 into some other format. For example:
import csv
import codecs
import urllib2
url = 'http://wildfire.alberta.ca/reports/activedd.csv'
response = urllib2.urlopen(url)
cr = csv.reader([x.strip() for x in codecs.iterdecode(response, 'UTF-16')])
data = [x for x in cr]
# Manipulate the data here
# Now to save the CSV:
with open('outputfile.csv', 'wb') as output:
writer = csv.writer(output)
writer.writerows(data)
If you just need to download the file, and not manipulate it, there are better ways (see minitoto's answer).
This is an example, and the newlines have to be stripped manually for this to work properly, so I'm sure there's probably a better way, but that's the main issue
I guess the easiest way is to use urlretrieve:
import urllib
url = 'http://wildfire.alberta.ca/reports/activedd.csv'
urllib.urlretrieve(url, "activedd.csv")
I am trying to write a script which downloads and saves a csv file in my local machine. Normally I would just do this
data = requests.get(URL) for this but for this case I'm not sure what goes there. I have linked the website below.
URL