In the latest version of pandas (0.19.2) you can directly pass the url
import pandas as pd
url = "https://raw.githubusercontent.com/cs109/2014_data/master/countries.csv"
c = pd.read_csv(url)
Answer from inodb on Stack OverflowIn the latest version of pandas (0.19.2) you can directly pass the url
import pandas as pd
url = "https://raw.githubusercontent.com/cs109/2014_data/master/countries.csv"
c = pd.read_csv(url)
Update: From pandas 0.19.2 you can now just pass read_csv() the url directly, although that will fail if it requires authentication.
For older pandas versions, or if you need authentication, or for any other HTTP-fault-tolerant reason:
Use pandas.read_csv with a file-like object as the first argument.
If you want to read the csv from a string, you can use
io.StringIO.For the URL
https://github.com/cs109/2014_data/blob/master/countries.csv, you gethtmlresponse, not raw csv; you should use the url given by theRawlink in the github page for getting raw csv response , which ishttps://raw.githubusercontent.com/cs109/2014_data/master/countries.csv
Example:
import pandas as pd
import io
import requests
url = "https://raw.githubusercontent.com/cs109/2014_data/master/countries.csv"
s = requests.get(url).content
c = pd.read_csv(io.StringIO(s.decode('utf-8')))
Note: in Python 2.x, the string-buffer object was StringIO.StringIO
python - Reading Data from URL into a Pandas Dataframe - Stack Overflow
python - How to read data from url to pandas dataframe - Stack Overflow
pandas - Return data from URL with Python - Stack Overflow
How to read Data from Url in python using Pandas? - Stack Overflow
pd.read_csv does not parse HTML. You might try pd.read_html, but would find that it works on <table> tags, not <pre> tags.
On inspecting the HTML content of the given URL, it is evident that the data is contained in a <pre> tag.
Use something like requests to get the page content, and BeautifulSoup4 to parse the HTML page contents (with an appropriate parsing engine, either lxml or html5lib). Then pull out the content of the <pre> tag, splitting on newlines, slicing to ignore unwanted lines, and then splitting on whitespace.
Minimal working code:
import pandas as pd
import requests
from bs4 import BeautifulSoup
url = 'https://psl.noaa.gov/cgi-bin/data/timeseries/timeseries.pl?ntype=1&var=Zonal+Wind&level=1000&lat1=50&lat2=25&lon1=-135&lon2=-65&iseas=0&mon1=0&mon2=0&iarea=0&typeout=1&Submit=Create+Timeseries'
res = requests.get(url)
# get the text from the 'pre' tag, split it on newlines
# slice off 1 head and 5 tail rows
# (inspect the contents of 'soup.find('pre').text' to determine correct values)
soup = BeautifulSoup(res.content, "html5lib")
data = soup.find('pre').text.split("\n")[1:-5]
df = pd.DataFrame([row.split() for row in data]).apply(pd.to_numeric)
df = df.set_index(df.iloc[:,0])
results in
>>> print(df.head(5))
0 1 2 3 4 5 6 7 8 9 10 11 12
0
1948 1948 0.878 0.779 0.851 0.393 0.461 0.747 0.867 0.539 -0.106 0.045 0.819 1.506
1949 1949 0.386 1.197 1.154 1.054 0.358 0.645 0.643 0.477 0.128 -0.091 1.500 0.390
1950 1950 0.674 0.973 1.640 0.821 0.572 1.002 0.635 0.196 -0.020 0.268 0.844 1.045
1951 1951 1.524 0.698 0.971 0.790 0.789 0.587 0.682 0.238 0.256 0.035 0.906 1.268
1952 1952 1.524 1.510 1.353 0.705 0.710 1.188 0.412 0.432 -0.091 0.415 0.443 1.509
and
>>> print(df.dtypes)
0 int64
1 float64
2 float64
...
12 float64
This answer is a good starting point for what you're trying to accomplish.
Its because the first one directly points to a dataset from storage in .data format but the second url points to a website (which is made up of html, css, json, etc files). You can only use pd.read_csv if you are parsing in a .csv file, and i guess a .data file too since it worked for you.
If you can find a link to the actual .data or .csv file on that website you will be able to parse it no problem. Since its a gov website, they probably will have a good file format.
If you cannot, and still need this data you will have to do some webscraping from that website (like using selenium), then you will need to store them as dataframes, and maybe preprocess it so it gets added like expected.