It would be the files listed here. https://open.fda.gov/apis/drug/event/download/
I have been using youtube videos on how to bulk download but I haven't figure it out.
Downloading JSON file through Python - Stack Overflow
How can I download multiple .json files in one go from a site?
How to download the json content using python - Stack Overflow
Generate (and download) JSON file from hosted site using server side python
Videos
Ok so I had a lot of trouble interacting with the site. I decided to just go with the webbrowser library.
import webbrowser
chrome_path="C:xxx\\Google\\Chrome\\Application\\chrome.exe"
webbrowser.register('chrome', None,webbrowser.BackgroundBrowser(chrome_path))
url = 'http://testsite/csv?date=2019-07-18'
Setting chrome to download files automatically populates my download folder from where i can automate everything else :)
You need to read the data out of the object that urlopen returns.
Try
import urllib
with urllib.request.urlopen("test.com/csv?date=2019-07-17") as f:
jsonl = f.read()
Get data from the URL and then call json.loads e.g.
Python3 example:
import urllib.request, json
with urllib.request.urlopen("http://maps.googleapis.com/maps/api/geocode/json?address=google") as url:
data = json.loads(url.read().decode())
print(data)
Python2 example:
import urllib, json
url = "http://maps.googleapis.com/maps/api/geocode/json?address=google"
response = urllib.urlopen(url)
data = json.loads(response.read())
print data
The output would result in something like this:
{
"results" : [
{
"address_components" : [
{
"long_name" : "Charleston and Huff",
"short_name" : "Charleston and Huff",
"types" : [ "establishment", "point_of_interest" ]
},
{
"long_name" : "Mountain View",
"short_name" : "Mountain View",
"types" : [ "locality", "political" ]
},
{
...
I'll take a guess that you actually want to get data from the URL:
jsonurl = urlopen(url)
text = json.loads(jsonurl.read()) # <-- read from it
Or, check out JSON decoder in the requests library.
import requests
r = requests.get('someurl')
print r.json() # if response type was set to JSON, then you'll automatically have a JSON response here...
I'm trying to download JSON files from a URL. When I've attempted to open the saved JSON files in python, I get the error:
"raise JSONDecodeError("Expecting value", s, err.value) from None".
Whenever I try opening the JSON files in a URL, I see this:
"SyntaxError: JSON.parse: unexpected character at line 1 column 1 of the JSON data"
Below is a simplified version of my code. Is there a way to download JSON files correctly?
def read_url(url):
urls = []
psfiles = []
url = url.replace(" ","%20")
req = Request(url)
a = urlopen(req).read()
soup = BeautifulSoup(a, 'html.parser')
x = (soup.find_all('a'))
for i in x:
file_name = i.extract().get_text()
url_new = url + file_name
url_new = url_new.replace(" ","%20")
if(file_name[-1]=='/' and file_name[0]!='.'):
read_url(url_new)
if url_new.endswith('json'):
urls.append(url_new)
for i in urls:
psfile = i.replace('url','')
psfiles.append(psfile)
for j in range(len(psfiles)):
urllib.request.urlretrieve("url", "path to directory"+psfiles[j])
if __name__ == '__main__':
while True:
read_url("url")
time.sleep(1800)