So I've got a JSON file I want to download from a website.

The data comes in a URL that ends with a 6 digit ID number (unfortunately I cannot find a logic to the ID but it's not a single increment system, so I'd like to be able to check like the next 10,000 ID's say once a week).

I'd then like to pull parts of the JSON files into some kind of spreadsheet (dumping to a CSV is fine). There is a lot of data I don't need/want in these files. I know how to pull the data into excel but I feel like there will be a better way to do this automagically.

It would be the files listed here. https://open.fda.gov/apis/drug/event/download/

I have been using youtube videos on how to bulk download but I haven't figure it out.

I'm trying to download JSON files from a URL. When I've attempted to open the saved JSON files in python, I get the error:

"raise JSONDecodeError("Expecting value", s, err.value) from None".

Whenever I try opening the JSON files in a URL, I see this:

"SyntaxError: JSON.parse: unexpected character at line 1 column 1 of the JSON data"

Below is a simplified version of my code. Is there a way to download JSON files correctly?

def read_url(url):

    urls = []
    psfiles = []
    
    url = url.replace(" ","%20")
    req = Request(url)
    a = urlopen(req).read()
    soup = BeautifulSoup(a, 'html.parser')
    x = (soup.find_all('a'))
    for i in x:
        file_name = i.extract().get_text()
        url_new = url + file_name
        url_new = url_new.replace(" ","%20")
        if(file_name[-1]=='/' and file_name[0]!='.'):
            read_url(url_new)
        if url_new.endswith('json'):
            urls.append(url_new)
      
    for i in urls:
        psfile = i.replace('url','')
        psfiles.append(psfile)
 
         
    for j in range(len(psfiles)):
        urllib.request.urlretrieve("url", "path to directory"+psfiles[j])


if __name__ == '__main__':
    while True:
        read_url("url")
        time.sleep(1800)