Use the requests module: import requests r = requests.get(https://api.fda.gov/drug/label.json) https://requests.readthedocs.io/en/master/user/quickstart/ The website you linked also has API documentation, check it out. Answer from the_shell_man_ on reddit.com
๐ŸŒ
GitHub
github.com โ€บ justinbalaguer โ€บ python-json-url-download
GitHub - justinbalaguer/python-json-url-download: python script for downloading files from json data url
python script for downloading files from json data url - justinbalaguer/python-json-url-download
Author ย  justinbalaguer
๐ŸŒ
Reddit
reddit.com โ€บ r/learnpython โ€บ trouble downloading json files from a url
r/learnpython on Reddit: Trouble downloading JSON files from a URL
April 14, 2022 -

I'm trying to download JSON files from a URL. When I've attempted to open the saved JSON files in python, I get the error:

"raise JSONDecodeError("Expecting value", s, err.value) from None".

Whenever I try opening the JSON files in a URL, I see this:

"SyntaxError: JSON.parse: unexpected character at line 1 column 1 of the JSON data"

Below is a simplified version of my code. Is there a way to download JSON files correctly?

def read_url(url):

    urls = []
    psfiles = []
    
    url = url.replace(" ","%20")
    req = Request(url)
    a = urlopen(req).read()
    soup = BeautifulSoup(a, 'html.parser')
    x = (soup.find_all('a'))
    for i in x:
        file_name = i.extract().get_text()
        url_new = url + file_name
        url_new = url_new.replace(" ","%20")
        if(file_name[-1]=='/' and file_name[0]!='.'):
            read_url(url_new)
        if url_new.endswith('json'):
            urls.append(url_new)
      
    for i in urls:
        psfile = i.replace('url','')
        psfiles.append(psfile)
 
         
    for j in range(len(psfiles)):
        urllib.request.urlretrieve("url", "path to directory"+psfiles[j])


if __name__ == '__main__':
    while True:
        read_url("url")
        time.sleep(1800)
Top answer
1 of 2
5
here's a code snippet with requests that works >>> import requests >>> >>> url = 'https://old.reddit.com/r/learnpython/comments/v02hmv/trouble_downloading_json_files_from_a_url.json' >>> >>> response = requests.get(url) >>> json_data = response.json() >>> print(json_data) some of the output: [{'kind': 'Listing', 'data': {'after': None, 'dist': 1, 'modhash': '', 'geo_filter': '', 'children': [{'kind': 't3', 'data': {'approved_at_utc': None, 'subreddit': 'learnpython', 'selftext': 'I\'m trying to download JSON files from a URL. When I\'ve attempted to open the saved JSON files in python, I get the error:\n\n"raise JSONDecodeError("Expecting value", s, err.value) from None".\n\nWhenever I try opening the JSON files in a URL, I see this:\n\n"SyntaxError: JSON.parse: [...] if that doesn't work at your URL it's probably not valid a json string in the response you're getting
2 of 2
2
Imho you should start by actively debugging the stages in your code. The prime place to start is to check what is actually downloaded, eg add html = urlopen(req).read() # avoid confusing names like 'a' with open("base.html", "w") as fp: fp.write(html) to check in an editor or browser if that's a functional webpage that includes your data. Then add prints to the data extracted in BS, so links = (soup.find_all('a')) # again, more useful name print("found", len(links), "urls") for link in links: # you see now it reads more like English than 'for i in x' file_name = link.extract().get_text() print("file_name", file_name) url_new = url + file_name url_new = url_new.replace(" ","%20") if(file_name[-1]=='/' and file_name[0]!='.'): print("calling read_url with", url_new) read_url(url_new) else: print("skipping read_url") # or whatever you would call this case if url_new.endswith('json'): print("appending", url_new) urls.append(url_new) else: print("not json", url_new) same for the rest. Then you can actually know what parts are working and if it behaves like you're expecting.
๐ŸŒ
Power CMS Technology
powercms.in โ€บ article โ€บ how-get-json-data-remote-url-python-script
How to get json data from remote url into Python script | Power CMS Technology
August 17, 2016 - import urllib, json url = "http://maps.googleapis.com/maps/api/geocode/json?address=googleplex&sensor=false" response = urllib.urlopen(url) data = json.loads(response.read()) print data
๐ŸŒ
Delft Stack
delftstack.com โ€บ home โ€บ howto โ€บ python โ€บ python get json from url
How to Get JSON From URL in Python | Delft Stack
February 2, 2024 - If you print the type of the data variable, then it will be of type <class 'list'> because in this case, the JSON response starts with square brackets [] and in Python, lists start with square brackets. Now that we have parsed the JSON data, we are ready to access the individual values which we want using the data variable. To access the details of the first user, like Name and Address, we can do the following. import json import requests url = requests.get("https://jsonplaceholder.typicode.com/users") text = url.text data = json.loads(text) user = data[0] print(user["name"]) address = user["address"] print(address)
๐ŸŒ
Finxter
blog.finxter.com โ€บ home โ€บ learn python blog โ€บ how to get json from url in python?
How to Get JSON from URL in Python? - Be on the Right Side of Change
May 20, 2021 - import urllib.request import json # Bitcoin Genesis Block Transactions your_url = 'https://blockchain.info/rawaddr/12c6DSiU4Rq3P4ZxziKxzrL5LmMBrzjrJX' with urllib.request.urlopen(your_url) as url: data = json.loads(url.read().decode()) print(data)
Find elsewhere
๐ŸŒ
Real Python
realpython.com โ€บ python-download-file-from-url
How to Download Files From URLs With Python โ€“ Real Python
January 25, 2025 - This offers the possibility for more advanced operations in web scraping and interacting with web APIs. To download a file from a URL using the urllib package, you can call urlretrieve() from the urllib.request module.
๐ŸŒ
GeeksforGeeks
geeksforgeeks.org โ€บ how-to-read-a-json-response-from-a-link-in-python
How to read a JSON response from a link in Python? - GeeksforGeeks
February 24, 2021 - This library helps to open the URL and read the JSON response from the web. To use this library in python and fetch JSON response we have to import the json and urllib in our code, The json.loads() method returns JSON object.
๐ŸŒ
Python Basics
pythonbasics.org โ€บ json
Working With JSON Data in Python - Python Tutorial
You can parse a JSON object with python. The object will then be converted to a python object. ... You can get JSON objects directly from the web and convert them to python objects.
๐ŸŒ
MyCleverAI
mycleverai.com โ€บ it-questions โ€บ how-can-i-download-a-json-file-from-a-link
How can I download a JSON file from a link?
A Blob object is created to hold the string data, and a download link is dynamically generated and clicked to trigger the download. ... - If you're working in a programming environment, you can use libraries to fetch and save JSON data. Hereโ€™s a Python example using the requests library: ...
๐ŸŒ
GitHub
gist.github.com โ€บ sirleech โ€บ 2660189
Python Read JSON from HTTP Request of URL ยท GitHub
To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters ... This is very helpful. Thank you. ... Thanks, i use your first example. I think the content will be cached. Any solution for that? ... I would suggest you don't create a variable named 'json' that overloads the class named 'json' in line 19. ... url="http://api.open-notify.org/iss-pass.json" r=requests.get(url) t=json.loads(r.content) for i in range(len(t)): print(t[i]['state'])
Top answer
1 of 2
2

Don't use beautiful soup to process a json http response. Use something like requests:

url = "https://www.daraz.pk/womens-kurtas-shalwar-kameez/?pathInfo=womens-kurtas-shalwar-kameez&page=2&YII_CSRF_TOKEN=31eb0a5d28f4dde909d3233b5a0c23bd03348f69&more_products=true"
header = {'x-requested-with': 'XMLHttpRequest'}
t = requests.get(url, headers=True)
newDictionary=json.loads(t)
print (newDictionary)

The beautiful soup object can't be parsed with json.loads() that way.

If you have HTML data on some of those json keys then you can use beautiful soup to parse those string values individually. If you have a key called content on your json, containing html, you can parse it like so:

BeautifulSoup(newDictionary.content, "lxml")

You may need to experiment with different parsers, if you have fragmentary html.

2 of 2
0

The following is an example of how to use various JSON data that has been loaded as an object with json.loads().

Working Example โ€” Tested with Python 2.6.9 and 2.7.10 and 3.3.5 and 3.5.0

import json

json_data = '''
{
    "array": [
        1,
        2,
        3
    ],
    "boolean": true,
    "null": null,
    "number": 123,
    "object": {
        "a": "b",
        "c": "d",
        "e": "f"
    },
    "string": "Hello World"
}
'''

data = json.loads(json_data)

list_0 = [
    data['array'][0],
    data['array'][1],
    data['array'][2],
    data['boolean'],
    data['null'],
    data['number'],
    data['object']['a'],
    data['object']['c'],
    data['object']['e'],
    data['string']
]

print('''
array value 0           {0}
array value 1           {1}
array value 2           {2}
boolean value           {3}
null value              {4}
number value            {5}
object value a value    {6}
object value c value    {7}
object value e value    {8}
string value            {9}
'''.format(*list_0))

Output

array value 0           1
array value 1           2
array value 2           3
boolean value           True
null value              None
number value            123
object value a value    b
object value c value    d
object value e value    f
string value            Hello World
๐ŸŒ
Python.org
discuss.python.org โ€บ python help
How can I download multiple .json files in one go from a site? - Python Help - Discussions on Python.org
August 12, 2023 - Hi all, I am using chatbase.co https://www.chatbase.co/ where I am hosting several chatbots. For each chatbot it is possible to download a .json file specified by a date interval which stores the chat conversation. Rigโ€ฆ
๐ŸŒ
Stack Abuse
stackabuse.com โ€บ how-to-get-json-from-a-url-in-python
How to Get JSON from a URL in Python
February 14, 2025 - In this article, we'll explore how to use Python to retrieve JSON data from a URL. We'll cover two popular libraries - requests and urllib, and show how to extract and parse the JSON data using Python's built-in json module.
๐ŸŒ
Russelwebber
russelwebber.github.io โ€บ xlslim-docs โ€บ html โ€บ samples โ€บ read_json_from_web.html
Read JSON data from a URL โ€” xlSlim v1.0 documentation
# Fetch JSON data from a web site import urllib.request import json def fetch_json(url): """GET data from the url and assume the response is JSON.""" with urllib.request.urlopen(url) as response: data = json.loads(response.read()) columns = ["userId", "id", "title", "completed"] result = [columns] for d in data: result.append([d[c] for c in columns]) return result if __name__ == "__main__": print(fetch_json("https://jsonplaceholder.typicode.com/todos")) ... All the Python code and Excel files shown are available from github in the xlslim-code-samples repo.
๐ŸŒ
Pluralsight
pluralsight.com โ€บ blog โ€บ guides
Importing Data from a JSON Resource with Python | Online Courses, ...
Although there are third party tools that can perform the same task (such as the requests module), in this guide we will use a utility that is available out of the box with Python: the urllib.request library. First, we will import the library and the json module: