even looking through the code, it is not obvious to me what the possible exceptions might be.

Here's my attempt at looking through it:

def json(self, **kwargs: typing.Any) -> typing.Any:
    if self.charset_encoding is None and self.content and len(self.content) > 3:
        encoding = guess_json_utf(self.content)
        if encoding is not None:
            return jsonlib.loads(self.content.decode(encoding), **kwargs)
    return jsonlib.loads(self.text, **kwargs)

• self.content can raise ResponseNotRead if (as it sounds like) the response has not been read yet. However, this is almost certainly due to a logical error in the code rather than any meaningful problem at runtime worth detecting; so there is no good reason to catch this. (It also wouldn't happen with the most straightforward use cases, such as the one shown in the documentation.) Otherwise, self.content will be a bytes, so len will work.

• self.charset_encoding will either return None (if there is no corresponding data in the response header) or else eventually use the standard library email.message.Message.get_content_charset to parse a content type from the response header. The latter is not documented to raise any exceptions; so there should not be any exception from accessing this value.

• guess_json_utf will necessarily be passed a bytes that is at least 4 bytes long. Nothing in its logic should be able to fail under those conditions. Either None or a string will be returned.

• If jsonlib.loads is called using self.content.decode(encoding), then encoding was necessarily not None, and is a valid encoding name returned from guess_json_utf However, it's possible that the self.content (bytes data) is not valid data (i.e., convertible to text using the guessed text encoding). This would cause UnicodeDecodeError.

• Otherwise, it's called with self.text. If the underlying _text was set before, it will be a string and gets returned. If there is no content in the response, an empty string is returned. Otherwise, decoding is attempted using a default text encoder with a "replace" error-handling policy. There again shouldn't be anything that could cause the text encoding name to be invalid, so again this can only raise UnicodeDecodeError, and even that shouldn't be possible with the "replace" error-handling policy.

• Finally: jsonlib.loads is simply json.loads (i.e., the standard library json). if the code gets this far, json.loads will definitely be given a string to load; all possible issues with the string contents will be reported as JSONDecodeError.

tl;dr: the possible exceptions that make sense to catch are JSONDecodeError (the response is not valid JSON - this includes e.g. an empty response) and UnicodeDecodeError (the response is corrupt in some way, for example it mixes bytes intended to encode text in two different ways, or it's supposed to be UTF-8 but contains bytes that are illegal in that encoding; or it's encoded using a non-UTF scheme, such as Latin-1, in a way that is incompatible with the corresponding guessed UTF scheme, and doesn't advertise the encoding in the header).

Looking at the documentation for this API it seems the only responses are in JSON format, so receiving HTML is strange. To increase the likelihood of receiving a JSON response, you can set the 'Accept' header to 'application/json'.

I tried querying this API many times with parameters and did not encounter a JSONDecodeError. This error is likely the result of another error on the server side. To handle it, except a json.decoder.JSONDecodeError in addition to the ConnectionError error you currently except and handle this error in the same way as the ConnectionError.

Here is an example with all that in mind:

import requests, json, time, random

def get_submission_records(client, since, try_number=1):
    url = 'http://reymisterio.net/data-dump/api.php/submission?filter[]=form,cs,'+client+'&filter[]=date,cs,'+since
    headers = {'Accept': 'application/json'}
    try:
        response = requests.get(url, headers=headers).json()
    except (requests.exceptions.ConnectionError, json.decoder.JSONDecodeError):
        time.sleep(2**try_number + random.random()*0.01) #exponential backoff
        return get_submission_records(client, since, try_number=try_number+1)
    else:
        return response['submission']['records']

I've also wrapped this logic in a recursive function, rather than using while loop because I think it is semantically clearer. This function also waits before trying again using exponential backoff (waiting twice as long after each failure).

Edit: For Python 2.7, the error from trying to parse bad json is a ValueError, not a JSONDecodeError

import requests, time, random

def get_submission_records(client, since, try_number=1):
    url = 'http://reymisterio.net/data-dump/api.php/submission?filter[]=form,cs,'+client+'&filter[]=date,cs,'+since
    headers = {'Accept': 'application/json'}
    try:
        response = requests.get(url, headers=headers).json()
    except (requests.exceptions.ConnectionError, ValueError):
        time.sleep(2**try_number + random.random()*0.01) #exponential backoff
        return get_submission_records(client, since, try_number=try_number+1)
    else:
        return response['submission']['records']

so just change that except line to include a ValueError instead of json.decoder.JSONDecodeError.

There is a rule in Python programming called "it is Easier to Ask for Forgiveness than for Permission" (in short: EAFP). It means that you should catch exceptions instead of checking values for validity.

Thus, try the following:

try:
    qByUser = byUsrUrlObj.read()
    qUserData = json.loads(qByUser).decode('utf-8')
    questionSubjs = qUserData["all"]["questions"]
except ValueError:  # includes simplejson.decoder.JSONDecodeError
    print('Decoding JSON has failed')

EDIT: Since simplejson.decoder.JSONDecodeError actually inherits from ValueError (proof here), I simplified the catch statement by just using ValueError.

response.json() does not convert any server response to a JSON, it simply parses 'stringified' JSONs. So if the server returns a string that is not a JSON, then this will throw a decode error.

Some servers do return JSON objects, in which case your code will work. In the case of https://www.google.com/search?q=london this actually returns HTML code (as you would expect since it's a webpage).

You can test this by printing the response:

print(response.text)

which outputs:

# some very long output that ends with:
...();})();google.drty&&google.drty();</script></body></html>

Notice the </html> tag at the end? So this cannot be parsed into a JSON.

So how do you parse this into a usable HTML? You can use beautiful soup:

import requests
from bs4 import BeautifulSoup

url = "https://www.google.com/search?q=london"

response = requests.get(url, headers={"Accept": "application/json"})

soup = BeautifulSoup(response.text)

print(soup.prettify())

Your code produced an empty response body; you'd want to check for that or catch the exception raised. It is possible the server responded with a 204 No Content response, or a non-200-range status code was returned (404 Not Found, etc.). Check for this.

Note:

• There is no need to decode a response from UTF8 to Unicode, the json.loads() method can handle UTF8-encoded data natively.

• pycurl has a very archaic API. Unless you have a specific requirement for using it, there are better choices.

Either requests or httpx offer much friendlier APIs, including JSON support.

If you can, replace your call with the following httpx code:

import httpx

response = httpx.get(url)
response.raise_for_status()  # raises exception when not a 2xx response
if response.status_code != 204:
    return response.json()

Of course, this won't protect you from a URL that doesn't comply with HTTP standards; when using arbitrary URLs where this is a possibility, check if the server intended to give you JSON by checking the Content-Type header, and for good measure catch the exception:

if (
    response.status_code != 204 and
    response.headers["content-type"].strip().startswith("application/json")
):
    try:
        return response.json()
    except ValueError:
        # decide how to handle a server that's misbehaving to this extent