Python exception classes hierarchy is:
BaseExceptionExceptionOSErrorRequestException*TimeoutError
* IOError that is a base class for RequestException have been merged with OSError after python version 3.3:
Changed in version 3.3: EnvironmentError, IOError, WindowsError, socket.error, select.error and mmap.error have been merged into OSError, and the constructor may return a subclass.
So as you can see RequestException is not a parent class of TimeoutError exception and can't be used to catch this type of errors:
import requests
try:
raise TimeoutError('TIMEOUT ERROR')
except requests.exceptions.RequestException as e:
# catch RequestException type errors that are specific for request library only
# do something
print("RequestExceptions will be caught")
except TimeoutError as e:
# catch TimeoutError type errors that has same level in hierarchy as RequestException errors
# do something
print("TimeoutErrors will be caught")
except OSError as e:
# catch all OSError type errors. Little bit wider than previous exceptions
# do something
print("TimeoutErrors or RequestExceptions or any other OSErrors will be caught")
except Exception as e:
# catch any python errors, because all errors in python are children (sub types) for Exception class. Most wider exception type
# do something
print("TimeoutErrors or RequestExceptions or OSErrors or any other python errors will be caught")
Answer from rzlvmp on Stack OverflowMy request-handling wrapper smells: any suggestions for best practices?
Python requests errors out and I don't know how to catch the error - Stack Overflow
Intercept exception handling
exception - Correct way to try/except using Python requests module? - Stack Overflow
I'm working on a simple wrapper for requests.get that does basic error handling. Right now, if there are no errors, it just returns the response, but returns an error message otherwise. I'm now thinking it is smelly to have it return one data type (a Response) for success, and another (string with error message) for another. Is there a best practice for this sort of thing? Note the function below borrows ideas from a Stack Overflow thread on the topic.
Here is the function:
def make_request(url, session):
"""Wrapper for requests.get with basic error handling."""
try:
response = requests.get(url, headers = request_headers)
if response.ok:
return response
else:
response.raise_for_status()
except requests.exceptions.ConnectTimeout as connectException:
message = f"make_request - no connection to remote server: {connectException}.\nurl: {url}"
except requests.exceptions.ReadTimeout as readException:
message = f"make_request - timed out waiting for response: {readException}.\nurl: {url}"
except requests.exceptions.InvalidURL as invalidException:
message = f"make_request - invalid URL: {timeoutException}.\nurl: {url}"
except requests.exceptions.HTTPError as invalidException:
message = f"make_request - HTTP error: {httpException}.\nURL: {url}"
except requests.exceptions.RequestException as requestException:
message = f"make_request - exception raised: {requestError}.\nurl: {url}"
return messageHave a look at the Requests exception docs. In short:
In the event of a network problem (e.g. DNS failure, refused connection, etc), Requests will raise a
ConnectionErrorexception.In the event of the rare invalid HTTP response, Requests will raise an
HTTPErrorexception.If a request times out, a
Timeoutexception is raised.If a request exceeds the configured number of maximum redirections, a
TooManyRedirectsexception is raised.All exceptions that Requests explicitly raises inherit from
requests.exceptions.RequestException.
To answer your question, what you show will not cover all of your bases. You'll only catch connection-related errors, not ones that time out.
What to do when you catch the exception is really up to the design of your script/program. Is it acceptable to exit? Can you go on and try again? If the error is catastrophic and you can't go on, then yes, you may abort your program by raising SystemExit (a nice way to both print an error and call sys.exit).
You can either catch the base-class exception, which will handle all cases:
try:
r = requests.get(url, params={'s': thing})
except requests.exceptions.RequestException as e: # This is the correct syntax
raise SystemExit(e)
Or you can catch them separately and do different things.
try:
r = requests.get(url, params={'s': thing})
except requests.exceptions.Timeout:
# Maybe set up for a retry, or continue in a retry loop
except requests.exceptions.TooManyRedirects:
# Tell the user their URL was bad and try a different one
except requests.exceptions.RequestException as e:
# catastrophic error. bail.
raise SystemExit(e)
As Christian pointed out:
If you want http errors (e.g. 401 Unauthorized) to raise exceptions, you can call
Response.raise_for_status. That will raise anHTTPError, if the response was an http error.
An example:
try:
r = requests.get('http://www.google.com/nothere')
r.raise_for_status()
except requests.exceptions.HTTPError as err:
raise SystemExit(err)
Will print:
404 Client Error: Not Found for url: http://www.google.com/nothere
One additional suggestion to be explicit. It seems best to go from specific to general down the stack of errors to get the desired error to be caught, so the specific ones don't get masked by the general one.
url='http://www.google.com/blahblah'
try:
r = requests.get(url,timeout=3)
r.raise_for_status()
except requests.exceptions.HTTPError as errh:
print ("Http Error:",errh)
except requests.exceptions.ConnectionError as errc:
print ("Error Connecting:",errc)
except requests.exceptions.Timeout as errt:
print ("Timeout Error:",errt)
except requests.exceptions.RequestException as err:
print ("OOps: Something Else",err)
Http Error: 404 Client Error: Not Found for url: http://www.google.com/blahblah
vs
url='http://www.google.com/blahblah'
try:
r = requests.get(url,timeout=3)
r.raise_for_status()
except requests.exceptions.RequestException as err:
print ("OOps: Something Else",err)
except requests.exceptions.HTTPError as errh:
print ("Http Error:",errh)
except requests.exceptions.ConnectionError as errc:
print ("Error Connecting:",errc)
except requests.exceptions.Timeout as errt:
print ("Timeout Error:",errt)
OOps: Something Else 404 Client Error: Not Found for url: http://www.google.com/blahblah
Is using the RequestsException class to catch errors considered good error handling? It feels a bit like using except Exception. I don't need to do a bunch of different things based on the kind of invalid response I get. I just need to know if the request was successful. Would this be considered "good" code?
from requests import RequestException
import requests
import traceback
def get_page_content(testURL:str) -> str | None:
try:
response = requests.get(testURL)
return response.content
except RequestException as error:
print(f'{e} \n\n {traceback.format_exc()}')
return NoneWould something like this work better? Do all status codes between 200 and 399 mean I've probably got content I can use? I know 400 - 599 are bad. Except 418, that's gold, obviously.
from requests import RequestException
import requests import traceback
def get_page_content(testURL:str) -> str | None:
status_codes = [200,201,202,301,303] response = requests.get(testURL)
# if 400 <= response.status_code < 600:
if response.status in status_codes:
return response.content
else:
print("Invalid Response.")
return None