requests does not use the defaults from ssl; it uses envvar REQUESTS_CA_BUNDLE or CURL_CA_BUNDLE if set and otherwise uses the (spunoff) certifi module which depending on how you installed requests&certifi which you didn't say and your environment which you didn't identify might use a system default (which might or might not be the same as the OpenSSL used in ssl) or might be its own copy of Mozilla. In the latter case it should include Digicert Global Root CA as needed for that site, because Firefox (also Mozilla) does.
Look at requests.certs.where() or python -m requests.certs
Mostly dupe
Python Requests - How to use system ca-certificates (debian/ubuntu)?
How to force requests use the certificates on my ubuntu system
Why python requests not use the system ssl cert by default?
ssl - How to force requests use the certificates on my ubuntu system - Stack Overflow
ssl - How can I use system CA certificates (Debian/Ubuntu) with Python's Requests? - Stack Overflow
CA-certs bundles and requests.certs.where - Improved documentation
Setting `REQUESTS_CA_BUNDLE` to directory causes exception
Videos
You can set the environment variable REQUESTS_CA_BUNDLE so you don't have to modify your code:
export REQUESTS_CA_BUNDLE=/usr/local/share/ca-certificates/hbrls-server.cert
Source: https://requests.readthedocs.io/en/master/user/advanced/#ssl-cert-verification
@and answered this best in 2017, and I think we've all given this person enough to to post what I think is the best answer as an answer. So now I'm doing it, after posting upvote #42 (such a fitting #) to the comment that saved my bacon. But we digress. Combining the well-celebrated answer with @and's golden comment...
You can set the environment variable REQUESTS_CA_BUNDLE so you don't have to modify your code:
export REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt
The improvement, if it's not clear is this: /etc/ssl/certs/ca-certificates.crt will contain not merely any self-cert you added to your trust store, but also all of the other standard certs. That's a big deal, because, for example, I ran into a situation where when REQUESTS_CA_BUNDLE was set to just my self-cert, the AWS CLI could no longer authenticate. (Don't ask me why AWS cares about REQUESTS_CA_BUNDLE. I don't know. I do know, however, that using ca-certificates.crt solved the problem.
From an answer to Python Requests throwing SSLError:
To make Python's Requests use the system ca-certificates bundle, it needs to be told to use it over its own embedded bundle
export REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt
Requests embeds its bundles here, for reference:
/usr/local/lib/python2.7/site-packages/requests/cacert.pem
/usr/lib/python3/dist-packages/requests/cacert.pem
Or in newer versions, use an additional package to obtain certificates from: Certifi
To verify from which file certificates are loaded, you can try:
Python 3.8.5 (default, Jul 28 2020, 12:59:40)
>>> import certifi
>>> certifi.where()
'/etc/ssl/certs/ca-certificates.crt'
I struggled with this for a week or so recently. I finally found that the way to verify a self-signed, or privately signed, certificate in Python. You need to create your own certificate bundle file. No need to update obscure certificate bundles every time you update a library, or add anything to the system certificate store.
Start by running the openssl command that you ran before, but add -showcerts. openssl s_client -connect mysite.local:443 -showcerts This will give you a long output, and at the top you'll see the entire certificate chain. Usually, this means three certs, the website's certificate, the intermediate certificate, and the root certificate in that order. We need to put just the root and intermediate certificates into a next file in the opposite order.
Copy the last cert, the root certificate, to a new text file. Grab just the stuff between, and including:
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
Copy the middle cert (aka the intermediate certificate) to the new text file under the root cert. Again, grab the Begin and End Certificate lines and everything in between.
Save this text file to the directory where your Python script resides. My recommendation is to call it CertBundle.pem. (If you give it a different name, or put it somewhere else in your folder structure, make sure that the verify line reflects that.) Update your script to reference the new certificate bundle:
response = requests.post("https://www.example.com/", headers=headerContents, json=bodyContents, verify="CertBundle.pem")
And that's it. If you have only the root or only the intermediate certificate, then Python can't validate the entire certificate chain. But, if you include both of the certificates in the certificate bundle that you created, then Python can validate that the intermediate was signed by the root, and then when it accesses the website it can validate that the website's certificate was signed by the intermediate certificate.
edit: Fixed the file extension for the cert bundle. Also, fixed a couple of grammatical mistakes.
Hi All,
We were POCing Zscaler for a while and didnt go through with it, I believe I removed everything Zscaler related but I am having major issues with my CA Root Cert. For some reason its pointing here:
REQUESTS_CA_BUNDLE=C:\Users\Jason Dannels\Downloads\ZscalerRootCerts (1)\ZscalerRootCerts\ca-bundle.pem
That folder and file does not exist anymore, anyone have an idea how to reset this to its default location?