I ran into this issue when making a call to Gitlab CI. Since the above did not work for me (gave me some kind of error about not being able to concatenate bytes to a string), and because the arguments I was attempting to pass were nested, I thought I would post what ended up working for me:

API_ENDPOINT = "https://gitlab.com/api/v4/projects/{}/pipeline".format(GITLAB_PROJECT_ID)

API_TOKEN = "SomeToken"

data = {
    "ref": ref,
    "variables": [
        {
            "key": "ENVIRONMENT",
            "value": some_env
        },
        {   "key": "S3BUCKET",
            "value": some_bucket
        },
    ]
}

req_headers = {
    'Content-Type': 'application/json',
    'PRIVATE-TOKEN': API_TOKEN,
}

http = urllib3.PoolManager()

encoded_data = json.dumps(data).encode('utf-8')

r = http.request('POST', API_ENDPOINT,
             headers=req_headers,
             body=encoded_data)

resp_body = r.data.decode('utf-8')
resp_dict = json.loads(r.data.decode('utf-8'))

logger.info('Response Code: {}'.format(r.status))
logger.info('Response Body: {}'.format(resp_body))

if 'message' in resp_body:
    logfile_msg = 'Failed Gitlab Response-- {} {message}'.format(r.status, **resp_dict)

You do not have to install urllib3. You can choose any HTTP-request-making library that fits your needs and feed the response to BeautifulSoup. The choice is though usually requests because of the rich feature set and convenient API. You can install requests by entering pip install requests in the command line. Here is a basic example:

from bs4 import BeautifulSoup
import requests

url = "url"
response = requests.get(url)

soup = BeautifulSoup(response.content, "html.parser")