You should take a look at using urlencode.
from urllib import urlencode
base_url = "http://en.wikipedia.org/w/api.php?"
arguments = dict(action="query",
meta="globaluserinfo",
guiuser="$cammer",
guiprop="groups|merged|unattached",
format="json")
url = base_url + urlencode(arguments)
If you don't need to build a complete url you can just use the quote function for a single string:
>>> import urllib
>>> urllib.quote("$cammer")
'%24cammer'
So you end up with:
r['guiprop'] = urllib.quote(u'groups|merged|unattached')
r['guiuser'] = urllib.quote(u'$cammer')
You can escape certain characters by specifying them explicitly as safe argument value
urllib.quote(str, safe='~()*!.\'')
More : https://docs.python.org/3.0/library/urllib.parse.html#urllib.parse.quote
You can do this by adding the query params as a string before hitting the endpoint.
I have used requests for making a request.
For example:
GET Request
import requests
url = "https://www.example.com/?"
query = "A=B,C"
url_final = url + query
url = requests.get(url_final)
print(url.url)
# https://www.example.com/?A=B,C
The comma (along with some other characters) is defined in RFC 3986 as a reserved character. This means the comma has defined meaning at various parts in a URL, and if it is not being used in that context it needs to be percent-encoded.
That said, the query parameter doesn't give the comma any special syntax, so in query parameters, we probably shouldn't be encoding it. That said, it's not entirely Requests' fault: the parameters are encoded using urllib.urlencode(), which is what is percent-encoding the query parameters.
This isn't easy to fix though, because some web services use , and some use %2C, and neither is wrong. You might just have to handle this encoding yourself.
Using urllib package (import urllib) :
Python 2.7
From official documentation :
urllib.unquote(string)Replace
%xxescapes by their single-character equivalent.Example:
unquote('/%7Econnolly/')yields'/~connolly/'.
Python 3
From official documentation :
urllib.parse.unquote(string, encoding='utf-8', errors='replace')[…]
Example:
unquote('/El%20Ni%C3%B1o/')yields'/El Niño/'.
And if you are using Python3 you could use:
import urllib.parse
urllib.parse.unquote(url)
You should have a look at urllib.quote - that should do the trick. Have a look at the docs for reference.
To expand on this answer: The problem is, that % (+ a hexadecimal number) is the escape sequence for special characters in URLs. If you want the server to interpret your % literaly, you need to escape it as well, which is done by replacing it with %25. The aforementioned qoute function does stuff like that for you.
Let requests construct the query string for you by passing the parameters in the params argument to requests.get() (see documentation):
api_url = 'http://myApiURL'
params = {'ID': 'something', 'parameter': '%w42'}
r = requests.get(api_url, params=params, auth=(user, pass))
requests should then percent encode the parameters in the query string for you. Having said that, at least with requests version 2.11.1 on my machine, I find that the % is encoded when passing it in the url, so perhaps you could check which version you are using.
Also for basic authentication you can simply pass the user name and password in a tuple as shown above.
From the Python 3 documentation:
urllib.parse.quote(string, safe='/', encoding=None, errors=None)
Replace special characters in string using the
%xxescape. Letters, digits, and the characters'_.-~'are never quoted. By default, this function is intended for quoting the path section of a URL. The optional safe parameter specifies additional ASCII characters that should not be quoted — its default value is'/'.
That means passing '' for safe will solve your first issue:
>>> import urllib.parse
>>> urllib.parse.quote('/test')
'/test'
>>> urllib.parse.quote('/test', safe='')
'%2Ftest'
(The function quote was moved from urllib to urllib.parse in Python 3.)
By the way, have a look at urlencode.
About the second issue, there was a bug report about it and it was fixed in Python 3.
For Python 2, you can work around it by encoding as UTF-8 like this:
>>> query = urllib.quote(u"Müller".encode('utf8'))
>>> print urllib.unquote(query).decode('utf8')
Müller
In Python 3, urllib.quote has been moved to urllib.parse.quote, and it does handle Unicode by default.
>>> from urllib.parse import quote
>>> quote('/test')
'/test'
>>> quote('/test', safe='')
'%2Ftest'
>>> quote('/El Niño/')
'/El%20Ni%C3%B1o/'
Convert a mapping object or a sequence of two-element tuples to a “percent-encoded” string[...]
The urlencode() method is acting as expected. If you want to prevent the encoding then you can first encode the entire object and then replace the encoded characters with pipes.
>>> u = urlencode(params)
>>> u.replace('%7C', '|')
'p=1+2+3+4+5%266&l=ab|cd|ef'
It is simpler in Python 3:
urllib.parse.urlencode(params, safe='|')