Try urllib.parse.urlencode:

>>> from urllib.parse import urlencode
>>> urlencode({'foo': 1, 'bar': 2})
'foo=1&bar=2'

Use urllib.parse.urlencode(). It takes a dictionary of key-value pairs, and converts it into a form suitable for a URL (e.g., key1=val1&key2=val2).

For your example:

>>> import urllib.parse
>>> params = {'a':'A', 'b':'B'}
>>> urllib.parse.urlencode(params)
'a=A&b=B'

If you want to make a URL with repetitive params such as: p=1&p=2&p=3 you have two options:

>>> a = (('p',1),('p',2), ('p', 3))
>>> urllib.parse.urlencode(a)
'p=1&p=2&p=3'

or:

>>> urllib.parse.urlencode({'p': [1, 2, 3]}, doseq=True)
'p=1&p=2&p=3'

If you are still using Python 2, use urllib.urlencode().

Your challenge is the encoding of the Chinese characters in python.

Doing some research around it, it seems that behind the scenes there is some fun going on with unicode and ascii.

You can see this - Try this as a debug line:

rawjson = {'toDate2': '08月26日', 'toWeek': 'END'}
encodedjson = urllib.parse.quote(json.dumps(rawjson))
print (encodedjson)

output is:

%7B%22toDate2%22%3A%20%2208%5Cu670826%5Cu65e5%22%2C%20%22toWeek%22%3A%20%22END%22%7D

Stick that in a URLdecoder = and you get:

{"toDate2": "08\u670826\u65e5", "toWeek": "END"}

The chinese characters have gone. You've now got \u6708 and \u65e5. It's not the expected format on the server so no wonder the server gives you a 500.

You need to tell python to make it UTF-8, and you need to ensure the content-type is set in the headers.

• headers: 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8'
• a UTF-8 string: json.dumps(payload, ensure_ascii=False).encode('utf-8')

This is a working script which returns a 200:

import requests
import urllib.parse
import json

#for copying simplicity from chrome devtools
null = None
false = False

#the json
payload = {
    "userType":"LTP001","startSite":"SK","endSite":"HKM","toDate":"2021-08-26","toDate2":"08月26日","toWeek":"周四","backDate":"","backDate2":"","backWeek":"","endSiteName":"Hong+Kong,+Hong+KongMacau","startSiteName":"Shekou,+shenzhen","sailingType":"0","lineId":"SK-HKM","airportTime":null,"lineType":null,"toTime":null,"flightCode":null,"flightName":null,"flightNo":null,"flightId":null,"flightDate":null,"flightHours":null,"flightMinute":null,"chanId":"1","isSeckill":false,"isSeckills":false,"nightFlight":0,"batchNo":null,"hsyLineId":null,"memberId":null,
}

payload = urllib.parse.quote(json.dumps(payload, ensure_ascii=False).encode('utf-8'))
payload = payload.replace('%20','')
payload = 'siteResJson=' + payload

headers = {
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
    'Accept-Encoding': 'gzip, deflate, br',
    'Accept-Language': 'en-GB,en;q=0.9',
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36',
    'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8'
}
Res_post = requests.post("https://www.cmskchp.com/sailingsJson", data=payload, headers=headers)

This is the response:

The normal call to urllib.parse.urlencode should give you a string that's correct for sending as x-www-form-urlencoded data. However, to answer your question about the resulting string formatting...

urllib.parse.urlencode has a keyword argument called quote_via, which allows you to specify a method for how some characters like spaces or slashes are encoded, and by default they are done with urllib.parse.quote_plus where spaces are substituted with +. You can use urllib.parse.quote instead, which will get you a string like what you want.

The resulting string is a series of key=value pairs separated by '&' characters, where both key and value are quoted using the quote_via function. By default, quote_plus() is used to quote the values, which means spaces are quoted as a '+' character and ‘/’ characters are encoded as %2F, which follows the standard for GET requests (application/x-www-form-urlencoded). An alternate function that can be passed as quote_via is quote(), which will encode spaces as %20 and not encode ‘/’ characters. For maximum control of what is quoted, use quote and specify a value for safe.

import urllib.parse

data = {
   "name":"test name",
   "file_urls":[
      "/Users/tarequzzamankhan/Desktop/instagram.webp",
      "/Users/tarequzzamankhan/Desktop/instagram.webp"
   ],
   "file_type":"pdf",
   "user_name":"Tareq",
   "mobile":"018xxxxxxxxx",
   "address":"Dhaka",
   "email":"[email protected]"
}

# I didn't use doseq=True because you seem to want all the `file_urls` as a single argument
qs = urllib.parse.urlencode(data, quote_via=urllib.parse.quote)

print(qs)

Output:

In [24]: print(qs)
name=test%20name&file_urls=%5B%27%2FUsers%2Ftarequzzamankhan%2FDesktop%2Finstagram.webp%27%2C%20%27%2FUsers%2Ftarequzzamankhan%2FDesktop%2Finstagram.webp%27%5D&file_type=pdf&user_name=Tareq&mobile=018xxxxxxxxx&address=Dhaka&email=example%40mail.com

The reason you're getting JSON is because you're explicitly calling json.dumps to generate a JSON string. Just don't do that, and you won't get a JSON string. In other words, change your first line to this:

data = {'param1': 'value1', 'param2': 'value2'}

As the docs explain, if you pass a dict as the data value, it will be form-encoded, while if you pass a string, it will be sent as-is.

For example, in one terminal window:

$ nc -kl 8765

In another:

$ python3
>>> import requests
>>> d = {'spam': 20, 'eggs': 3}
>>> requests.post("http://localhost:8765", data=d)
^C
>>> import json
>>> j = json.dumps(d)
>>> requests.post("http://localhost:8765", data=j)
^C

In the first terminal, you'll see that the first request body is this (and Content-Type application/x-www-form-urlencoded):

spam=20&eggs=3

… while the second is this (and has no Content-Type):

{"spam": 20, "eggs": 3}

It clearly looks like, the data your server API is expecting is not being passed in expected format which is breaking the server side code execution. Please try to check what are the exact API's requirements along with headers.

If it expects Content-Type header to be 'x-www-form-urlencoded', then make sure you pass it as well when you are making request to the API URL.

With requests module it can be easier for implementing your code without even using urllib's urlencode. So better you also look at here also: http://docs.python-requests.org/en/v0.10.7/user/quickstart/

You don't need to explicitly encode it, simply pass your dict to data argument and it will be encoded automatically.

>>> r = requests.post(URL, data = {'key':'value'})

From the official documentation:

Typically, you want to send some form-encoded data — much like an HTML form. To do this, simply pass a dictionary to the data argument. Your dictionary of data will automatically be form-encoded when the request is made

OK people. I implemented it myself:

import urllib

def recursive_urlencode(d):
    """URL-encode a multidimensional dictionary.

    >>> data = {'a': 'b&c', 'd': {'e': {'f&g': 'h*i'}}, 'j': 'k'}
    >>> recursive_urlencode(data)
    u'a=b%26c&j=k&d[e][f%26g]=h%2Ai'
    """
    def recursion(d, base=[]):
        pairs = []

        for key, value in d.items():
            new_base = base + [key]
            if hasattr(value, 'values'):
                pairs += recursion(value, new_base)
            else:
                new_pair = None
                if len(new_base) > 1:
                    first = urllib.quote(new_base.pop(0))
                    rest = map(lambda x: urllib.quote(x), new_base)
                    new_pair = "%s[%s]=%s" % (first, ']['.join(rest), urllib.quote(unicode(value)))
                else:
                    new_pair = "%s=%s" % (urllib.quote(unicode(key)), urllib.quote(unicode(value)))
                pairs.append(new_pair)
        return pairs

    return '&'.join(recursion(d))

if __name__ == "__main__":
    import doctest
    doctest.testmod()

Still, I'd be interested to know if there's a better way to do this. I can't believe Python's standard library doesn't implement this.

You have to get the data on the server side (Flask app) as json, so you can use request.get_json() or request.json:

from flask import Flask, request

app = Flask(__name__)

@app.route("/", methods=['POST'])
def home():
    content = request.get_json()

if __name__ == "__main__":
    app.run()

Send it on the client side as:

import requests

data = dict()
data['a'] = "foo"
data['b'] = "bar"
data['ninja'] = { "Name": "John doe" }

r = requests.post("https://my_own_url.io/", json=data, verify=False)