Basically what you do is correct. Looking at redmine docs you linked to, it seems that suffix after the dot in the url denotes type of posted data (.json for JSON, .xml for XML), which agrees with the response you get - Processing by AttachmentsController#upload as XML. I guess maybe there's a bug in docs and to post binary data you should try using http://redmine/uploads url instead of http://redmine/uploads.xml.

Btw, I highly recommend very good and very popular Requests library for http in Python. It's much better than what's in the standard lib (urllib2). It supports authentication as well but I skipped it for brevity here.

import requests
with open('./x.png', 'rb') as f:
    data = f.read()
res = requests.post(url='http://httpbin.org/post',
                    data=data,
                    headers={'Content-Type': 'application/octet-stream'})

# let's check if what we sent is what we intended to send...
import json
import base64

assert base64.b64decode(res.json()['data'][len('data:application/octet-stream;base64,'):]) == data

UPDATE

To find out why this works with Requests but not with urllib2 we have to examine the difference in what's being sent. To see this I'm sending traffic to http proxy (Fiddler) running on port 8888:

Using Requests

import requests

data = 'test data'
res = requests.post(url='http://localhost:8888',
                    data=data,
                    headers={'Content-Type': 'application/octet-stream'})

we see

POST http://localhost:8888/ HTTP/1.1
Host: localhost:8888
Content-Length: 9
Content-Type: application/octet-stream
Accept-Encoding: gzip, deflate, compress
Accept: */*
User-Agent: python-requests/1.0.4 CPython/2.7.3 Windows/Vista

test data

and using urllib2

import urllib2

data = 'test data'    
req = urllib2.Request('http://localhost:8888', data)
req.add_header('Content-Length', '%d' % len(data))
req.add_header('Content-Type', 'application/octet-stream')
res = urllib2.urlopen(req)

we get

POST http://localhost:8888/ HTTP/1.1
Accept-Encoding: identity
Content-Length: 9
Host: localhost:8888
Content-Type: application/octet-stream
Connection: close
User-Agent: Python-urllib/2.7

test data

I don't see any differences which would warrant different behavior you observe. Having said that it's not uncommon for http servers to inspect User-Agent header and vary behavior based on its value. Try to change headers sent by Requests one by one making them the same as those being sent by urllib2 and see when it stops working.

Answer from Piotr Dobrogost on Stack Overflow
Top answer
1 of 4
104

Basically what you do is correct. Looking at redmine docs you linked to, it seems that suffix after the dot in the url denotes type of posted data (.json for JSON, .xml for XML), which agrees with the response you get - Processing by AttachmentsController#upload as XML. I guess maybe there's a bug in docs and to post binary data you should try using http://redmine/uploads url instead of http://redmine/uploads.xml.

Btw, I highly recommend very good and very popular Requests library for http in Python. It's much better than what's in the standard lib (urllib2). It supports authentication as well but I skipped it for brevity here.

import requests
with open('./x.png', 'rb') as f:
    data = f.read()
res = requests.post(url='http://httpbin.org/post',
                    data=data,
                    headers={'Content-Type': 'application/octet-stream'})

# let's check if what we sent is what we intended to send...
import json
import base64

assert base64.b64decode(res.json()['data'][len('data:application/octet-stream;base64,'):]) == data

UPDATE

To find out why this works with Requests but not with urllib2 we have to examine the difference in what's being sent. To see this I'm sending traffic to http proxy (Fiddler) running on port 8888:

Using Requests

import requests

data = 'test data'
res = requests.post(url='http://localhost:8888',
                    data=data,
                    headers={'Content-Type': 'application/octet-stream'})

we see

POST http://localhost:8888/ HTTP/1.1
Host: localhost:8888
Content-Length: 9
Content-Type: application/octet-stream
Accept-Encoding: gzip, deflate, compress
Accept: */*
User-Agent: python-requests/1.0.4 CPython/2.7.3 Windows/Vista

test data

and using urllib2

import urllib2

data = 'test data'    
req = urllib2.Request('http://localhost:8888', data)
req.add_header('Content-Length', '%d' % len(data))
req.add_header('Content-Type', 'application/octet-stream')
res = urllib2.urlopen(req)

we get

POST http://localhost:8888/ HTTP/1.1
Accept-Encoding: identity
Content-Length: 9
Host: localhost:8888
Content-Type: application/octet-stream
Connection: close
User-Agent: Python-urllib/2.7

test data

I don't see any differences which would warrant different behavior you observe. Having said that it's not uncommon for http servers to inspect User-Agent header and vary behavior based on its value. Try to change headers sent by Requests one by one making them the same as those being sent by urllib2 and see when it stops working.

2 of 4
4

This has nothing to do with a malformed upload. The HTTP error clearly specifies 401 unauthorized, and tells you the CSRF token is invalid. Try sending a valid CSRF token with the upload.

More about csrf tokens here:

What is a CSRF token ? What is its importance and how does it work?

🌐
Python
bugs.python.org › issue11898
Issue 11898: Sending binary data with a POST request in httplib can cause Unicode exceptions - Python tracker
This issue tracker has been migrated to GitHub, and is currently read-only. For more information, see the GitHub FAQs in the Python's Developer Guide · This issue has been migrated to GitHub: https://github.com/python/cpython/issues/56107
🌐
GitHub
gist.github.com › yoavram › 4351498
Example of uploading binary files programmatically in python, including both client and server code. Client implemented with the requests library and the server is implemented with the flask library. · GitHub
Example of uploading binary files programmatically in python, including both client and server code. Client implemented with the requests library and the server is implemented with the flask libra...
🌐
Python-requests
docs.python-requests.org › en › v1.2.3 › user › quickstart
https://docs.python-requests.org/en/v1.2.3/user/qu...
>>> url = 'http://httpbin.org/post' >>> files = {'file': ('report.xls', open('report.xls', 'rb'))} >>> r = requests.post(url, files=files) >>> r.text { ... "files": { "file": "<censored...binary...data>" }, ...
🌐
YouTube
youtube.com › watch
python requests post binary data - YouTube
Instantly Download or Run the code at https://codegive.com in this tutorial, we'll explore how to use the requests library in python to send post requests w...
Published   February 22, 2024
🌐
GitHub
github.com › psf › requests › issues › 403
Binary POST data stopped working in requests 0.10.1 · Issue #403 · psf/requests
January 30, 2012 - The following method for POSTing binary data worked with requests 0.10.0 on Python 2.7.1 (Ubuntu 11.04 x64) : bodybytes = '\xDE\xAD\xBE\xEF' r = requests.post(url, data=bodybytes) With requ...
Author   vimalb
🌐
Beautiful Soup
tedboy.github.io › requests › qs4.html
2.4. Binary Response Content — Requests API
For example, to create an image from binary data returned by a request, you can use the following code:
🌐
Python
bugs.python.org › issue12398
Issue 12398: Sending binary data with a POST request in httplib can cause Unicode exceptions - Python tracker
This issue tracker has been migrated to GitHub, and is currently read-only. For more information, see the GitHub FAQs in the Python's Developer Guide · This issue has been migrated to GitHub: https://github.com/python/cpython/issues/56607
🌐
Requests
requests.readthedocs.io › en › latest › user › quickstart
Quickstart — Requests 2.33.1 documentation
>>> url = 'https://httpbin.org/post' >>> files = {'file': ('report.xls', open('report.xls', 'rb'), 'application/vnd.ms-excel', {'Expires': '0'})} >>> r = requests.post(url, files=files) >>> r.text { ... "files": { "file": "<censored...binary...data>" }, ...
Find elsewhere
🌐
CSDN
devpress.csdn.net › python › 63044f14c67703293080ac25.html
Python POST binary data - DevPress官方社区
August 23, 2022 - import requests data = 'test data' res = requests.post(url='http://localhost:8888', data=data, headers={'Content-Type': 'application/octet-stream'}) ... POST http://localhost:8888/ HTTP/1.1 Host: localhost:8888 Content-Length: 9 Content-Type: application/octet-stream Accept-Encoding: gzip, deflate, compress Accept: */* User-Agent: python-requests/1.0.4 CPython/2.7.3 Windows/Vista test data
Top answer
1 of 1
15

The curl -d switch sends a POST request, but you are using requests.get() instead, sending a GET request (whose body is ignored).

Make it a POST instead, by using request.post():

import requests
import json

url = "http://cdcnepal.com/Modules/HOmeMoviesLists/WebService2.asmx/GetShowsByDate"
headers = {"content-type": "application/json; charset=UTF-8"}
payload = {"portalId":"1","showDate":"26/05/2014","flag":0,"size":9}
r = requests.post(url, headers=headers, data=json.dumps(payload))
print r.text

You also need to:

  1. not use a list for the content-type header, there is no support for paramaters being specified separately.
  2. Encode your JSON data to a JSON string; requests doesn't do this for you. Instead, a dictionary passed to data is encoded as application/x-www-form-urlencoded data instead.

You can compare the curl command with requests more easily using http://httpbin.org/post:

$ curl http://httpbin.org/post \
>  -H 'Content-Type: application/json; charset=UTF-8' \
>  -d '{"portalId":"1","showDate":"26/05/2014","flag":0,"size":9}'

{
  "args": {},
  "data": "{\"portalId\":\"1\",\"showDate\":\"26/05/2014\",\"flag\":0,\"size\":9}",
  "files": {},
  "form": {},
  "headers": {
    "Accept": "*/*",
    "Connection": "close",
    "Content-Length": "58",
    "Content-Type": "application/json; charset=UTF-8",
    "Host": "httpbin.org",
    "User-Agent": "curl/7.30.0",
    "X-Request-Id": "78d7bb7d-e29b-482b-908a-48d2395a050f"
  },
  "json": {
    "flag": 0,
    "portalId": "1",
    "showDate": "26/05/2014",
    "size": 9
  },
  "origin": "84.92.98.170",
  "url": "http://httpbin.org/post"
}

and

>>> import requests
>>> import json
>>> from pprint import pprint
>>> url = 'http://httpbin.org/post'
>>> headers = {"content-type":"application/json; charset=UTF-8"}
>>> payload = {"portalId":"1","showDate":"26/05/2014","flag":0,"size":9}
>>> r = requests.post(url, headers=headers, data=json.dumps(payload))
>>> pprint(r.json())
{u'args': {},
 u'data': u'{"portalId": "1", "flag": 0, "size": 9, "showDate": "26/05/2014"}',
 u'files': {},
 u'form': {},
 u'headers': {u'Accept': u'*/*',
              u'Accept-Encoding': u'gzip, deflate, compress',
              u'Connection': u'close',
              u'Content-Length': u'65',
              u'Content-Type': u'application/json; charset=UTF-8',
              u'Host': u'httpbin.org',
              u'User-Agent': u'python-requests/2.2.1 CPython/2.7.6 Darwin/13.1.0',
              u'X-Request-Id': u'06d6b542-c279-4898-8701-2c0d502aa36e'},
 u'json': {u'flag': 0,
           u'portalId': u'1',
           u'showDate': u'26/05/2014',
           u'size': 9},
 u'origin': u'84.92.98.170',
 u'url': u'http://httpbin.org/post'}

Both cases show the same json dictionary being returned.

If you are using requests version 2.4.2 or newer, you can also leave the JSON encoding to the library; it'll set the correct Content-Type header too, if you pass in the data to send as the json keyword argument:

import requests

url = "http://cdcnepal.com/Modules/HOmeMoviesLists/WebService2.asmx/GetShowsByDate"
payload = {"portalId":"1","showDate":"26/05/2014","flag":0,"size":9}
r = requests.post(url, json=payload)
print r.text
🌐
Python.org
discuss.python.org › python help
Python requests module with json data some fields are binary date - Python Help - Discussions on Python.org
March 20, 2024 - Hello In Python requests module how to send json data, some fields are a binary data. And doesn’t wants to convert base64 or file as well. Do we have any other options for sending json data along with binary data by u…
🌐
Pythonrequests
pythonrequests.com › python-requests-post-binary-data
python requests post binary data
July 12, 2023 - This method is useful when we want to send multiple files or data along with the binary file. import requests url = 'https://example.com/upload' files = {'file': open('file.pdf', 'rb')} response = requests.post(url, files=files) print(response.status_code) # prints 200 if the upload was successful
🌐
Python-requests
docs.python-requests.org › en › latest › user › quickstart
Quickstart — Requests 2.32.3 documentation - docs.python …
In the event you are posting a very large file as a multipart/form-data request, you may want to stream the request. By default, requests does not support this, but there is a separate package which does - requests-toolbelt. You should read the toolbelt’s documentation for more details about how to use it. For sending multiple files in one request refer to the advanced section. ... It is strongly recommended that you open files in binary mode.
🌐
WebScraping.AI
webscraping.ai › faq › requests › how-do-i-handle-binary-data-responses-with-requests
How do I handle binary data responses with Requests? | WebScraping.AI
# Download and verify an image file python -c " import requests response = requests.get('https://httpbin.org/image/png') with open('test.png', 'wb') as f: f.write(response.content) print(f'Downloaded {len(response.content)} bytes') print(f'Content-Type: {response.headers.get(\"content-type\")}') " # Check file integrity python -c " import hashlib with open('test.png', 'rb') as f: content = f.read() print(f'File size: {len(content)} bytes') print(f'MD5 hash: {hashlib.md5(content).hexdigest()}') " Handling binary data with Python Requests requires understanding the distinction between text and binary content, proper use of response.content, and implementing appropriate streaming for large files.
🌐
DNMTechs
dnmtechs.com › sending-binary-data-with-pythons-post-method-in-python-3
Sending Binary Data with Python’s POST Method in Python 3 – DNMTechs – Sharing and Storing Technology Knowledge
We make a POST request to the specified URL using the `requests.post()` method. We pass the binary data as the `data` parameter. The response object contains the server’s response, which we can access using the `status_code` and `text` attributes. Sending binary data with Python’s POST ...
🌐
Strapi Community
forum.strapi.io › questions and answers
How do I upload binary files using Python? - Questions and Answers - Strapi Community Forum
January 15, 2022 - ▶ System Information on the Upload documentation here: Upload - Strapi Developer Docs The docs give a code snippet for uploading a binary file in JavaScript. How do I do it in Python? This is ...
🌐
Requests
requests.readthedocs.io › en › v1.2.3 › user › quickstart
Quickstart — Requests 1.2.3 documentation
>>> url = 'http://httpbin.org/post' >>> files = {'file': ('report.xls', open('report.xls', 'rb'))} >>> r = requests.post(url, files=files) >>> r.text { ... "files": { "file": "<censored...binary...data>" }, ...