Basically what you do is correct. Looking at redmine docs you linked to, it seems that suffix after the dot in the url denotes type of posted data (.json for JSON, .xml for XML), which agrees with the response you get - Processing by AttachmentsController#upload as XML. I guess maybe there's a bug in docs and to post binary data you should try using http://redmine/uploads url instead of http://redmine/uploads.xml.

Btw, I highly recommend very good and very popular Requests library for http in Python. It's much better than what's in the standard lib (urllib2). It supports authentication as well but I skipped it for brevity here.

import requests
with open('./x.png', 'rb') as f:
    data = f.read()
res = requests.post(url='http://httpbin.org/post',
                    data=data,
                    headers={'Content-Type': 'application/octet-stream'})

# let's check if what we sent is what we intended to send...
import json
import base64

assert base64.b64decode(res.json()['data'][len('data:application/octet-stream;base64,'):]) == data

UPDATE

To find out why this works with Requests but not with urllib2 we have to examine the difference in what's being sent. To see this I'm sending traffic to http proxy (Fiddler) running on port 8888:

Using Requests

import requests

data = 'test data'
res = requests.post(url='http://localhost:8888',
                    data=data,
                    headers={'Content-Type': 'application/octet-stream'})

we see

POST http://localhost:8888/ HTTP/1.1
Host: localhost:8888
Content-Length: 9
Content-Type: application/octet-stream
Accept-Encoding: gzip, deflate, compress
Accept: */*
User-Agent: python-requests/1.0.4 CPython/2.7.3 Windows/Vista

test data

and using urllib2

import urllib2

data = 'test data'    
req = urllib2.Request('http://localhost:8888', data)
req.add_header('Content-Length', '%d' % len(data))
req.add_header('Content-Type', 'application/octet-stream')
res = urllib2.urlopen(req)

we get

POST http://localhost:8888/ HTTP/1.1
Accept-Encoding: identity
Content-Length: 9
Host: localhost:8888
Content-Type: application/octet-stream
Connection: close
User-Agent: Python-urllib/2.7

test data

I don't see any differences which would warrant different behavior you observe. Having said that it's not uncommon for http servers to inspect User-Agent header and vary behavior based on its value. Try to change headers sent by Requests one by one making them the same as those being sent by urllib2 and see when it stops working.

Answer from Piotr Dobrogost on Stack Overflow
Top answer
1 of 4
104

Basically what you do is correct. Looking at redmine docs you linked to, it seems that suffix after the dot in the url denotes type of posted data (.json for JSON, .xml for XML), which agrees with the response you get - Processing by AttachmentsController#upload as XML. I guess maybe there's a bug in docs and to post binary data you should try using http://redmine/uploads url instead of http://redmine/uploads.xml.

Btw, I highly recommend very good and very popular Requests library for http in Python. It's much better than what's in the standard lib (urllib2). It supports authentication as well but I skipped it for brevity here.

import requests
with open('./x.png', 'rb') as f:
    data = f.read()
res = requests.post(url='http://httpbin.org/post',
                    data=data,
                    headers={'Content-Type': 'application/octet-stream'})

# let's check if what we sent is what we intended to send...
import json
import base64

assert base64.b64decode(res.json()['data'][len('data:application/octet-stream;base64,'):]) == data

UPDATE

To find out why this works with Requests but not with urllib2 we have to examine the difference in what's being sent. To see this I'm sending traffic to http proxy (Fiddler) running on port 8888:

Using Requests

import requests

data = 'test data'
res = requests.post(url='http://localhost:8888',
                    data=data,
                    headers={'Content-Type': 'application/octet-stream'})

we see

POST http://localhost:8888/ HTTP/1.1
Host: localhost:8888
Content-Length: 9
Content-Type: application/octet-stream
Accept-Encoding: gzip, deflate, compress
Accept: */*
User-Agent: python-requests/1.0.4 CPython/2.7.3 Windows/Vista

test data

and using urllib2

import urllib2

data = 'test data'    
req = urllib2.Request('http://localhost:8888', data)
req.add_header('Content-Length', '%d' % len(data))
req.add_header('Content-Type', 'application/octet-stream')
res = urllib2.urlopen(req)

we get

POST http://localhost:8888/ HTTP/1.1
Accept-Encoding: identity
Content-Length: 9
Host: localhost:8888
Content-Type: application/octet-stream
Connection: close
User-Agent: Python-urllib/2.7

test data

I don't see any differences which would warrant different behavior you observe. Having said that it's not uncommon for http servers to inspect User-Agent header and vary behavior based on its value. Try to change headers sent by Requests one by one making them the same as those being sent by urllib2 and see when it stops working.

2 of 4
4

This has nothing to do with a malformed upload. The HTTP error clearly specifies 401 unauthorized, and tells you the CSRF token is invalid. Try sending a valid CSRF token with the upload.

More about csrf tokens here:

What is a CSRF token ? What is its importance and how does it work?

🌐
Python
bugs.python.org › issue11898
Issue 11898: Sending binary data with a POST request in httplib can cause Unicode exceptions - Python tracker
This issue tracker has been migrated to GitHub, and is currently read-only. For more information, see the GitHub FAQs in the Python's Developer Guide · This issue has been migrated to GitHub: https://github.com/python/cpython/issues/56107
🌐
GitHub
gist.github.com › yoavram › 4351498
Example of uploading binary files programmatically in python, including both client and server code. Client implemented with the requests library and the server is implemented with the flask library. · GitHub
Example of uploading binary files programmatically in python, including both client and server code. Client implemented with the requests library and the server is implemented with the flask libra...
🌐
Python-requests
docs.python-requests.org › en › v1.2.3 › user › quickstart
https://docs.python-requests.org/en/v1.2.3/user/qu...
>>> url = 'http://httpbin.org/post' >>> files = {'file': ('report.xls', open('report.xls', 'rb'))} >>> r = requests.post(url, files=files) >>> r.text { ... "files": { "file": "<censored...binary...data>" }, ...
🌐
YouTube
youtube.com › watch
python requests post binary data - YouTube
Instantly Download or Run the code at https://codegive.com in this tutorial, we'll explore how to use the requests library in python to send post requests w...
Published   February 22, 2024
🌐
GitHub
github.com › psf › requests › issues › 403
Binary POST data stopped working in requests 0.10.1 · Issue #403 · psf/requests
January 30, 2012 - The following method for POSTing binary data worked with requests 0.10.0 on Python 2.7.1 (Ubuntu 11.04 x64) : bodybytes = '\xDE\xAD\xBE\xEF' r = requests.post(url, data=bodybytes) With requ...
Author   vimalb
🌐
Python
bugs.python.org › issue12398
Issue 12398: Sending binary data with a POST request in httplib can cause Unicode exceptions - Python tracker
This issue tracker has been migrated to GitHub, and is currently read-only. For more information, see the GitHub FAQs in the Python's Developer Guide · This issue has been migrated to GitHub: https://github.com/python/cpython/issues/56607
🌐
Requests
requests.readthedocs.io › en › latest › user › quickstart
Quickstart — Requests 2.33.0 documentation
>>> url = 'https://httpbin.org/post' >>> files = {'file': ('report.xls', open('report.xls', 'rb'), 'application/vnd.ms-excel', {'Expires': '0'})} >>> r = requests.post(url, files=files) >>> r.text { ... "files": { "file": "<censored...binary...data>" }, ...
🌐
Beautiful Soup
tedboy.github.io › requests › qs4.html
2.4. Binary Response Content — Requests API
For example, to create an image from binary data returned by a request, you can use the following code:
Find elsewhere
🌐
Python.org
discuss.python.org › python help
Python requests module with json data some fields are binary date - Python Help - Discussions on Python.org
March 20, 2024 - Hello In Python requests module how to send json data, some fields are a binary data. And doesn’t wants to convert base64 or file as well. Do we have any other options for sending json data along with binary data by u…
Top answer
1 of 1
15

The curl -d switch sends a POST request, but you are using requests.get() instead, sending a GET request (whose body is ignored).

Make it a POST instead, by using request.post():

import requests
import json

url = "http://cdcnepal.com/Modules/HOmeMoviesLists/WebService2.asmx/GetShowsByDate"
headers = {"content-type": "application/json; charset=UTF-8"}
payload = {"portalId":"1","showDate":"26/05/2014","flag":0,"size":9}
r = requests.post(url, headers=headers, data=json.dumps(payload))
print r.text

You also need to:

  1. not use a list for the content-type header, there is no support for paramaters being specified separately.
  2. Encode your JSON data to a JSON string; requests doesn't do this for you. Instead, a dictionary passed to data is encoded as application/x-www-form-urlencoded data instead.

You can compare the curl command with requests more easily using http://httpbin.org/post:

$ curl http://httpbin.org/post \
>  -H 'Content-Type: application/json; charset=UTF-8' \
>  -d '{"portalId":"1","showDate":"26/05/2014","flag":0,"size":9}'

{
  "args": {},
  "data": "{\"portalId\":\"1\",\"showDate\":\"26/05/2014\",\"flag\":0,\"size\":9}",
  "files": {},
  "form": {},
  "headers": {
    "Accept": "*/*",
    "Connection": "close",
    "Content-Length": "58",
    "Content-Type": "application/json; charset=UTF-8",
    "Host": "httpbin.org",
    "User-Agent": "curl/7.30.0",
    "X-Request-Id": "78d7bb7d-e29b-482b-908a-48d2395a050f"
  },
  "json": {
    "flag": 0,
    "portalId": "1",
    "showDate": "26/05/2014",
    "size": 9
  },
  "origin": "84.92.98.170",
  "url": "http://httpbin.org/post"
}

and

>>> import requests
>>> import json
>>> from pprint import pprint
>>> url = 'http://httpbin.org/post'
>>> headers = {"content-type":"application/json; charset=UTF-8"}
>>> payload = {"portalId":"1","showDate":"26/05/2014","flag":0,"size":9}
>>> r = requests.post(url, headers=headers, data=json.dumps(payload))
>>> pprint(r.json())
{u'args': {},
 u'data': u'{"portalId": "1", "flag": 0, "size": 9, "showDate": "26/05/2014"}',
 u'files': {},
 u'form': {},
 u'headers': {u'Accept': u'*/*',
              u'Accept-Encoding': u'gzip, deflate, compress',
              u'Connection': u'close',
              u'Content-Length': u'65',
              u'Content-Type': u'application/json; charset=UTF-8',
              u'Host': u'httpbin.org',
              u'User-Agent': u'python-requests/2.2.1 CPython/2.7.6 Darwin/13.1.0',
              u'X-Request-Id': u'06d6b542-c279-4898-8701-2c0d502aa36e'},
 u'json': {u'flag': 0,
           u'portalId': u'1',
           u'showDate': u'26/05/2014',
           u'size': 9},
 u'origin': u'84.92.98.170',
 u'url': u'http://httpbin.org/post'}

Both cases show the same json dictionary being returned.

If you are using requests version 2.4.2 or newer, you can also leave the JSON encoding to the library; it'll set the correct Content-Type header too, if you pass in the data to send as the json keyword argument:

import requests

url = "http://cdcnepal.com/Modules/HOmeMoviesLists/WebService2.asmx/GetShowsByDate"
payload = {"portalId":"1","showDate":"26/05/2014","flag":0,"size":9}
r = requests.post(url, json=payload)
print r.text
🌐
CSDN
devpress.csdn.net › python › 63044f14c67703293080ac25.html
Python POST binary data - DevPress官方社区
August 23, 2022 - import requests data = 'test data' res = requests.post(url='http://localhost:8888', data=data, headers={'Content-Type': 'application/octet-stream'}) ... POST http://localhost:8888/ HTTP/1.1 Host: localhost:8888 Content-Length: 9 Content-Type: application/octet-stream Accept-Encoding: gzip, deflate, compress Accept: */* User-Agent: python-requests/1.0.4 CPython/2.7.3 Windows/Vista test data
🌐
Python-requests
docs.python-requests.org › en › latest › user › quickstart
Quickstart — Requests 2.32.3 documentation - docs.python …
In the event you are posting a very large file as a multipart/form-data request, you may want to stream the request. By default, requests does not support this, but there is a separate package which does - requests-toolbelt. You should read the toolbelt’s documentation for more details about how to use it. For sending multiple files in one request refer to the advanced section. ... It is strongly recommended that you open files in binary mode.
🌐
WebScraping.AI
webscraping.ai › faq › requests › how-do-i-handle-binary-data-responses-with-requests
How do I handle binary data responses with Requests? | WebScraping.AI
# Download and verify an image file python -c " import requests response = requests.get('https://httpbin.org/image/png') with open('test.png', 'wb') as f: f.write(response.content) print(f'Downloaded {len(response.content)} bytes') print(f'Content-Type: {response.headers.get(\"content-type\")}') " # Check file integrity python -c " import hashlib with open('test.png', 'rb') as f: content = f.read() print(f'File size: {len(content)} bytes') print(f'MD5 hash: {hashlib.md5(content).hexdigest()}') " Handling binary data with Python Requests requires understanding the distinction between text and binary content, proper use of response.content, and implementing appropriate streaming for large files.
🌐
Strapi Community
forum.strapi.io › questions and answers
How do I upload binary files using Python? - Questions and Answers - Strapi Community Forum
January 15, 2022 - ▶ System Information on the Upload documentation here: Upload - Strapi Developer Docs The docs give a code snippet for uploading a binary file in JavaScript. How do I do it in Python? This is ...
Top answer
1 of 2
13

from what you say, you're running python3 (as the StringIO package has been renamed io in python3, not python2) and your example is python2 (for obvious reasons).

So for your issue:

"TypeError:initial_value must be str or None, not bytes".

What that means is that in:

response = requests.get('http://www.github.com')

you're either getting None or a response in bytes for response.content. Given that your request worked, and you can access response.content, it is very likely to be in bytes.

As the requests library works at a quite low level, and all data coming in and to sockets (including the HTTP socket) is plain binary (i.e. not interpreted), to be able to use the output in string functions you need to convert it into something.

In python3 str is the old unicode from python2, and bytes is close to the old str of python2. So you would need to convert the bytes into a string to feed StringIO:

i = Image.open(StringIO(response.content.decode('utf-8')))

for example. But then I'm expecting Image.open() to yell at you that it does not know wtf it is supposed to do with a unicode buffer, all it really wants is a byte array!

But because Image.open() is actually expecting a stream of bytes, and not a unicode stream, what you shall be doing is actually use a BytesIO instead of a StringIO:

from io import BytesIO
i = Image.open(BytesIO(response.content))

Finally, you're sweet to give an example, but it's not one that would work, as you're giving a link to an HTML page, instead of an image.

HTH

2 of 2
0

It's a good idea to actually fetch an image from the internet if one wants to parse images :D (as opposed to fetching the index page at github.com)

import requests
from PIL import Image
from StringIO import StringIO

url = "https://upload.wikimedia.org/wikipedia/commons/thumb/4/46/Venn0110.svg/576px-Venn0110.svg.png"
response = requests.get(url)
i = Image.open(StringIO(response.content))

The example you're trying to use looks different from what you've posted here:

3.3.4 Binary Response Content
You can also access the response body as bytes, for non-text requests:
>>> r.content
b'[{"repository":{"open_issues":0,"url":"https://github.com/...
The gzip and deflate transfer-encodings are automatically decoded for you.
For example, to create an image from binary data returned by a request, you can use the following code:
>>> from PIL import Image
>>> from StringIO import StringIO
>>> i = Image.open(StringIO(r.content))

https://github.com/... <-- these three dots (ellipses) indicate that the URL has been shortened in the example.

source: Requests Documentation Release 2.9.1

🌐
Requests
requests.readthedocs.io › en › v1.2.3 › user › quickstart
Quickstart — Requests 1.2.3 documentation
>>> url = 'http://httpbin.org/post' >>> files = {'file': ('report.xls', open('report.xls', 'rb'))} >>> r = requests.post(url, files=files) >>> r.text { ... "files": { "file": "<censored...binary...data>" }, ...
🌐
ProxiesAPI
proxiesapi.com › articles › downloading-binary-files-with-python-requests
Downloading Binary Files with Python Requests | ProxiesAPI
By default, the response from requests is decoded as text (UTF-8). To treat the response as a binary file instead, you need to set the ... This ensures requests doesn't try to decode the binary data as text.