Basically what you do is correct. Looking at redmine docs you linked to, it seems that suffix after the dot in the url denotes type of posted data (.json for JSON, .xml for XML), which agrees with the response you get - Processing by AttachmentsController#upload as XML. I guess maybe there's a bug in docs and to post binary data you should try using http://redmine/uploads url instead of http://redmine/uploads.xml.

Btw, I highly recommend very good and very popular Requests library for http in Python. It's much better than what's in the standard lib (urllib2). It supports authentication as well but I skipped it for brevity here.

import requests
with open('./x.png', 'rb') as f:
    data = f.read()
res = requests.post(url='http://httpbin.org/post',
                    data=data,
                    headers={'Content-Type': 'application/octet-stream'})

# let's check if what we sent is what we intended to send...
import json
import base64

assert base64.b64decode(res.json()['data'][len('data:application/octet-stream;base64,'):]) == data

UPDATE

To find out why this works with Requests but not with urllib2 we have to examine the difference in what's being sent. To see this I'm sending traffic to http proxy (Fiddler) running on port 8888:

Using Requests

import requests

data = 'test data'
res = requests.post(url='http://localhost:8888',
                    data=data,
                    headers={'Content-Type': 'application/octet-stream'})

we see

POST http://localhost:8888/ HTTP/1.1
Host: localhost:8888
Content-Length: 9
Content-Type: application/octet-stream
Accept-Encoding: gzip, deflate, compress
Accept: */*
User-Agent: python-requests/1.0.4 CPython/2.7.3 Windows/Vista

test data

and using urllib2

import urllib2

data = 'test data'    
req = urllib2.Request('http://localhost:8888', data)
req.add_header('Content-Length', '%d' % len(data))
req.add_header('Content-Type', 'application/octet-stream')
res = urllib2.urlopen(req)

we get

POST http://localhost:8888/ HTTP/1.1
Accept-Encoding: identity
Content-Length: 9
Host: localhost:8888
Content-Type: application/octet-stream
Connection: close
User-Agent: Python-urllib/2.7

test data

I don't see any differences which would warrant different behavior you observe. Having said that it's not uncommon for http servers to inspect User-Agent header and vary behavior based on its value. Try to change headers sent by Requests one by one making them the same as those being sent by urllib2 and see when it stops working.

Answer from Piotr Dobrogost on Stack Overflow
๐ŸŒ
Hustle Play
hustleplay.wordpress.com โ€บ 2010 โ€บ 01 โ€บ 05 โ€บ sending-binary-with-json
Sending Binary with JSON | Hustle Play - WordPress.com
January 5, 2010 - In my work, I used Python on the packaging side and Java on the unpacking side. First, you need to open the file. Since the file is binary, you have to open it with an additional "b" flag.โ€ฆ
Top answer
1 of 4
104

Basically what you do is correct. Looking at redmine docs you linked to, it seems that suffix after the dot in the url denotes type of posted data (.json for JSON, .xml for XML), which agrees with the response you get - Processing by AttachmentsController#upload as XML. I guess maybe there's a bug in docs and to post binary data you should try using http://redmine/uploads url instead of http://redmine/uploads.xml.

Btw, I highly recommend very good and very popular Requests library for http in Python. It's much better than what's in the standard lib (urllib2). It supports authentication as well but I skipped it for brevity here.

import requests
with open('./x.png', 'rb') as f:
    data = f.read()
res = requests.post(url='http://httpbin.org/post',
                    data=data,
                    headers={'Content-Type': 'application/octet-stream'})

# let's check if what we sent is what we intended to send...
import json
import base64

assert base64.b64decode(res.json()['data'][len('data:application/octet-stream;base64,'):]) == data

UPDATE

To find out why this works with Requests but not with urllib2 we have to examine the difference in what's being sent. To see this I'm sending traffic to http proxy (Fiddler) running on port 8888:

Using Requests

import requests

data = 'test data'
res = requests.post(url='http://localhost:8888',
                    data=data,
                    headers={'Content-Type': 'application/octet-stream'})

we see

POST http://localhost:8888/ HTTP/1.1
Host: localhost:8888
Content-Length: 9
Content-Type: application/octet-stream
Accept-Encoding: gzip, deflate, compress
Accept: */*
User-Agent: python-requests/1.0.4 CPython/2.7.3 Windows/Vista

test data

and using urllib2

import urllib2

data = 'test data'    
req = urllib2.Request('http://localhost:8888', data)
req.add_header('Content-Length', '%d' % len(data))
req.add_header('Content-Type', 'application/octet-stream')
res = urllib2.urlopen(req)

we get

POST http://localhost:8888/ HTTP/1.1
Accept-Encoding: identity
Content-Length: 9
Host: localhost:8888
Content-Type: application/octet-stream
Connection: close
User-Agent: Python-urllib/2.7

test data

I don't see any differences which would warrant different behavior you observe. Having said that it's not uncommon for http servers to inspect User-Agent header and vary behavior based on its value. Try to change headers sent by Requests one by one making them the same as those being sent by urllib2 and see when it stops working.

2 of 4
4

This has nothing to do with a malformed upload. The HTTP error clearly specifies 401 unauthorized, and tells you the CSRF token is invalid. Try sending a valid CSRF token with the upload.

More about csrf tokens here:

What is a CSRF token ? What is its importance and how does it work?

Discussions

Upload file as JSON to Python webserver
You are trying to upload binary files (word, jpg), serialised as JSON, and store them on the server. More on stackoverflow.com
๐ŸŒ stackoverflow.com
August 31, 2018
Python - Send file through JSON - Stack Overflow
I'm trying to create a live chat application using ws4py (if there are better socket libraries I would love recommendations) and I want to implement file transfer (specifically .wav files) in my More on stackoverflow.com
๐ŸŒ stackoverflow.com
March 20, 2017
python - sending binary data over json - Stack Overflow
I wanted to upload file to s3 using python. I am using requests_aws4 auth library for this import requests from requests_aws4auth import AWS4Auth # data=encode_audio(data) endpoint... More on stackoverflow.com
๐ŸŒ stackoverflow.com
April 21, 2017
rest - Uploading binary file to API using python - Stack Overflow
I am trying to upload a package binary over to the RestAPI of storage using python. But it keeps throwing error and couldn't upload the file. Below is the code I am using to achieve it : jsonhead... More on stackoverflow.com
๐ŸŒ stackoverflow.com
๐ŸŒ
Python
docs.python.org โ€บ 3 โ€บ library โ€บ json.html
JSON encoder and decoder โ€” Python 3.14.3 documentation
February 23, 2026 - JSONDecodeError โ€“ When the data being deserialized is not a valid JSON document. UnicodeDecodeError โ€“ When the data being deserialized does not contain UTF-8, UTF-16 or UTF-32 encoded data. ... Added the optional object_pairs_hook parameter. parse_constant doesnโ€™t get called on โ€˜nullโ€™, โ€˜trueโ€™, โ€˜falseโ€™ anymore. ... All optional parameters are now keyword-only. fp can now be a binary file.
๐ŸŒ
Stack Overflow
stackoverflow.com โ€บ questions โ€บ 43538675 โ€บ sending-binary-data-over-json
python - sending binary data over json - Stack Overflow
April 21, 2017 - I wanted to upload file to s3 using python. I am using requests_aws4 auth library for this ยท import requests from requests_aws4auth import AWS4Auth # data=encode_audio(data) endpoint = 'http://bucket.s3.amazonaws.com/testing89.mp3' data = //some binary data(mp3) here auth = AWS4Auth('xxxxxx', 'xxxxxx', 'eu-west-2', 's3') response = requests.put(endpoint,data=data, auth=auth, headers={'x-amz-acl': 'public-read'}) print response.text ยท
Find elsewhere
๐ŸŒ
Python for the Lab
pythonforthelab.com โ€บ blog โ€บ storing-binary-data-and-serializing
Storing Binary Data and Serializing | Python For The Lab
August 11, 2018 - The limit of JSON is, however, that you have to store data as text files, thus limiting its native capabilities. Fortunately, combining Pickle and base64, you can transform bytes to an ascii string and save it next to easy to read metadata. This article has gone much more in depth regarding how to store data in different formats, but the topic is far from complete. Keep tuned to find more articles regarding how to save data with Python.
๐ŸŒ
cyberangles
cyberangles.org โ€บ blog โ€บ using-python-requests-to-send-file-and-json-in-single-request
How to Use Python Requests to Send File and JSON in a Single POST Request (Troubleshooting Guide) โ€” CyberAngles.org
The key steps are: Serialize JSON data to a string with json.dumps(). Package JSON and files into a files dictionary, with explicit content types. Open files in binary mode ("rb") to avoid corruption.
๐ŸŒ
GitHub
github.com โ€บ fangq โ€บ pybj
GitHub - fangq/pybj: Binary JSON (BJData/UBJSON) Support for Python ยท GitHub
python3 -mbjdata USAGE: bjdata (fromjson|tojson) (INFILE|-) [OUTFILE] EXAMPLES: python3 -mbjdata fromjson input.json output.bjd python3 -mbjdata tojson input.bjd output.json
Author ย  fangq
๐ŸŒ
Quora
quora.com โ€บ How-do-you-send-binary-data-in-JSON
How to send binary data in JSON - Quora
When interoperability matters, specify padding and URL-safe vs standard Base64 in API docs. Summary Encode binary into text (Base64 is the default) or place binary outside JSON and reference it.
๐ŸŒ
FastAPI
fastapi.tiangolo.com โ€บ tutorial โ€บ request-files
Request Files - FastAPI
You can declare multiple File and Form parameters in a path operation, but you can't also declare Body fields that you expect to receive as JSON, as the request will have the body encoded using multipart/form-data instead of application/json. This is not a limitation of FastAPI, it's part of the HTTP protocol. You can make a file optional by using standard type annotations and setting a default value of None: ... from typing import Annotated from fastapi import FastAPI, File, UploadFile app = FastAPI() @app.post("/files/") async def create_file(file: Annotated[bytes | None, File()] = None): if not file: return {"message": "No file sent"} else: return {"file_size": len(file)} @app.post("/uploadfile/") async def create_upload_file(file: UploadFile | None = None): if not file: return {"message": "No upload file sent"} else: return {"filename": file.filename}
๐ŸŒ
PyTutorial
pytutorial.com โ€บ handling-binary-data-in-python-json
PyTutorial | Handling Binary Data in Python JSON
November 7, 2024 - # Reading binary file with open('image.png', 'rb') as file: binary_data = file.read() # Encode and save to JSON image_data = { 'filename': 'image.png', 'content': base64.b64encode(binary_data).decode('utf-8') } with open('image_data.json', 'w') as json_file: json.dump(image_data, json_file)
๐ŸŒ
Stack Abuse
stackabuse.com โ€บ how-to-upload-files-with-pythons-requests-library
How to Upload Files with Python's requests Library
September 19, 2021 - The path of the file can be an absolute path or a relative path to where the script is being run. If you're uploading a file in the same directory, you can just use the file's name. The second argument, mode, will take the "read binary" value which is represented by rb.
๐ŸŒ
PyPI
pypi.org โ€บ project โ€บ pbjson
pbjson ยท PyPI
The pbjson module works ust like the json module. You can pbjson.load, pbjson.loads, pbjson.dump, and pbjson.dumps. After you have installed pbjson, you can use the pbjson command-line tool to convert files to or from pbjson. Run pbjson -h for details. Packed Binary JSON is not the same as BSON.
      ยป pip install pbjson
    
Published ย  Aug 20, 2023
Version ย  1.19.0
๐ŸŒ
Stack Overflow
stackoverflow.com โ€บ questions โ€บ 21645910 โ€บ binary-file-upload-python-pycurl-c
Binary file upload (python, PyCurl, C#) - Stack Overflow
WebClient client = new WebClient(); client.UploadFileAsync(uri, "POST", "C:\\Users\\user1\\file\\to\\the\\firmware\\fw-container.efc"); Here the problem is that upload does not succeed (doesn't even start uploading in Python) although the script runs through. The problem should be in the code, because other way the upload could be completed Python code: import pycurl from cStringIO import StringIO import urllib2 import simplejson as json url = 'http://eData/pvi?rName=FirmwareUpload' req = urllib2.Request(url) req.add_header('Content-Type','application/json') c = pycurl.Curl() c.setopt(c.POST,