Try something like this
import requests
data='param1,state=test,param2=1 param3=2.932,param4=3250 1497064544944 '
p = requests.post('http://myServerAddress/write?db=some_data', data.encode())
Answer from Junyong Yao on Stack OverflowThe issue seems to be that you're missing a trailing slash '/' on the end of the URL. Without the trailing slash the server seems to redirect to a non-existent page and you get the 404.
To fix, just add a trailing slash:
requests.post('https://cleardata.proofhub.com/files/upload/', data=data, headers=create_headers)
# Add slash ^
I think you need to post your file as form data using the files argument.
files = {'file': open(r"C:\Users\dlogan.CLEARDATA\Desktop\ProofHub Upload\test.txt",'rb')
}
create_headers = {'X-API-KEY': '', 'Content-Type': 'application/json', 'User-Agent': '@cleardata.co.uk'}
r = requests.post('https://cleardata.proofhub.com/files/upload', files=files, headers=create_headers)
You'll need to get the file name right - I can't see the post form, so I don't know what should be.
Basically what you do is correct. Looking at redmine docs you linked to, it seems that suffix after the dot in the url denotes type of posted data (.json for JSON, .xml for XML), which agrees with the response you get - Processing by AttachmentsController#upload as XML. I guess maybe there's a bug in docs and to post binary data you should try using http://redmine/uploads url instead of http://redmine/uploads.xml.
Btw, I highly recommend very good and very popular Requests library for http in Python. It's much better than what's in the standard lib (urllib2). It supports authentication as well but I skipped it for brevity here.
import requests
with open('./x.png', 'rb') as f:
data = f.read()
res = requests.post(url='http://httpbin.org/post',
data=data,
headers={'Content-Type': 'application/octet-stream'})
# let's check if what we sent is what we intended to send...
import json
import base64
assert base64.b64decode(res.json()['data'][len('data:application/octet-stream;base64,'):]) == data
UPDATE
To find out why this works with Requests but not with urllib2 we have to examine the difference in what's being sent. To see this I'm sending traffic to http proxy (Fiddler) running on port 8888:
Using Requests
import requests
data = 'test data'
res = requests.post(url='http://localhost:8888',
data=data,
headers={'Content-Type': 'application/octet-stream'})
we see
POST http://localhost:8888/ HTTP/1.1
Host: localhost:8888
Content-Length: 9
Content-Type: application/octet-stream
Accept-Encoding: gzip, deflate, compress
Accept: */*
User-Agent: python-requests/1.0.4 CPython/2.7.3 Windows/Vista
test data
and using urllib2
import urllib2
data = 'test data'
req = urllib2.Request('http://localhost:8888', data)
req.add_header('Content-Length', '%d' % len(data))
req.add_header('Content-Type', 'application/octet-stream')
res = urllib2.urlopen(req)
we get
POST http://localhost:8888/ HTTP/1.1
Accept-Encoding: identity
Content-Length: 9
Host: localhost:8888
Content-Type: application/octet-stream
Connection: close
User-Agent: Python-urllib/2.7
test data
I don't see any differences which would warrant different behavior you observe. Having said that it's not uncommon for http servers to inspect User-Agent header and vary behavior based on its value. Try to change headers sent by Requests one by one making them the same as those being sent by urllib2 and see when it stops working.
This has nothing to do with a malformed upload. The HTTP error clearly specifies 401 unauthorized, and tells you the CSRF token is invalid. Try sending a valid CSRF token with the upload.
More about csrf tokens here:
What is a CSRF token ? What is its importance and how does it work?
The requests library is meant to keep things like this simple:
import requests
r = requests.post('http://server/myapp/method', data={'aaa': 'bbb'})
Or depending on how the receiving end expects data:
import requests
r = requests.post('http://server/myapp/method',
data=file('binary_data_file.bin','rb').read())
From libcurl, setopt(...) try this option:
CURLOPT_POSTFIELDSIZE
If you want to post data to the server without letting libcurl do a strlen() to measure the data size, this option must be used. When this option is used you can post fully binary data, which otherwise is likely to fail. If this size is set to -1, the library will use strlen() to get the size.
http://curl.haxx.se/libcurl/c/curl_easy_setopt.html#CURLOPTPOSTFIELDSIZE
The "HTTP Error 500" may be from forgetting to include an index name or index type.
Also: for bulk inserts, elasticsearch requires a trailing "\n" character after the last record, or it won't insert that record.
Try:
import urllib2
import json
url = 'http://localhost:9200/myindex/mydoc/_bulk?pretty=true'
data = json.dumps({"index": {"_parent": "btaCovzjQhqrP4s3iPjZKQ"}}) + "\n" + json.dumps({"title":"hello"}) + "\n" + json.dumps({"index": {"_parent": "btaCovzjQhqrP4s3iPjZKQ"}}) + "\n" + json.dumps({"title":"world"})
req = urllib2.Request(url,data=data+"\n")
f = urllib2.urlopen(req)
print f.read()
Or, with some refactoring:
import urllib2
import json
url = 'http://localhost:9200/myindex/mydoc/_bulk?pretty=true'
data = [
{"index": {"_parent": "btaCovzjQhqrP4s3iPjZKQ"}},
{"title":"hello"},
{"index": {"_parent": "btaCovzjQhqrP4s3iPjZKQ"}},
{"title":"world"}
]
encoded_data = "\n".join(map(json.dumps,data)) + "\n"
req = urllib2.Request(url,data=encoded_data)
f = urllib2.urlopen(req)
print f.read()
The usecase for me is actually a bulk_index request for ElasticSearch.
This is made a lot easier with rawes:
import rawes
es = rawes.Elastic('localhost:9200')
with open('myfile.json') as f:
lines = f.readlines()
es.post('someindex/sometype/_bulk', data=lines)
Python requests its a great library for this kind of stuff. What you have there can be simply done by:
import requests
headers = {'content-type': 'text/xml; charset=utf-8'}
response = requests.post(url, data="<data>the data is here</data>", headers=headers)
with open("filename.xml", "w") as fd:
fd.write(response.text)
The problem of pycurl and some other url and http client libraries for python is that it requires more effort than the one you should need to achieve something relatively simple. requests its way more user friendly and I think its what you are looking for on this problem.
Hope this helps
The question is about uploading a file, while the accepted answer was just saving to file!
Below the correct code:
import requests
# Here you set the file you want to upload and it's content type
files = {'upload_file': ('filename.xml', open('filename.xml','rb'), 'text/xml' }
headers = {} # Do not set content type here, let the library do its job
response = requests.post(url,
data="<data>the data is here</data>",
files=files,
headers=headers)
fd = open("output-response.txt", "w")
fd.write(response.text)
fd.close()
The code above will read a file from filename.xml and upload it via POST, then it will also store to output-response.txt the response received.