Any suggestions?

Use Streaming Upload, as docs put it:

Requests supports streaming uploads, which allow you to send large streams or files without reading them into memory. To stream and upload, simply provide a file-like object for your body:

with open('massive-body', 'rb') as f:
    requests.post('http://some.url/streamed', data=f)

You can use the requests-toolbelt to do this:

import requests
from requests_toolbelt.multipart import encoder

session = requests.Session()
with open('my_file.csv', 'rb') as f:
    form = encoder.MultipartEncoder({
        "documents": ("my_file.csv", f, "application/octet-stream"),
        "composite": "NONE",
    })
    headers = {"Prefer": "respond-async", "Content-Type": form.content_type}
    resp = session.post(url, headers=headers, data=form)
session.close()

This will cause requests to stream the multipart/form-data upload for you.

Reading through the mailing list thread linked to by systempuntoout, I found a clue towards the solution.

The mmap module allows you to open file that acts like a string. Parts of the file are loaded into memory on demand.

Here's the code I'm using now:

Copyimport urllib2
import mmap

# Open the file as a memory mapped string. Looks like a string, but 
# actually accesses the file behind the scenes. 
f = open('somelargefile.zip','rb')
mmapped_file_as_string = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ)

# Do the request
request = urllib2.Request(url, mmapped_file_as_string)
request.add_header("Content-Type", "application/zip")
response = urllib2.urlopen(request)

#close everything
mmapped_file_as_string.close()
f.close()

Using an open file object as the data parameter ensures that requests will stream the data for you.

If a file size can be determined (via the OS filesystem), the file object is streamed using a 8kb buffer. If no filesize can be determined, a Transfer-Encoding: chunked request is sent sending the data per line instead (the object is used as an iterable).

If you were to use the files= parameter for a multipart POST, on the other hand, the file would be loaded into memory before sending. Use the requests-toolbelt package to stream multi-part uploads:

import requests
from requests_toolbelt.multipart.encoder import MultipartEncoder

csvfile = '/path/file.csv'
with open(csvfile) as f:
    m = MultipartEncoder(fields={'csv_field_name': ('file.csv', f, 'text/csv')})
    headers = {'Content-Type': m.content_type}
    r = requests.post(url, data=m, headers=headers)

The missing piece is to turn on streaming with stream=True—that's what tells Requests not to read the whole content into memory before you have a chance to look at it.

With the following streaming code, the Python memory usage is restricted regardless of the size of the downloaded file:

def download_file(url):
    local_filename = url.split('/')[-1]
    # NOTE the stream=True parameter below
    with requests.get(url, stream=True) as r:
        r.raise_for_status()
        with open(local_filename, 'wb') as f:
            for chunk in r.iter_content(chunk_size=8192): 
                # If you have chunk encoded response uncomment if
                # and set chunk_size parameter to None.
                #if chunk: 
                f.write(chunk)
    return local_filename

Note that the number of bytes returned using iter_content is not exactly the chunk_size; it's expected to be a random number that is often far bigger, and is expected to be different in every iteration.

See body-content-workflow and Response.iter_content for further reference.

This works fairly well for me (Python3 with requests):

def upload_file(local_file, remote_file):  
    params = {"file": os.path.basename(remote_file),
              "folder": os.path.dirname(remote_file),
              "submit": "Submit"}
    with open(local_file, 'rb') as file_:
        try:
           response = requests.post(url=URL, data=params, auth=(USER, PASSWORD),
                                    files={"zip_file": file_}, verify=False)
        except TimeoutError:
            print("Connection timed out!")
        else:
            print(response)