Using df.to_csv() with the keyword argument compression='gzip' should produce a gzip archive. I tested it using same keyword arguments as you, and it worked.

You may need to upgrade pandas, as gzip was not implemented until version 0.17.1, but trying to use it on prior versions will not raise an error, and just produce a regular csv. You can determine your current version of pandas by looking at the output of pd.__version__.

The main issue I have with this is that I have to generate an intermediary file that is used up by the splitter/csv writer for writing the csv. https://docs.python.org/3/library/tempfile.html One of my favorite libraries. Does exactly what you're looking for.

Use the gzip module:

with gzip.open(filename, mode='rt') as f:
    reader = csv.reader(f)
    #...

Default mode for gzip.open is rb, if you wish to work with strs, you have to specify it extra:

f = gzip.open(filename, mode="rt")

OT: it is a good practice to write I/O operations in a with block:

with gzip.open(filename, mode="rt") as f:

Hi ! According to the error message, it seems you have a "<" somewhere in the path or name of your file, which is not allowed. I noticed that you are declaring a file handle (f, at the line starting by "with gzip.open"), and later on it seems you are building a path with it ( "/" + today2 + "/" + today1 + "/" + str(f) ) . This will print the object definition in the path, which do contains "<"... Did you mean "/" + today2 + "/" + today1 + "/" + str(f.name) ? Regards, Alex

You need to change the mode of gzip to wt :

fz = gzip.open(fn, 'wt')

Also a little-known feature of gzip.open() and bz2.open() is that they can be layered on top of an existing file opened in binary mode. For example, this works:

import gzip
f = open('somefile.gz', 'rb')
with gzip.open(f, 'rt') as g:
    text = g.read()

This allows the gzip and bz2 modules to work with various file-like objects such as sockets, pipes, and in-memory files.

Pretty much what you've already done, except read_csv also has nrows where you can specify the number of rows you want from the data set.

Additionally, to prevent the errors you were getting, you can set error_bad_lines to False. You'll still get warnings (if that bothers you, set warn_bad_lines to False as well). These are there to indicate inconsistency in how your dataset is filled out.

import pandas as pd
data = pd.read_csv('google-us-data.csv.gz', nrows=100, compression='gzip',
                   error_bad_lines=False)
print(data)

You can easily do something similar with the csv built-in library, but it'll require a for loop to iterate over the data, has shown in other examples.

I've read and considered your concern w/keeping the GZip file in binary mode, and I think you can still use TextIOWrapper. My understanding is that its job is to provide an interface for writing bytes from text (my emphasis):

A buffered text stream providing higher-level access to a BufferedIOBase buffered binary stream.

I interpret that as "text in, bytes out"... which is what your GZip application needs, right? If so, then for Python3 we need to give the CSV writer something that accepts strings but ultimately writes bytes.

Enter TextIOWrapper with a UTF-8 encoding, accepting strings from csv.writer's writerow/s() methods and writing UTF-8-encoded bytes to gzip_file.

I've run this in Python2 and 3, and unzipped the file and it looks good:

import csv, gzip, io, six

def get_gzip_writer(path):
  with open(path, 'wb') as s3_file:
    with gzip.GzipFile(fileobj=s3_file, mode='wb') as gzip_file:
        if six.PY3:
            with io.TextIOWrapper(gzip_file, encoding='utf-8') as wrapper:
                yield csv.writer(wrapper)
        elif six.PY2:
            yield csv.writer(gzip_file)
        else:
            raise ValueError('Neither Python2 or 3?!')


data = [[1,2,3],['a','b','c']]
url = 'output.gz'

for writer in get_gzip_writer(url):
    for row in data:
        writer.writerow(row)