It's quite kludgy (self referencing, etc; just put a few minutes writing it, nothing really elegant), but it does what you want if you're still interested in using gzip instead of zlib directly.

Basically, GzipWrap is a (very limited) file-like object that produces a gzipped file out of a given iterable (e.g., a file-like object, a list of strings, any generator...)

Of course, it produces binary so there was no sense in implementing "readline".

You should be able to expand it to cover other cases or to be used as an iterable object itself.

from gzip import GzipFile

class GzipWrap(object):
    # input is a filelike object that feeds the input
    def __init__(self, input, filename = None):
        self.input = input
        self.buffer = ''
        self.zipper = GzipFile(filename, mode = 'wb', fileobj = self)

    def read(self, size=-1):
        if (size < 0) or len(self.buffer) < size:
            for s in self.input:
                self.zipper.write(s)
                if size > 0 and len(self.buffer) >= size:
                    self.zipper.flush()
                    break
            else:
                self.zipper.close()
            if size < 0:
                ret = self.buffer
                self.buffer = ''
        else:
            ret, self.buffer = self.buffer[:size], self.buffer[size:]
        return ret

    def flush(self):
        pass

    def write(self, data):
        self.buffer += data

    def close(self):
        self.input.close()
Answer from Ricardo Cárdenes on Stack Overflow
Top answer
1 of 6
15

It's quite kludgy (self referencing, etc; just put a few minutes writing it, nothing really elegant), but it does what you want if you're still interested in using gzip instead of zlib directly.

Basically, GzipWrap is a (very limited) file-like object that produces a gzipped file out of a given iterable (e.g., a file-like object, a list of strings, any generator...)

Of course, it produces binary so there was no sense in implementing "readline".

You should be able to expand it to cover other cases or to be used as an iterable object itself.

from gzip import GzipFile

class GzipWrap(object):
    # input is a filelike object that feeds the input
    def __init__(self, input, filename = None):
        self.input = input
        self.buffer = ''
        self.zipper = GzipFile(filename, mode = 'wb', fileobj = self)

    def read(self, size=-1):
        if (size < 0) or len(self.buffer) < size:
            for s in self.input:
                self.zipper.write(s)
                if size > 0 and len(self.buffer) >= size:
                    self.zipper.flush()
                    break
            else:
                self.zipper.close()
            if size < 0:
                ret = self.buffer
                self.buffer = ''
        else:
            ret, self.buffer = self.buffer[:size], self.buffer[size:]
        return ret

    def flush(self):
        pass

    def write(self, data):
        self.buffer += data

    def close(self):
        self.input.close()
2 of 6
11

Here is a cleaner, non-self-referencing version based on Ricardo Cárdenes' very helpful answer.

from gzip import GzipFile
from collections import deque


CHUNK = 16 * 1024


class Buffer (object):
    def __init__ (self):
        self.__buf = deque()
        self.__size = 0
    def __len__ (self):
        return self.__size
    def write (self, data):
        self.__buf.append(data)
        self.__size += len(data)
    def read (self, size=-1):
        if size < 0: size = self.__size
        ret_list = []
        while size > 0 and len(self.__buf):
            s = self.__buf.popleft()
            size -= len(s)
            ret_list.append(s)
        if size < 0:
            ret_list[-1], remainder = ret_list[-1][:size], ret_list[-1][size:]
            self.__buf.appendleft(remainder)
        ret = ''.join(ret_list)
        self.__size -= len(ret)
        return ret
    def flush (self):
        pass
    def close (self):
        pass


class GzipCompressReadStream (object):
    def __init__ (self, fileobj):
        self.__input = fileobj
        self.__buf = Buffer()
        self.__gzip = GzipFile(None, mode='wb', fileobj=self.__buf)
    def read (self, size=-1):
        while size < 0 or len(self.__buf) < size:
            s = self.__input.read(CHUNK)
            if not s:
                self.__gzip.close()
                break
            self.__gzip.write(s)
        return self.__buf.read(size)

Advantages:

  • Avoids repeated string concatenation, which would cause the entire string to be copied repeatedly.
  • Reads a fixed CHUNK size from the input stream, instead of reading whole lines at a time (which can be arbitrarily long).
  • Avoids circular references.
  • Avoids misleading public "write" method of GzipCompressStream(), which is really only used internally.
  • Takes advantage of name mangling for internal member variables.
🌐
PyPI
pypi.org › project › gzip-stream
gzip-stream · PyPI
GZIPCompressedStream does not read entire stream, but instead read it by chunks, until compressed output size will not satisfy read size. AsyncGZIPDecompressedStream class can async read from another source with zlib and gzip decompression on-the-fly · # aiobotocore example import aiobotocore from gzip_stream import AsyncGZIPDecompressedStream AWS_ACCESS_KEY_ID = "KEY_ID" AWS_SECRET_ACCESS_KEY = "ACCESS_KEY" BUCKET = "AWESOME_BUCKET" upload_client = MyAsyncUploadClient() session = aiobotocore.get_session() async with session.create_client( service_name="s3", endpoint_url="s3_endpoint", aws_se
      » pip install gzip-stream
    
Published   Dec 08, 2021
Version   1.2.0
🌐
GitHub
github.com › commoncrawl › gzipstream
GitHub - commoncrawl/gzipstream: gzipstream allows Python to process multi-part gzip files from a streaming source
from gzipstream import GzipStreamFile f = open('huge_file.gz') # Any streaming file object that supports `read` gz = GzipStreamFile(f)
Starred by 23 users
Forked by 12 users
Languages   Python 100.0% | Python 100.0%
🌐
Python Module of the Week
pymotw.com › 2 › gzip
gzip – Read and write GNU zip files - Python Module of the Week
The same line, over and over. The same line, over and over. The same line, over and over. The same line, over and over. The same line, over and over. The same line, over and over. To read data back from previously compressed files, simply open the file with mode 'r'.
🌐
GitHub
github.com › leenr › gzip-stream
GitHub - leenr/gzip-stream: Streaming GZIP compression for Python · GitHub
GZIPCompressedStream does not read entire stream, but instead read it by chunks, until compressed output size will not satisfy read size. AsyncGZIPDecompressedStream class can async read from another source with zlib and gzip decompression on-the-fly · # aiobotocore example import aiobotocore from gzip_stream import AsyncGZIPDecompressedStream AWS_ACCESS_KEY_ID = "KEY_ID" AWS_SECRET_ACCESS_KEY = "ACCESS_KEY" BUCKET = "AWESOME_BUCKET" upload_client = MyAsyncUploadClient() session = aiobotocore.get_session() async with session.create_client( service_name="s3", endpoint_url="s3_endpoint", aws_se
Starred by 32 users
Forked by 5 users
Languages   Python
🌐
Python
docs.python.org › 3 › library › gzip.html
gzip — Support for gzip files
Example of how to read a compressed file: import gzip with gzip.open('/home/joe/file.txt.gz', 'rb') as f: file_content = f.read() Example of how to create a compressed GZIP file: import gzip content = b"Lots of content here" with gz
🌐
GitHub
github.com › Smerity › gzipstream
GitHub - Smerity/gzipstream: gzipstream allows Python to process multi-part gzip files from a streaming source · GitHub
from gzipstream import GzipStreamFile f = open('huge_file.gz') # Any streaming file object that supports `read` gz = GzipStreamFile(f)
Starred by 17 users
Forked by 19 users
Languages   Python
🌐
DevGenius
blog.devgenius.io › optimizing-memory-usage-streaming-gzip-files-from-s3-in-python-e04e2bf1ec19
Optimizing Memory Usage: Streaming GZIP Files from S3 in Python | by John Sener | Dev Genius
August 1, 2024 - with gzip.GzipFile(fileobj=body) as gz: decompressed_chunks = [] while True: chunk = gz.read(1024) if not chunk: break decompressed_chunks.append(chunk) file_content = b''.join(decompressed_chunks).decode('utf-8') return file_content · ...
Find elsewhere
🌐
GitHub
gist.github.com › beaufour › 4205533
Python: Streaming Gzip reader · GitHub
June 8, 2017 - Python: Streaming Gzip reader · Raw · gzipinputstream.py · This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters · Show hidden characters · Copy link · Have an example on use case?
🌐
Python Module of the Week
pymotw.com › 3 › gzip
gzip — Read and Write GNU zip Files
March 18, 2018 - import gzip with gzip.open('example.txt.gz', 'rb') as input_file: print('Entire file:') all_data = input_file.read() print(all_data) expected = all_data[5:15] # rewind to beginning input_file.seek(0) # move ahead 5 bytes input_file.seek(5) print('Starting at position 5 for 10 bytes:') partial ...
🌐
Michaelehead
michaelehead.com › 2019 › 12 › 28 › python-streaming-unzip.html
Unzipping a large gzip file in Python | Michael Head
December 28, 2019 - Pandas can “stream” unzip and decompress a file, grabbing lines of a few chunks at a time (say 10000 lines), and inserting it into a database. I was pretty impressed, given the gzipped file size. Inserting all those records via Pandas took between 4-5 hours.
🌐
Spacewalkproject
spacewalkproject.github.io › documentation › python-gzipstream › gzipstream.GzipStream-class.html
gzipstream.GzipStream
This doesn't allow for streaming gzipped data to be processed easily (e.g. can't seek a socket). Using the _StreamBuf class enables streaming gzipped data to be processed by buffering that data at it passes through. For Python versions 1.5.2 & 2.1.*: Normal data version.
🌐
Ejosh
ejosh.co › de › 2022 › 08 › stream-a-massive-gzipped-json-file-in-python
Stream a Massive Gzipped Json File in Python - YACB: Yet Another Code Blog
August 15, 2022 - Essentially you need a pipeline. Read a chunk of the raw gzipped data > Decompress it > give it to ijson to parse. This may be my inexperience with Python as I could not find an idiomatic way to do this.
Top answer
1 of 2
6

Using GCS, cloudstorage.open(filename, 'r') will give you a read-only file-like object (earlier created similarly but with 'w':-) which you can use, a chunk at a time, with the standard Python library's zlib module, specifically a zlib.decompressobj, if, of course, the GS object was originally created in the complementary way (with a zlib.compressobj).

Alternatively, for convenience, you can use the standard Python library's gzip module, e.g for the reading phase something like:

compressed_flo = cloudstorage.open('objname', 'r')
uncompressed_flo = gzip.GzipFile(fileobj=compressed_flo,mode='rb')
csvReader = csv.reader(uncompressed_flo)

and vice versa for the earlier writing phase, of course.

Note that when you run locally (with the dev_appserver), the GCS client library uses local disk files to simulate GCS -- in my experience that's good for development purposes, and I can use gsutil or other tools when I need to interact with "real" GCS storage from my local workstation... GCS is for when I need such interaction from my GAE app (and for developing said GAE app locally in the first place:-).

2 of 2
4

So, you have gzipped files stored on GCS. You can process the data stored on GCS in a stream-like fashion. That is, you can download, unzip, and process simultaneously. This avoids

  • to have the unzipped file on disk
  • to have to wait until the download is complete before being able to process the data.

gzip files have a small header and footer, and the body is a compressed stream, consisting of a series of blocks, and each block is decompressable on its own. Python's zlib package helps you with that!

Edit: This is example code for how to decompress and analzye a zlib or gzip stream chunk-wise, purely based on zlib:

import zlib
from collections import Counter


def stream(filename):
    with open(filename, "rb") as f:
        while True:
            chunk = f.read(1024)
            if not chunk:
                break
            yield chunk


def decompress(stream):
    # Generate decompression object. Auto-detect and ignore
    # gzip wrapper, if present.
    z = zlib.decompressobj(32+15)
    for chunk in stream:
        r = z.decompress(chunk)
        if r:
            yield r


c = Counter()
s = stream("data.gz")
for chunk in decompress(s):
    for byte in chunk:
        c[byte] += 1


print c

I tested this code with an example file data.gz, created with GNU gzip.

Quotes from http://www.zlib.net/manual.html:

windowBits can also be greater than 15 for optional gzip decoding. Add 32 to windowBits to enable zlib and gzip decoding with automatic header detection, or add 16 to decode only the gzip format (the zlib format will return a Z_DATA_ERROR). If a gzip stream is being decoded, strm->adler is a crc32 instead of an adler32.

and

Any information contained in the gzip header is not retained [...]

🌐
MicroPython
docs.micropython.org › en › latest › library › gzip.html
gzip – gzip compression & decompression — MicroPython latest documentation
If compression support is enabled then the mode argument can be set to "wb", and writes to the GzipFile instance will be compressed and written to the underlying stream. By default the GzipFile class will read and write data using the gzip file format, including a header and footer with checksum and a window size of 512 bytes.
🌐
Rational Pie
rationalpie.wordpress.com › 2010 › 06 › 02 › python-streaming-gzip-decompression
Python Streaming gzip Decompression | Rational Pie
June 1, 2010 - The gzip module in python works with files. It does not have an interface for decompressing a stream of data. This however becomes a problem if you want to work with files that are difficult (or impossible!) to read in memory at once. A small trick using the zlib library (again in standard python distribution), however, allows one to achieve this. To demonstrate the problem and the solution, take the example of downloading a web page with compression enabled and saving it to a local file:
🌐
Python Forum
python-forum.io › thread-6033.html
Need help to read a gzip stream...
Following code errors, and I have tried every mechanism that is available on google search. Still could not get the stream to uncompress without errors. But the same file can be read in both Java and Scala. orig_file_desc = GzipFile(mode='r', fileob...