It's quite kludgy (self referencing, etc; just put a few minutes writing it, nothing really elegant), but it does what you want if you're still interested in using gzip instead of zlib directly.

Basically, GzipWrap is a (very limited) file-like object that produces a gzipped file out of a given iterable (e.g., a file-like object, a list of strings, any generator...)

Of course, it produces binary so there was no sense in implementing "readline".

You should be able to expand it to cover other cases or to be used as an iterable object itself.

from gzip import GzipFile

class GzipWrap(object):
    # input is a filelike object that feeds the input
    def __init__(self, input, filename = None):
        self.input = input
        self.buffer = ''
        self.zipper = GzipFile(filename, mode = 'wb', fileobj = self)

    def read(self, size=-1):
        if (size < 0) or len(self.buffer) < size:
            for s in self.input:
                self.zipper.write(s)
                if size > 0 and len(self.buffer) >= size:
                    self.zipper.flush()
                    break
            else:
                self.zipper.close()
            if size < 0:
                ret = self.buffer
                self.buffer = ''
        else:
            ret, self.buffer = self.buffer[:size], self.buffer[size:]
        return ret

    def flush(self):
        pass

    def write(self, data):
        self.buffer += data

    def close(self):
        self.input.close()
Answer from Ricardo Cárdenes on Stack Overflow
🌐
GitHub
github.com › leenr › gzip-stream
GitHub - leenr/gzip-stream: Streaming GZIP compression for Python · GitHub
Streaming GZIP compression for Python. Contribute to leenr/gzip-stream development by creating an account on GitHub.
Starred by 32 users
Forked by 5 users
Languages   Python
🌐
PyPI
pypi.org › project › gzip-stream
gzip-stream · PyPI
from gzip_stream import GZIPCompressedStream from my_upload_lib import MyUploadClient upload_client = MyUploadClient() with open('my_very_big_1tb_file.txt') as file_to_upload: compressed_stream = GZIPCompressedStream( file_to_upload, compression_level=7 ) upload_client.upload_fileobj(compressed_stream)
      » pip install gzip-stream
    
Published   Dec 08, 2021
Version   1.2.0
🌐
Python
docs.python.org › 3 › library › gzip.html
gzip — Support for gzip files
The optional mtime argument is the timestamp requested by gzip. The time is in Unix format, i.e., seconds since 00:00:00 UTC, January 1, 1970. If mtime is omitted or None, the current time is used. Use mtime = 0 to generate a compressed stream that does not depend on creation time.
Top answer
1 of 6
15

It's quite kludgy (self referencing, etc; just put a few minutes writing it, nothing really elegant), but it does what you want if you're still interested in using gzip instead of zlib directly.

Basically, GzipWrap is a (very limited) file-like object that produces a gzipped file out of a given iterable (e.g., a file-like object, a list of strings, any generator...)

Of course, it produces binary so there was no sense in implementing "readline".

You should be able to expand it to cover other cases or to be used as an iterable object itself.

from gzip import GzipFile

class GzipWrap(object):
    # input is a filelike object that feeds the input
    def __init__(self, input, filename = None):
        self.input = input
        self.buffer = ''
        self.zipper = GzipFile(filename, mode = 'wb', fileobj = self)

    def read(self, size=-1):
        if (size < 0) or len(self.buffer) < size:
            for s in self.input:
                self.zipper.write(s)
                if size > 0 and len(self.buffer) >= size:
                    self.zipper.flush()
                    break
            else:
                self.zipper.close()
            if size < 0:
                ret = self.buffer
                self.buffer = ''
        else:
            ret, self.buffer = self.buffer[:size], self.buffer[size:]
        return ret

    def flush(self):
        pass

    def write(self, data):
        self.buffer += data

    def close(self):
        self.input.close()
2 of 6
11

Here is a cleaner, non-self-referencing version based on Ricardo Cárdenes' very helpful answer.

from gzip import GzipFile
from collections import deque


CHUNK = 16 * 1024


class Buffer (object):
    def __init__ (self):
        self.__buf = deque()
        self.__size = 0
    def __len__ (self):
        return self.__size
    def write (self, data):
        self.__buf.append(data)
        self.__size += len(data)
    def read (self, size=-1):
        if size < 0: size = self.__size
        ret_list = []
        while size > 0 and len(self.__buf):
            s = self.__buf.popleft()
            size -= len(s)
            ret_list.append(s)
        if size < 0:
            ret_list[-1], remainder = ret_list[-1][:size], ret_list[-1][size:]
            self.__buf.appendleft(remainder)
        ret = ''.join(ret_list)
        self.__size -= len(ret)
        return ret
    def flush (self):
        pass
    def close (self):
        pass


class GzipCompressReadStream (object):
    def __init__ (self, fileobj):
        self.__input = fileobj
        self.__buf = Buffer()
        self.__gzip = GzipFile(None, mode='wb', fileobj=self.__buf)
    def read (self, size=-1):
        while size < 0 or len(self.__buf) < size:
            s = self.__input.read(CHUNK)
            if not s:
                self.__gzip.close()
                break
            self.__gzip.write(s)
        return self.__buf.read(size)

Advantages:

  • Avoids repeated string concatenation, which would cause the entire string to be copied repeatedly.
  • Reads a fixed CHUNK size from the input stream, instead of reading whole lines at a time (which can be arbitrarily long).
  • Avoids circular references.
  • Avoids misleading public "write" method of GzipCompressStream(), which is really only used internally.
  • Takes advantage of name mangling for internal member variables.
🌐
GitHub
github.com › commoncrawl › gzipstream
GitHub - commoncrawl/gzipstream: gzipstream allows Python to process multi-part gzip files from a streaming source
gzipstream allows Python to process multi-part gzip files from a streaming source - commoncrawl/gzipstream
Starred by 23 users
Forked by 12 users
Languages   Python 100.0% | Python 100.0%
🌐
Python Module of the Week
pymotw.com › 2 › gzip
gzip – Read and write GNU zip files - Python Module of the Week
Now available for Python 3! Buy the book! ... The gzip module provides a file-like interface to GNU zip files, using zlib to compress and uncompress the data.
🌐
GitHub
github.com › Smerity › gzipstream
GitHub - Smerity/gzipstream: gzipstream allows Python to process multi-part gzip files from a streaming source · GitHub
gzipstream allows Python to process multi-part gzip files from a streaming source - Smerity/gzipstream
Starred by 17 users
Forked by 19 users
Languages   Python
🌐
MicroPython
docs.micropython.org › en › latest › library › gzip.html
gzip – gzip compression & decompression — MicroPython latest documentation
This module allows compression and decompression of binary data with the DEFLATE algorithm used by the gzip file format. ... Prefer to use deflate.DeflateIO instead of the functions in this module as it provides a streaming interface to compression and decompression which is convenient and ...
🌐
Michaelehead
michaelehead.com › 2019 › 12 › 28 › python-streaming-unzip.html
Unzipping a large gzip file in Python | Michael Head
December 28, 2019 - To showcase the power of this method for decompressing a large file without exhausting memory or disk space, I put together a repository that demonstrates the various scenarios I tried: https://github.com/headquarters/python-streaming-unzip. Here you can play around with a Docker container that can execute a Python script that attempts to decompress a gzipped file into memory, onto disk, or via the streaming method.
Find elsewhere
🌐
CodersLegacy
coderslegacy.com › home › python › python gzip module – compress and decompress files
Python gzip module - compress and decompress files - CodersLegacy
January 30, 2022 - This method is used for creating files and storing compressed data in them. If you open the file stream with the gzip.GzipFile() method, any data you write to it will be automatically compressed.
🌐
GitHub
gist.github.com › beaufour › 4205533
Python: Streaming Gzip reader · GitHub
June 8, 2017 - Python: Streaming Gzip reader. GitHub Gist: instantly share code, notes, and snippets.
🌐
Spacewalkproject
spacewalkproject.github.io › documentation › python-gzipstream › gzipstream.GzipStream-class.html
gzipstream.GzipStream
This doesn't allow for streaming gzipped data to be processed easily (e.g. can't seek a socket). Using the _StreamBuf class enables streaming gzipped data to be processed by buffering that data at it passes through. For Python versions 1.5.2 & 2.1.*: Normal data version.
🌐
Rational Pie
rationalpie.wordpress.com › 2010 › 06 › 02 › python-streaming-gzip-decompression
Python Streaming gzip Decompression | Rational Pie
June 1, 2010 - The gzip module in python works with files. It does not have an interface for decompressing a stream of data. This however becomes a problem if you want to work with files that are difficult (or impossible!) to read in memory at once.
🌐
Medium
eraliod.medium.com › python-streaming-vs-batched-compression-test-4e94b51a4879
Python Streaming vs Batched Compression Test | by Damian Eralio | Medium
January 9, 2024 - To make the code portable for anyone ... the itertools library. The function creates a compressed csv file (gzip compression) on disk, and appends small amounts of data into it until the source is depleted....
🌐
GitHub
gist.github.com › davidcrawford › 6774493
Python UnzipStream — A file-like wrapper around a gzipped stream that uncompresses as it reads.
February 1, 2018 - Python UnzipStream — A file-like wrapper around a gzipped stream that uncompresses as it reads. - unzip.py