GeeksforGeeks
geeksforgeeks.org › python › gzip-decompresss-in-python
gzip.decompress(s) in Python - GeeksforGeeks
March 26, 2020 - # import gzip and decompress import gzip s = b'This is GFG author, and final year student.' s = gzip.compress(s) # using gzip.decompress(s) method t = gzip.decompress(s) print(t) Output : b'This is GFG author, and final year student.' Example #2 :
Python
docs.python.org › 3 › library › gzip.html
gzip — Support for gzip files
This function is capable of decompressing multi-member gzip data (multiple gzip blocks concatenated together). When the data is certain to contain only one member the zlib.decompress() function with wbits set to 31 is faster. Added in version 3.2. Changed in version 3.11: Speed is improved by decompressing members at once in memory instead of in a streamed fashion.
Decompression of GZIP Compressed String
1f8b is gzip as already mentioned. Extraction with gzip works fine. You falsely assume that you can push some encoding on binary data. Checking it out in a hex editor it seems the file is just an array of plain ints, mostly ascending (though not always) until there is a cut in the middle of the file and it repeats exactly the same values. from binascii import hexlify, unhexlify import gzip from struct import pack, unpack payload = unhexlify("1F8B080000000000000AEDD2CF4F0F701CC7F1A77C554828BE7DA9A4A25F8AEF37947E51E8DB56072E6CB994A5AD1CFC1807ADAD360EB1E1C08C038772D096696CD5212B0E6C3860B41933077360B30E75505BE679F8FE0D4E1D1EC7CFFBF3FEF10A00A92A51838EAB5BB734AC57FAAE05A52D81B01AD5AE3E8DE8B72271D0AB4FAA590A0F95178021552E83B76A8B873FBA9A00D989BE55E372F8AA332B206125DC5538095EA87995B575311982ABADA5DA35F0519D6BED493752FCC701C675689DBDEAC27A480A42BFCAD2E0B55A4230A3CB1B2063233C567D3A7CD6A90C88CB84DB2ADE04CF74240B7EAA6733A464C30355E5C03B9DC885395DDB02395B61544D79F04DE7F321B100EE2952082F75AC08A675691B848ADD8DEA5CF8944E6E87BFBAB903F2C3F0548723F0435DA590BC130654BE0BDEA87537CCEA4A196496C31345F7C0179DAE804025DC5149153CD7D16AF8A5DE1AEFBC170655BD0FDEABBD16E675BD0E72F7C3989A0E3887CE1D84F87A6B2914F5767AA4098DE8BE8635A90F9A56D00045D5157336A6436D6A55738CB158CCDD62EEF8DFB9FB07D858DEE0F0040000") data = gzip.decompress(payload) values = unpack(str(len(data)//4)+"i",data) v1,v2 = values[:len(values)//2], values[len(values)//2:] assert v1 == v2 # Same data stored twice? More on reddit.com
2
1
July 16, 2023Python decompressing gzip chunk-by-chunk - Stack Overflow
I've a memory- and disk-limited environment where I need to decompress the contents of a gzip file sent to me in string-based chunks (over xmlrpc binary transfer). However, using the zlib.decompres... More on stackoverflow.com
python - How to uncompress gzipped data in a byte array? - Stack Overflow
I have a byte array containing data that is compressed by gzip. Now I need to uncompress this data. How can this be achieved? More on stackoverflow.com
Introducing zune-inflate: The fastest Rust implementation of gzip/Zlib/DEFLATE
Is streaming support planned? Also, is it possible to decompress into a user provided buffer -- even if this buffer has to be initialized? More on reddit.com
30
214
February 13, 2023Michaelehead
michaelehead.com › 2019 › 12 › 28 › python-streaming-unzip.html
Unzipping a large gzip file in Python | Michael Head
December 28, 2019 - To showcase the power of this method for decompressing a large file without exhausting memory or disk space, I put together a repository that demonstrates the various scenarios I tried: https://github.com/headquarters/python-streaming-unzip. Here you can play around with a Docker container that can execute a Python script that attempts to decompress a gzipped file into memory, onto disk, or via the streaming method.
GitHub
github.com › leenr › gzip-stream
GitHub - leenr/gzip-stream: Streaming GZIP compression for Python · GitHub
GZIPCompressedStream does not read entire stream, but instead read it by chunks, until compressed output size will not satisfy read size. AsyncGZIPDecompressedStream class can async read from another source with zlib and gzip decompression on-the-fly · # aiobotocore example import aiobotocore from gzip_stream import AsyncGZIPDecompressedStream AWS_ACCESS_KEY_ID = "KEY_ID" AWS_SECRET_ACCESS_KEY = "ACCESS_KEY" BUCKET = "AWESOME_BUCKET" upload_client = MyAsyncUploadClient() session = aiobotocore.get_session() async with session.create_client( service_name="s3", endpoint_url="s3_endpoint", aws_se
Starred by 32 users
Forked by 5 users
Languages Python
GitHub
gist.github.com › jiankaiwang › f5056a34bbbf595c9553efb36aff64ec
Compress and decompress a gzip file in python. · GitHub
Compress and decompress a gzip file in python. GitHub Gist: instantly share code, notes, and snippets.
Reddit
reddit.com › r/learnpython › decompression of gzip compressed string
r/learnpython on Reddit: Decompression of GZIP Compressed String
July 16, 2023 -
Hi all! In order to get graph data out of our machines, I was looking to get into the data. The data seemed to be in the database, stored in the deprecated IMAGE format. An example string looked like this:
0x1F8B080000000000000AEDD2CF4F0F701CC7F1A77C554828BE7DA9A4A25F8AEF37947E51E8DB56072E6CB994A5AD1CFC1807ADAD360EB1E1C08C038772D096696CD5212B0E6C3860B41933077360B30E75505BE679F8FE0D4E1D1EC7CFFBF3FEF10A00A92A51838EAB5BB734AC57FAAE05A52D81B01AD5AE3E8DE8B72271D0AB4FAA590A0F95178021552E83B76A8B873FBA9A00D989BE55E372F8AA332B206125DC5538095EA87995B575311982ABADA5DA35F0519D6BED493752FCC701C675689DBDEAC27A480A42BFCAD2E0B55A4230A3CB1B2063233C567D3A7CD6A90C88CB84DB2ADE04CF74240B7EAA6733A464C30355E5C03B9DC885395DDB02395B61544D79F04DE7F321B100EE2952082F75AC08A675691B848ADD8DEA5CF8944E6E87BFBAB903F2C3F0548723F0435DA590BC130654BE0BDEA87537CCEA4A196496C31345F7C0179DAE804025DC5149153CD7D16AF8A5DE1AEFBC170655BD0FDEABBD16E675BD0E72F7C3989A0E3887CE1D84F87A6B2914F5767AA4098DE8BE8635A90F9A56D00045D5157336A6436D6A55738CB158CCDD62EEF8DFB9FB07D858DEE0F0040000
By googling (and ChatGPT ofc) it turned out it looks like a GZIP compressed string, given the format start (1F8B080, GZIP format spec RFC 1952). Howerver, I can't figure to get this string decompressed using Python.
Code:
import gzip
import binascii import zlib import bz2 import lzma
The input hexadecimal string
input_hex_string = "0x1F8B080000000000000AEDD2CF4F0F701CC7F1A77C554828BE7DA9A4A25F8AEF37947E51E8DB56072E6CB994A5AD1CFC1807ADAD360EB1E1C08C038772D096696CD5212B0E6C3860B41933077360B30E75505BE679F8FE0D4E1D1EC7CFFBF3FEF10A00A92A51838EAB5BB734AC57FAAE05A52D81B01AD5AE3E8DE8B72271D0AB4FAA590A0F95178021552E83B76A8B873FBA9A00D989BE55E372F8AA332B206125DC5538095EA87995B575311982ABADA5DA35F0519D6BED493752FCC701C675689DBDEAC27A480A42BFCAD2E0B55A4230A3CB1B2063233C567D3A7CD6A90C88CB84DB2ADE04CF74240B7EAA6733A464C30355E5C03B9DC885395DDB02395B61544D79F04DE7F321B100EE2952082F75AC08A675691B848ADD8DEA5CF8944E6E87BFBAB903F2C3F0548723F0435DA590BC130654BE0BDEA87537CCEA4A196496C31345F7C0179DAE804025DC5149153CD7D16AF8A5DE1AEFBC170655BD0FDEABBD16E675BD0E72F7C3989A0E3887CE1D84F87A6B2914F5767AA4098DE8BE8635A90F9A56D00045D5157336A6436D6A55738CB158CCDD62EEF8DFB9FB07D858DEE0F0040000"
Remove the "0x" prefix and convert the hexadecimal string to bytes
compressed_data = binascii.unhexlify(input_hex_string[2:])
List of character encodings to try
encodings_to_try = ["utf-8", "latin-1", "ascii", "utf-16", "utf-32", "cp1252", "cp850"]
List of decompression methods to try
decompression_methods = [ ("gzip", gzip.decompress), ("zlib", zlib.decompress), ("bz2", bz2.decompress), ("lzma", lzma.decompress), ]
for decompression_name, decompression_func in decompression_methods: print(f"Trying decompression with {decompression_name}:") for encoding in encodings_to_try: try: decompressed_data = decompression_func(compressed_data) print(f"Decompressed and decoded with {encoding}:") print(decompressed_data.decode(encoding)) except Exception as e: print(f"Decompression with {decompression_name} and {encoding} failed: {e}") print()
Does someone of you have experience on this type of compression/ can someone get useful data out of this string? Thanks in advance
ProgramCreek
programcreek.com › python › example › 93867 › gzip.decompress
Python Examples of gzip.decompress
def __iter__(self): keys = self._get_keys() if self.shuffle: random.shuffle(keys) for key in keys: chunk = json.loads(gzip.decompress(self.db[key]).decode('utf-8')) # discard long examples trunc_chunk = [] lens = [] for feat in chunk: if feat['input_len'] > self.max_len: continue trunc_chunk.append(feat) lens.append(feat['input_len']) dataset = GPT2FeatureDataset(trunc_chunk, self.max_len) sampler = BucketSampler(lens, self.bucket_size, self.batch_size, droplast=True, shuffle=self.shuffle) loader = DataLoader(dataset, batch_sampler=sampler, num_workers=0, # can test multi-worker collate_fn=GPT2FeatureDataset.collate) yield from loader
PyPI
pypi.org › project › gzip-stream
gzip-stream · PyPI
# aiobotocore example import aiobotocore from gzip_stream import AsyncGZIPDecompressedStream AWS_ACCESS_KEY_ID = "KEY_ID" AWS_SECRET_ACCESS_KEY = "ACCESS_KEY" BUCKET = "AWESOME_BUCKET" upload_client = MyAsyncUploadClient() session = aiobotocore.get_session() async with session.create_client( service_name="s3", endpoint_url="s3_endpoint", aws_secret_access_key=AWS_SECRET_ACCESS_KEY, aws_access_key_id=AWS_ACCESS_KEY_ID, ) as client: response = await client.get_object(Bucket=BUCKET, Key='my_very_big_1tb_file.txt.gz') async for decompressed_chunk in GzipAsyncReaderWrapper(response["Body"])): await upload_client.upload_fileobj(decompressed_chunk) Module works on Python ~= 3.5. ... Public Domain: CC0 1.0 Universal. ... Download the file for your platform. If you're not sure which to choose, learn more about installing packages. ... Details for the file gzip-stream-1.2.0.tar.gz.
» pip install gzip-stream
MicroPython
docs.micropython.org › en › latest › library › gzip.html
gzip – gzip compression & decompression — MicroPython latest documentation
It is itself a stream and implements the standard read/readinto/write/close methods. When the mode argument is "rb", reads from the GzipFile instance will decompress the data in the underlying stream and return decompressed data.
Rational Pie
rationalpie.wordpress.com › 2010 › 06 › 02 › python-streaming-gzip-decompression
Python Streaming gzip Decompression | Rational Pie
June 1, 2010 - The gzip module in python works with files. It does not have an interface for decompressing a stream of data. This however becomes a problem if you want to work with files that are difficult (or impossible!) to read in memory at once. A small trick using the zlib library (again in standard python distribution), however, allows one to achieve this. To demonstrate the problem and the solution, take the example of downloading a web page with compression enabled and saving it to a local file:
CodersLegacy
coderslegacy.com › home › python › python gzip module – compress and decompress files
Python gzip module - compress and decompress files - CodersLegacy
January 30, 2022 - The solution to this is compressing files with the help of a compression library to produce smaller file sizes. In Python, we can make use of the gzip library, used to compress and decompress files. Serialization is the process of converting an object to a byte stream, and the inverse of which ...
Linux Hint
linuxhint.com › python-gzip-decompress-function
Python Gzip Decompress Function – Linux Hint
It is an essential function for any Python developer to work with compressed data files. This function takes a compressed data string as an argument and returns the original uncompressed data. This Python post provided a thorough guide on the “gzip.decompress()” function using numerous examples.
Python Module of the Week
pymotw.com › 2 › gzip
gzip – Read and write GNU zip files - Python Module of the Week
When re-reading the previously ... may want to prefix the data with an integer representing the actual amount of data to be read. $ python gzip_StringIO.py UNCOMPRESSED: 300 The same line, over and over....
Top answer 1 of 2
58
gzip and zlib use slightly different headers.
See How can I decompress a gzip stream with zlib?
Try d = zlib.decompressobj(16+zlib.MAX_WBITS).
And you might try changing your chunk size to a power of 2 (say CHUNKSIZE=1024) for possible performance reasons.
2 of 2
4
I've got a more detailed answer here: https://stackoverflow.com/a/22310760/1733117
d = zlib.decompressobj(zlib.MAX_WBITS|32)
per documentation this automatically detects the header (zlib or gzip).
Scaler
scaler.com › home › topics › python gzip module
Python gzip Module| Scaler Topics
December 15, 2022 - Intro: Used to decompress the data returns the bytes of the decompressed data. This Python gzip.decompress() function is capable of decompressing the multi-member gzip data (that is, multiple gzip blocks concatenated together).
Quora
quora.com › Python-How-can-I-decompress-a-string-compressed-by-GZipStream-in-NET
Python: How can I decompress a string compressed by GZipStream in .NET? - Quora
Quora is a place to gain and share knowledge. It's a platform to ask questions and connect with people who contribute unique insights and quality answers.
Delft Stack
delftstack.com › home › howto › python › python gzip decompress
How to Decompress Gzip in Python | Delft Stack
February 2, 2024 - The syntax of the gzip decompression method is shown below. ... Now, let’s use this function in our examples. First, we need to install the library using the following command. ... Once the Gzip library is installed, we can import it using the following line of code. ... Let’s start with an example, as shown below. # python import gzip value = b"This string is encoded in Python." value = gzip.compress(value) print("compressed value is: ", value) compressed = gzip.decompress(value) print("Decompressed value is: ", compressed)
Openmv
docs.openmv.io › micropython › micropython libraries › gzip – gzip compression & decompression
gzip – gzip compression & decompression - MicroPython 1.28 documentation
A copy of this module can be installed ... will only be available if compression support is enabled in the built-in deflate module. gzip.open(filename: str, mode: str, /) → GzipFile¶...
GitHub
gist.github.com › davidcrawford › 6774493
Python UnzipStream — A file-like wrapper around a gzipped stream that uncompresses as it reads.
February 1, 2018 - Python UnzipStream — A file-like wrapper around a gzipped stream that uncompresses as it reads. - unzip.py
Top answer 1 of 2
43
zlib.decompress(data, 15 + 32) should autodetect whether you have gzip data or zlib data.
zlib.decompress(data, 15 + 16) should work if gzip and barf if zlib.
Here it is with Python 2.7.1, creating a little gz file, reading it back, and decompressing it:
>>> import gzip, zlib
>>> f = gzip.open('foo.gz', 'wb')
>>> f.write(b"hello world")
11
>>> f.close()
>>> c = open('foo.gz', 'rb').read()
>>> c
'\x1f\x8b\x08\x08\x14\xf4\xdcM\x02\xfffoo\x00\xcbH\xcd\xc9\xc9W(\xcf/\xcaI\x01\x00\x85\x11J\r\x0b\x00\x00\x00'
>>> ba = bytearray(c)
>>> ba
bytearray(b'\x1f\x8b\x08\x08\x14\xf4\xdcM\x02\xfffoo\x00\xcbH\xcd\xc9\xc9W(\xcf/\xcaI\x01\x00\x85\x11J\r\x0b\x00\x00\x00')
>>> zlib.decompress(ba, 15+32)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: must be string or read-only buffer, not bytearray
>>> zlib.decompress(bytes(ba), 15+32)
'hello world'
>>>
Python 3.x usage would be very similar.
Update based on comment that you are running Python 2.2.1.
Sigh. That's not even the last release of Python 2.2. Anyway, continuing with the foo.gz file created as above:
Python 2.2.3 (#42, May 30 2003, 18:12:08) [MSC 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> strobj = open('foo.gz', 'rb').read()
>>> strobj
'\x1f\x8b\x08\x08\x14\xf4\xdcM\x02\xfffoo\x00\xcbH\xcd\xc9\xc9W(\xcf/\xcaI\x01\x00\x85\x11J\r\x0b\x00\x00\x00'
>>> import zlib
>>> zlib.decompress(strobj, 15+32)
Traceback (most recent call last):
File "<stdin>", line 1, in ?
zlib.error: Error -2 while preparing to decompress data
>>> zlib.decompress(strobj, 15+16)
Traceback (most recent call last):
File "<stdin>", line 1, in ?
zlib.error: Error -2 while preparing to decompress data
# OK, we can't use the back door method. Plan B: use the
# documented approach i.e. gzip.GzipFile with a file-like object.
>>> import gzip, cStringIO
>>> fileobj = cStringIO.StringIO(strobj)
>>> gzf = gzip.GzipFile('dummy-name', 'rb', 9, fileobj)
>>> gzf.read()
'hello world'
# Success. Now let's assume you have an array.array object-- which requires
# premeditation; they aren't created accidentally!
# The following code assumes subtype 'B' but should work for any subtype.
>>> import array, sys
>>> aaB = array.array('B')
>>> aaB.fromfile(open('foo.gz', 'rb'), sys.maxint)
Traceback (most recent call last):
File "<stdin>", line 1, in ?
EOFError: not enough items in file
#### Don't panic, just read the fine manual
>>> aaB
array('B', [31, 139, 8, 8, 20, 244, 220, 77, 2, 255, 102, 111, 111, 0, 203, 72, 205, 201, 201, 87, 40, 207, 47, 202, 73, 1, 0, 133, 17, 74, 13, 11, 0, 0, 0])
>>> strobj2 = aaB.tostring()
>>> strobj2 == strobj
1 #### means True
# You can make a str object and use that as above.
# ... or you can plug it directly into StringIO:
>>> gzip.GzipFile('dummy-name', 'rb', 9, cStringIO.StringIO(aaB)).read()
'hello world'
2 of 2
6
Apparently you can do this
import zlib
# ...
ungziped_str = zlib.decompressobj().decompress('x\x9c' + gziped_str)
Or this:
zlib.decompress( data ) # equivalent to gzdecompress()
For more info, look here: Python docs