gzip and zlib use slightly different headers.
See How can I decompress a gzip stream with zlib?
Try d = zlib.decompressobj(16+zlib.MAX_WBITS).
And you might try changing your chunk size to a power of 2 (say CHUNKSIZE=1024) for possible performance reasons.
gzip and zlib use slightly different headers.
See How can I decompress a gzip stream with zlib?
Try d = zlib.decompressobj(16+zlib.MAX_WBITS).
And you might try changing your chunk size to a power of 2 (say CHUNKSIZE=1024) for possible performance reasons.
I've got a more detailed answer here: https://stackoverflow.com/a/22310760/1733117
d = zlib.decompressobj(zlib.MAX_WBITS|32)
per documentation this automatically detects the header (zlib or gzip).
Faster decompression of gzip files
Decompression of GZIP Compressed String
python - How to uncompress gzipped data in a byte array? - Stack Overflow
Introducing zune-inflate: The fastest Rust implementation of gzip/Zlib/DEFLATE
» pip install gzip-stream
Hi all! In order to get graph data out of our machines, I was looking to get into the data. The data seemed to be in the database, stored in the deprecated IMAGE format. An example string looked like this:
0x1F8B080000000000000AEDD2CF4F0F701CC7F1A77C554828BE7DA9A4A25F8AEF37947E51E8DB56072E6CB994A5AD1CFC1807ADAD360EB1E1C08C038772D096696CD5212B0E6C3860B41933077360B30E75505BE679F8FE0D4E1D1EC7CFFBF3FEF10A00A92A51838EAB5BB734AC57FAAE05A52D81B01AD5AE3E8DE8B72271D0AB4FAA590A0F95178021552E83B76A8B873FBA9A00D989BE55E372F8AA332B206125DC5538095EA87995B575311982ABADA5DA35F0519D6BED493752FCC701C675689DBDEAC27A480A42BFCAD2E0B55A4230A3CB1B2063233C567D3A7CD6A90C88CB84DB2ADE04CF74240B7EAA6733A464C30355E5C03B9DC885395DDB02395B61544D79F04DE7F321B100EE2952082F75AC08A675691B848ADD8DEA5CF8944E6E87BFBAB903F2C3F0548723F0435DA590BC130654BE0BDEA87537CCEA4A196496C31345F7C0179DAE804025DC5149153CD7D16AF8A5DE1AEFBC170655BD0FDEABBD16E675BD0E72F7C3989A0E3887CE1D84F87A6B2914F5767AA4098DE8BE8635A90F9A56D00045D5157336A6436D6A55738CB158CCDD62EEF8DFB9FB07D858DEE0F0040000
By googling (and ChatGPT ofc) it turned out it looks like a GZIP compressed string, given the format start (1F8B080, GZIP format spec RFC 1952). Howerver, I can't figure to get this string decompressed using Python.
Code:
import gzip
import binascii import zlib import bz2 import lzma
The input hexadecimal string
input_hex_string = "0x1F8B080000000000000AEDD2CF4F0F701CC7F1A77C554828BE7DA9A4A25F8AEF37947E51E8DB56072E6CB994A5AD1CFC1807ADAD360EB1E1C08C038772D096696CD5212B0E6C3860B41933077360B30E75505BE679F8FE0D4E1D1EC7CFFBF3FEF10A00A92A51838EAB5BB734AC57FAAE05A52D81B01AD5AE3E8DE8B72271D0AB4FAA590A0F95178021552E83B76A8B873FBA9A00D989BE55E372F8AA332B206125DC5538095EA87995B575311982ABADA5DA35F0519D6BED493752FCC701C675689DBDEAC27A480A42BFCAD2E0B55A4230A3CB1B2063233C567D3A7CD6A90C88CB84DB2ADE04CF74240B7EAA6733A464C30355E5C03B9DC885395DDB02395B61544D79F04DE7F321B100EE2952082F75AC08A675691B848ADD8DEA5CF8944E6E87BFBAB903F2C3F0548723F0435DA590BC130654BE0BDEA87537CCEA4A196496C31345F7C0179DAE804025DC5149153CD7D16AF8A5DE1AEFBC170655BD0FDEABBD16E675BD0E72F7C3989A0E3887CE1D84F87A6B2914F5767AA4098DE8BE8635A90F9A56D00045D5157336A6436D6A55738CB158CCDD62EEF8DFB9FB07D858DEE0F0040000"
Remove the "0x" prefix and convert the hexadecimal string to bytes
compressed_data = binascii.unhexlify(input_hex_string[2:])
List of character encodings to try
encodings_to_try = ["utf-8", "latin-1", "ascii", "utf-16", "utf-32", "cp1252", "cp850"]
List of decompression methods to try
decompression_methods = [ ("gzip", gzip.decompress), ("zlib", zlib.decompress), ("bz2", bz2.decompress), ("lzma", lzma.decompress), ]
for decompression_name, decompression_func in decompression_methods: print(f"Trying decompression with {decompression_name}:") for encoding in encodings_to_try: try: decompressed_data = decompression_func(compressed_data) print(f"Decompressed and decoded with {encoding}:") print(decompressed_data.decode(encoding)) except Exception as e: print(f"Decompression with {decompression_name} and {encoding} failed: {e}") print()
Does someone of you have experience on this type of compression/ can someone get useful data out of this string? Thanks in advance
zlib.decompress(data, 15 + 32) should autodetect whether you have gzip data or zlib data.
zlib.decompress(data, 15 + 16) should work if gzip and barf if zlib.
Here it is with Python 2.7.1, creating a little gz file, reading it back, and decompressing it:
>>> import gzip, zlib
>>> f = gzip.open('foo.gz', 'wb')
>>> f.write(b"hello world")
11
>>> f.close()
>>> c = open('foo.gz', 'rb').read()
>>> c
'\x1f\x8b\x08\x08\x14\xf4\xdcM\x02\xfffoo\x00\xcbH\xcd\xc9\xc9W(\xcf/\xcaI\x01\x00\x85\x11J\r\x0b\x00\x00\x00'
>>> ba = bytearray(c)
>>> ba
bytearray(b'\x1f\x8b\x08\x08\x14\xf4\xdcM\x02\xfffoo\x00\xcbH\xcd\xc9\xc9W(\xcf/\xcaI\x01\x00\x85\x11J\r\x0b\x00\x00\x00')
>>> zlib.decompress(ba, 15+32)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: must be string or read-only buffer, not bytearray
>>> zlib.decompress(bytes(ba), 15+32)
'hello world'
>>>
Python 3.x usage would be very similar.
Update based on comment that you are running Python 2.2.1.
Sigh. That's not even the last release of Python 2.2. Anyway, continuing with the foo.gz file created as above:
Python 2.2.3 (#42, May 30 2003, 18:12:08) [MSC 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> strobj = open('foo.gz', 'rb').read()
>>> strobj
'\x1f\x8b\x08\x08\x14\xf4\xdcM\x02\xfffoo\x00\xcbH\xcd\xc9\xc9W(\xcf/\xcaI\x01\x00\x85\x11J\r\x0b\x00\x00\x00'
>>> import zlib
>>> zlib.decompress(strobj, 15+32)
Traceback (most recent call last):
File "<stdin>", line 1, in ?
zlib.error: Error -2 while preparing to decompress data
>>> zlib.decompress(strobj, 15+16)
Traceback (most recent call last):
File "<stdin>", line 1, in ?
zlib.error: Error -2 while preparing to decompress data
# OK, we can't use the back door method. Plan B: use the
# documented approach i.e. gzip.GzipFile with a file-like object.
>>> import gzip, cStringIO
>>> fileobj = cStringIO.StringIO(strobj)
>>> gzf = gzip.GzipFile('dummy-name', 'rb', 9, fileobj)
>>> gzf.read()
'hello world'
# Success. Now let's assume you have an array.array object-- which requires
# premeditation; they aren't created accidentally!
# The following code assumes subtype 'B' but should work for any subtype.
>>> import array, sys
>>> aaB = array.array('B')
>>> aaB.fromfile(open('foo.gz', 'rb'), sys.maxint)
Traceback (most recent call last):
File "<stdin>", line 1, in ?
EOFError: not enough items in file
#### Don't panic, just read the fine manual
>>> aaB
array('B', [31, 139, 8, 8, 20, 244, 220, 77, 2, 255, 102, 111, 111, 0, 203, 72, 205, 201, 201, 87, 40, 207, 47, 202, 73, 1, 0, 133, 17, 74, 13, 11, 0, 0, 0])
>>> strobj2 = aaB.tostring()
>>> strobj2 == strobj
1 #### means True
# You can make a str object and use that as above.
# ... or you can plug it directly into StringIO:
>>> gzip.GzipFile('dummy-name', 'rb', 9, cStringIO.StringIO(aaB)).read()
'hello world'
Apparently you can do this
import zlib
# ...
ungziped_str = zlib.decompressobj().decompress('x\x9c' + gziped_str)
Or this:
zlib.decompress( data ) # equivalent to gzdecompress()
For more info, look here: Python docs