To decompress a gzip format file with zlib, call inflateInit2 with the windowBits parameter as 16+MAX_WBITS, like this:
inflateInit2(&stream, 16+MAX_WBITS);
If you don't do this, zlib will complain about a bad stream format. By default, zlib creates streams with a zlib header, and on inflate does not recognise the different gzip header unless you tell it so. Although this is documented starting in version 1.2.1 of the zlib.h header file, it is not in the zlib manual. From the header file:
Answer from Greg Hewgill on Stack Overflow
windowBitscan also be greater than 15 for optional gzip decoding. Add 32 towindowBitsto enable zlib and gzip decoding with automatic header detection, or add 16 to decode only the gzip format (the zlib format will return aZ_DATA_ERROR). If a gzip stream is being decoded,strm->adleris a crc32 instead of an adler32.
To decompress a gzip format file with zlib, call inflateInit2 with the windowBits parameter as 16+MAX_WBITS, like this:
inflateInit2(&stream, 16+MAX_WBITS);
If you don't do this, zlib will complain about a bad stream format. By default, zlib creates streams with a zlib header, and on inflate does not recognise the different gzip header unless you tell it so. Although this is documented starting in version 1.2.1 of the zlib.h header file, it is not in the zlib manual. From the header file:
windowBitscan also be greater than 15 for optional gzip decoding. Add 32 towindowBitsto enable zlib and gzip decoding with automatic header detection, or add 16 to decode only the gzip format (the zlib format will return aZ_DATA_ERROR). If a gzip stream is being decoded,strm->adleris a crc32 instead of an adler32.
python
zlib library supports:
- RFC 1950 (
zlibcompressed format) - RFC 1951 (
deflatecompressed format) - RFC 1952 (
gzipcompressed format)
The python zlib module will support these as well.
choosing windowBits
But zlib can decompress all those formats:
- to (de-)compress
deflateformat, usewbits = -zlib.MAX_WBITS - to (de-)compress
zlibformat, usewbits = zlib.MAX_WBITS - to (de-)compress
gzipformat, usewbits = zlib.MAX_WBITS | 16
See documentation in http://www.zlib.net/manual.html#Advanced (section inflateInit2)
examples
test data:
>>> deflate_compress = zlib.compressobj(9, zlib.DEFLATED, -zlib.MAX_WBITS)
>>> zlib_compress = zlib.compressobj(9, zlib.DEFLATED, zlib.MAX_WBITS)
>>> gzip_compress = zlib.compressobj(9, zlib.DEFLATED, zlib.MAX_WBITS | 16)
>>>
>>> text = '''test'''
>>> deflate_data = deflate_compress.compress(text) + deflate_compress.flush()
>>> zlib_data = zlib_compress.compress(text) + zlib_compress.flush()
>>> gzip_data = gzip_compress.compress(text) + gzip_compress.flush()
>>>
obvious test for zlib:
>>> zlib.decompress(zlib_data)
'test'
test for deflate:
>>> zlib.decompress(deflate_data)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
zlib.error: Error -3 while decompressing data: incorrect header check
>>> zlib.decompress(deflate_data, -zlib.MAX_WBITS)
'test'
test for gzip:
>>> zlib.decompress(gzip_data)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
zlib.error: Error -3 while decompressing data: incorrect header check
>>> zlib.decompress(gzip_data, zlib.MAX_WBITS|16)
'test'
the data is also compatible with gzip module:
>>> import gzip
>>> import StringIO
>>> fio = StringIO.StringIO(gzip_data)
>>> f = gzip.GzipFile(fileobj=fio)
>>> f.read()
'test'
>>> f.close()
automatic header detection (zlib or gzip)
adding 32 to windowBits will trigger header detection
>>> zlib.decompress(gzip_data, zlib.MAX_WBITS|32)
'test'
>>> zlib.decompress(zlib_data, zlib.MAX_WBITS|32)
'test'
using gzip instead
For gzip data with gzip header you can use gzip module directly; but please remember that under the hood, gzip uses zlib.
fh = gzip.open('abc.gz', 'rb')
cdata = fh.read()
fh.close()
It is also possible to decompress it using standard shell-script + gzip, if you don't have, or want to use openssl or other tools.
The trick is to prepend the gzip magic number and compress method to the actual data from zlib.compress:
printf "\x1f\x8b\x08\x00\x00\x00\x00\x00" |cat - /tmp/data |gzip -dc >/tmp/out
Edits:
@d0sboots commented: For RAW Deflate data, you need to add 2 more null bytes:
→ "\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x00"
This Q on SO gives more information about this approach. An answer there suggests that there is also an 8 byte footer.
Users @Vitali-Kushner and @mark-bessey reported success even with truncated files, so a gzip footer does not seem strictly required.
@tobias-kienzler suggested this function for the bashrc:
zlibd() (printf "\x1f\x8b\x08\x00\x00\x00\x00\x00" | cat - "$@" | gzip -dc)
zlib-flate -uncompress < IN_FILE > OUT_FILE
I tried this and it worked for me.
zlib-flate can be found in package qpdf (in Debian Squeeze, Fedora 23, and brew on MacOS according to comments in other answers)
(Thanks to user @tino who provided this as a comment below the OpenSSL answer. Made into propper answer for easy access.)