🌐
Hacker News
news.ycombinator.com › item
It's not just Python that has this problem. In Debian the zlib package is called... | Hacker News
October 3, 2020 - I guess it's too late to fix this apparent typo? The project is called zlib, ''libz'' is simply incorrect · librust-libz-sys-dev is a Debian package for a Rust library that's literally called "libz-sys" [1]. Why the developers of this package decided to call it this and not "zlib" is a question ...
🌐
Oracle
docs.oracle.com › cd › E88353_01 › html › E37842 › libz-3.html
libz - man pages section 3: Library Interfaces and Headers
July 27, 2022 - LIBZ(3) Library Functions Manual LIBZ(3) NAME zlib - compression/decompression library SYNOPSIS [see zlib.h for full description] DESCRIPTION The libz library is a general purpose data compression library. The code is thread safe, assuming that the standard library functions used are thread ...
Discussions

linker - How does gcc/ld find zlib.so? - Stack Overflow
How does ld know to add the lib after the z for zlib? I've never seen this happen to another library. ... It doesn't. It looks for libz.so, which is usually in /lib or /lib64, but might in some distributions actually be in /usr/lib[64]... If you have a zlib.so that applications are using, the ... More on stackoverflow.com
🌐 stackoverflow.com
compression - How are zlib, gzip and zip related? What do they have in common and how are they different? - Stack Overflow
The compression algorithm used in zlib is essentially the same as that in gzip and zip. What are gzip and zip? How are they different and how are they same? Mod note: This post is off-topic, but t... More on stackoverflow.com
🌐 stackoverflow.com
Analyzing Python Compression Libraries: zlib, LZ4, Brotli, and Zstandard
You need to compare levels to make this useful. Brotli defaults to the maximum level (11) which is why its taking so long -- if you turn it down to 6 you'll get a ~1.17s compression time with a ~57% ratio. Zstandard defaults to 3 on a scale of 22. If you turn it up to 22 you'll get a >100 second compression time and a ~60% ratio. If you go 6 for Brotli and 11 for Zstandard you'll get roughly the same speed (for compression) and ratio. Point is, level matters a lot. There's other factors like streaming performance and window size that you need to account for but bare minimum you need to do like-to-like comparisons on level. More on reddit.com
🌐 r/Python
15
23
April 30, 2024
PSA: libz-sys, flate2, and git2 now support the high-performance zlib-ng - Rust Internals
libz-sys provides Rust bindings to the zlib compression library, for handling DEFLATE/gzip compression. I've just released libz-sys 1.1.0, which provides opt-in support for the high-performance zlib-ng library in zlib-compat mode. zlib-ng provides 2-3x performance improvements for compression, ... More on internals.rust-lang.org
🌐 internals.rust-lang.org
15
August 19, 2020
🌐
GitLab
gitlab.com › sortix › libz › repository
sortix / Libz · GitLab
libz is a general purpose data compression library. The data format used by libz is described by RFC 1950 (zlib format), RFC 1951 (deflate format), and RFC 1952 (gzip format). This libz has been cleaned up: * Support for anything but modern ...
🌐
Sortix
sortix.org › libz
Sortix libz
This is a fork of zlib. libz is a general purpose data compression library.
lossless compression library
zlib (/ˈziːlɪb/ or "zeta-lib", /ˈziːtəˌlɪb/) is a data format and a lossless data compression software library created by Jean-Loup Gailly and Mark Adler. The library implements the Deflate algorithm and supports compressing … Wikipedia
Factsheet
zlib (library)
Release 1 May 1995 (1995-05-01)
Factsheet
zlib (library)
Release 1 May 1995 (1995-05-01)
🌐
Wikipedia
en.wikipedia.org › wiki › Zlib
zlib - Wikipedia
1 week ago - Today, zlib is something of a de facto standard, to the point that zlib and DEFLATE are often used interchangeably in standards documents, with thousands of applications relying on it for compression, either directly or indirectly.
🌐
Unix.com
unix.com › man_page › sunos › 3 › libz
sunos libz man page on unix.com
DESCRIPTION The libz (zlib) library is a general purpose data compression library. The code is thread safe. It provides in-memory compression and decompression functions, including integrity checks of the uncompressed data.
Top answer
1 of 1
5

gcc has several different methods for linking libraries, shared or static. If you specify -lz, gcc is going to look for libz.so (possibly with some version bits between the libz and the .so, but the important part is the file name will start with libz and end with .so), or for libz.a (again, possibly with version info) if you are compiling statically, or as a fallback if the shared library does not exist. If you specify -lzlib it will look for libzlib.so (which is not the standard name - the package is often named zlib, but the library itself is libz). Another way of linking would be to not use the -l<lib> option, and just specify /path/to/zlib.so or -L /path/to zlib.so (or zlib.a if you want). In this case, the library doesn't have to have the lib prefix, but you would have to explicitly provide any version info, unless provisions are made for a symbolic link or something similar to provide the literal name zlib.so.

Applications can also load shared libraries at runtime via dlopen() and it's other associated functions, in which case the library can also be named whatever you want it to be (this doesn't work for static libraries, of course).

So, if the library you are looking at is actually called zlib.so, then it is not being found by gcc ... -lz, unless it just happens to be a symbolic link to libz.so (or vice versa, in which case gcc is really just using libz.so, which happens to have the same content as your zlib.so). However gcc might be using it if the build process explicitly names the library in the link stage (not using -l<lib>) or if your application loads it via dlopen() (but in that case, it's not really linked to your program - it's just loaded at run time).

Top answer
1 of 3
3283

Short form:

.zip is an archive format using, usually, the Deflate compression method. The .gz gzip format is for single files, also using the Deflate compression method. Often gzip is used in combination with tar to make a compressed archive format, .tar.gz. The zlib library provides Deflate compression and decompression code for use by zip, gzip, png (which uses the zlib wrapper on deflate data), and many other applications.

Long form:

The ZIP format was developed by Phil Katz as an open format with an open specification, where his implementation, PKZIP, was shareware. It is an archive format that stores files and their directory structure, where each file is individually compressed. The file type is .zip. The files, as well as the directory structure, can optionally be encrypted.

The ZIP format supports several compression methods:

    0 - The file is stored (no compression)
    1 - The file is Shrunk
    2 - The file is Reduced with compression factor 1
    3 - The file is Reduced with compression factor 2
    4 - The file is Reduced with compression factor 3
    5 - The file is Reduced with compression factor 4
    6 - The file is Imploded
    7 - Reserved for Tokenizing compression algorithm
    8 - The file is Deflated
    9 - Enhanced Deflating using Deflate64(tm)
   10 - PKWARE Data Compression Library Imploding (old IBM TERSE)
   11 - Reserved by PKWARE
   12 - File is compressed using BZIP2 algorithm
   13 - Reserved by PKWARE
   14 - LZMA
   15 - Reserved by PKWARE
   16 - IBM z/OS CMPSC Compression
   17 - Reserved by PKWARE
   18 - File is compressed using IBM TERSE (new)
   19 - IBM LZ77 z Architecture 
   20 - deprecated (use method 93 for zstd)
   93 - Zstandard (zstd) Compression 
   94 - MP3 Compression 
   95 - XZ Compression 
   96 - JPEG variant
   97 - WavPack compressed data
   98 - PPMd version I, Rev 1
   99 - AE-x encryption marker (see APPENDIX E)

Methods 1 to 7 are historical and are not in use. Methods 9 through 98 are relatively recent additions and are in varying, small amounts of use. The only method in truly widespread use in the ZIP format is method 8, Deflate, and to some smaller extent method 0, which is no compression at all. Virtually every .zip file that you will come across in the wild will use exclusively methods 8 and 0, likely just method 8. (Method 8 also has a means to effectively store the data with no compression and relatively little expansion, and Method 0 cannot be streamed whereas Method 8 can be.)

The ISO/IEC 21320-1:2015 standard for file containers is a restricted zip format, such as used in Java archive files (.jar), Office Open XML files (Microsoft Office .docx, .xlsx, .pptx), Office Document Format files (.odt, .ods, .odp), and EPUB files (.epub). That standard limits the compression methods to 0 and 8, as well as other constraints such as no encryption or signatures.

Around 1990, the Info-ZIP group wrote portable, free, open-source implementations of zip and unzip utilities, supporting compression with the Deflate format, and decompression of that and the earlier formats. This greatly expanded the use of the .zip format.

In the early '90s, the gzip format was developed as a replacement for the Unix compress utility, derived from the Deflate code in the Info-ZIP utilities. Unix compress was designed to compress a single file or stream, appending a .Z to the file name. compress uses the LZW compression algorithm, which at the time was under patent and its free use was in dispute by the patent holders. Though some specific implementations of Deflate were patented by Phil Katz, the format was not, and so it was possible to write a Deflate implementation that did not infringe on any patents. That implementation has not been so challenged in the last 20+ years. The Unix gzip utility was intended as a drop-in replacement for compress, and in fact is able to decompress compress-compressed data (assuming that you were able to parse that sentence). gzip appends a .gz to the file name. gzip uses the Deflate compressed data format, which compresses quite a bit better than Unix compress, has very fast decompression, and adds a CRC-32 as an integrity check for the data. The header format also permits the storage of more information than the compress format allowed, such as the original file name and the file modification time.

Though compress only compresses a single file, it was common to use the tar utility to create an archive of files, their attributes, and their directory structure into a single .tar file, and then compress it with compress to make a .tar.Z file. In fact, the tar utility had and still has the option to do the compression at the same time, instead of having to pipe the output of tar to compress. This all carried forward to the gzip format, and tar has an option to compress directly to the .tar.gz format. The tar.gz format compresses better than the .zip approach, since the compression of a .tar can take advantage of redundancy across files, especially many small files. .tar.gz is the most common archive format in use on Unix due to its very high portability, but there are more effective compression methods in use as well, so you will often see .tar.bz2 and .tar.xz archives.

Unlike .tar, .zip has a central directory at the end, which provides a list of the contents. That and the separate compression provides random access to the individual entries in a .zip file. A .tar file would have to be decompressed and scanned from start to end in order to build a directory, which is how a .tar file is listed.

Shortly after the introduction of gzip, around the mid-1990s, the same patent dispute called into question the free use of the .gif image format, very widely used on bulletin boards and the World Wide Web (a new thing at the time). So a small group created the PNG losslessly compressed image format, with file type .png, to replace .gif. That format also uses the Deflate format for compression, which is applied after filters on the image data expose more of the redundancy. In order to promote widespread usage of the PNG format, two free code libraries were created. libpng and zlib. libpng handled all of the features of the PNG format, and zlib provided the compression and decompression code for use by libpng, as well as for other applications. zlib was adapted from the gzip code.

All of the mentioned patents have since expired.

The zlib library supports Deflate compression and decompression, and three kinds of wrapping around the deflate streams. Those are no wrapping at all ("raw" deflate), zlib wrapping, which is used in the PNG format data blocks, and gzip wrapping, to provide gzip routines for the programmer. The main difference between zlib and gzip wrapping is that the zlib wrapping is more compact, six bytes vs. a minimum of 18 bytes for gzip, and the integrity check, Adler-32, runs faster than the CRC-32 that gzip uses. Raw deflate is used by programs that read and write the .zip format, which is another format that wraps around deflate compressed data.

zlib is now in wide use for data transmission and storage. For example, most HTTP transactions by servers and browsers compress and decompress the data using zlib, specifically HTTP header Content-Encoding: deflate means deflate compression method wrapped inside the zlib data format.

Different implementations of deflate can result in different compressed output for the same input data, as evidenced by the existence of selectable compression levels that allow trading off compression effectiveness for CPU time. zlib and PKZIP are not the only implementations of deflate compression and decompression. Both the 7-Zip archiving utility and Google's zopfli library have the ability to use much more CPU time than zlib in order to squeeze out the last few bits possible when using the deflate format, reducing compressed sizes by a few percent as compared to zlib's highest compression level. The pigz utility, a parallel implementation of gzip, includes the option to use zlib (compression levels 1-9) or zopfli (compression level 11), and somewhat mitigates the time impact of using zopfli by splitting the compression of large files over multiple processors and cores.

2 of 3
65

ZIP is a file format used for storing an arbitrary number of files and folders together with lossless compression. It makes no strict assumptions about the compression methods used, but is most frequently used with DEFLATE.

Gzip is both a compression algorithm based on DEFLATE but less encumbered with potential patents et al, and a file format for storing a single compressed file. It supports compressing an arbitrary number of files and folders when combined with tar. The resulting file has an extension of .tgz or .tar.gz and is commonly called a tarball.

zlib is a library of functions encapsulating DEFLATE in its most common LZ77 incarnation.

Find elsewhere
🌐
zlib
zlib.net
zlib Home Site
Unlike the LZW compression method used in Unix compress(1) and in the GIF image format, the compression method currently used in zlib essentially never expands the data. (LZW can double or triple the file size in extreme cases.) zlib's memory footprint is also independent of the input data ...
🌐
Python
mail.python.org › pipermail › python-list › 2002-February › 155081.html
zlib.so and libz.so and Python
February 14, 2002 - > > "zlib" is the name of a Python module. "zlib.so" is the runtime incarnation > of that module. My guess is that libz.so is a shared library version of the > C "zlib" library. The Python module is essentially just a wrapper around > the zlib library.
🌐
discoversdk
discoversdk.com › compare › libzip-vs-zlib
Compare libzip vs zlib | DiscoverSdk
compare products libzip vs zlib on www.discoversdk.com: Compare products
🌐
zlib
zlib.net › zlib_faq.html
Frequently Asked Questions about zlib
February 7, 2024 - After installing zlib 1.1.4 on Solaris 2.6, running applications using zlib generates an error such as: ld.so.1: rpm: fatal: relocation error: file /usr/local/lib/libz.so: symbol __register_frame_info: referenced symbol not found
🌐
GitHub
github.com › openbsd › src › blob › master › lib › libz › zlib.h
src/lib/libz/zlib.h at master · openbsd/src
/* zlib.h -- interface of the 'zlib' general purpose compression library
Author   openbsd
🌐
Reddit
reddit.com › r/python › analyzing python compression libraries: zlib, lz4, brotli, and zstandard
r/Python on Reddit: Analyzing Python Compression Libraries: zlib, LZ4, Brotli, and Zstandard
April 30, 2024 -

Source Code: https://github.com/dhilipsiva/py-compress-compare

Analyzing Python Compression Libraries: zlib, LZ4, Brotli, and Zstandard

When dealing with large volumes of data, compression can be a critical factor in enhancing performance, reducing storage costs, and speeding up network transfers. In this blog post, we will dive into a comparison of four popular Python compression libraries—zlib, LZ4, Brotli, and Zstandard—using a real-world dataset to evaluate their performance in terms of compression ratio and time efficiency.

The Experiment Setup

Our test involved a dataset roughly 581 KB in size, named sample_data.json. We executed compression and decompression using each library as follows:

  • Compression was performed 1000 times.

  • Decompression was repeated 10,000 times.

This rigorous testing framework ensures that we obtain a solid understanding of each library's performance under heavy load.

Compression Ratio

The compression ratio is a key metric that represents how effectively a compression algorithm can reduce the size of the input data. Here’s how each library scored:

  • Zlib achieved a compression ratio of 27.84,

  • LZ4 came in at 18.23,

  • Brotli impressed with a ratio of 64.78,

  • Zstandard offered a ratio of 43.42.

From these results, Brotli leads with the highest compression ratio, indicating its superior efficiency in data size reduction. Zstandard also shows strong performance, while LZ4, though lower, still provides a reasonable reduction.

Compression Time

Efficiency isn't just about space savings; time is equally crucial. Here’s how long each library took to compress the data:

  • Zlib: 7.34 seconds,

  • LZ4: 0.13 seconds,

  • Brotli: 204.18 seconds,

  • Zstandard: 0.15 seconds.

LZ4 and Zstandard excel in speed, with LZ4 being slightly faster. Zlib offers a middle ground, but Brotli, despite its high compression efficiency, takes significantly longer, which could be a drawback for real-time applications.

Decompression Time

Decompression time is vital for applications where data needs to be rapidly restored to its original state:

  • Zlib: 11.99 seconds,

  • LZ4: 0.46 seconds,

  • Brotli: 0.99 seconds,

  • Zstandard: 0.46 seconds.

Again, LZ4 and Zstandard show excellent performance, both under half a second. Brotli presents a decent time despite its lengthy compression time, while zlib lags behind in this aspect.

Conclusion

Each library has its strengths and weaknesses:

  • Brotli is your go-to for maximum compression but at the cost of time, making it suitable for applications where compression time is less critical.

  • Zstandard offers a great balance between compression ratio and speed, recommended for a wide range of applications.

  • LZ4 shines in speed, ideal for scenarios requiring rapid data processing.

  • Zlib provides moderate performance across the board.

Choosing the right library depends on your specific needs, whether it’s speed, space, or a balance of both. This experiment provides a clear picture of what to expect from these libraries, helping you make an informed decision based on your application's requirements.

🌐
GitHub
github.com › zlib-ng › zlib-ng
GitHub - zlib-ng/zlib-ng: zlib replacement with optimizations for "next generation" systems. · GitHub
I decided to make a fork, merge all the Intel optimizations, some of the Cloudflare optimizations, plus a couple other smaller patches. Then started cleaning out workarounds, various dead code, all contrib and example code. The result is a better performing and easier to maintain zlib-ng.
Starred by 2K users
Forked by 330 users
Languages   C 82.6% | CMake 8.0% | C++ 7.2% | Makefile 1.5%
🌐
Rust Internals
internals.rust-lang.org › t › psa-libz-sys-flate2-and-git2-now-support-the-high-performance-zlib-ng › 12924
PSA: libz-sys, flate2, and git2 now support the high-performance zlib-ng - Rust Internals
August 19, 2020 - libz-sys provides Rust bindings to the zlib compression library, for handling DEFLATE/gzip compression. I've just released libz-sys 1.1.0, which provides opt-in support for the high-performance zlib-ng library in zlib-compat mode. zlib-ng provides ...
🌐
Baeldung
baeldung.com › home › algorithms › data compression: zlib vs. gzip vs. zip
Data Compression: ZLib vs. GZip vs. Zip | Baeldung on Computer Science
March 18, 2024 - The main drawback of zlib is that it doesn’t have any checksum mechanism to maintain the integrity of data.
🌐
GitHub
github.com › zlib-ng › zlib-ng › discussions › 871
2.0.0 Benchmark comparisons · zlib-ng/zlib-ng · Discussion #871
Gzip compression is about twice as fast as zlib, but zlib compresses slightly better than gzip (due to minigzip using a bare minimum of headers?), and decompression with gzip takes about 30% less time.
Author   zlib-ng
🌐
GitHub
github.com › zlib-ng › zlib-ng › issues › 1279
`libz-ng.a` and `libz.a` from zlib-compat mode have symbol conflicts · Issue #1279 · zlib-ng/zlib-ng
May 30, 2022 - zlib-ng in native mode avoids symbol conflicts with stock zlib, but still has symbol conflicts with zlib supplied from zlib-ng built in zlib-compat mode. This can cause an issue in a few different ...
Author   zlib-ng
🌐
OpenMamba
openmamba.org › en › rpms › base › libz › aarch64
libz: The zlib compression and decompression library
The zlib library is used by many different system programs. ... /usr/lib/.build-id /usr/lib/.build-id/96 /usr/lib/.build-id/96/eb23480b5ebc70f336f6e92778959446358ab2 /usr/lib64/libz.so.1 /usr/lib64/libz.so.1.3.2 /usr/share/doc/libz-1.3.2 /usr/share/doc/libz-1.3.2/LICENSE