Zstandard v1.5.7 is a significant release, featuring over 500 commits accumulated over the past year. This update brings enhancements across various domains, including performance, stability, and functionality, and is particularly recommended for 32-bit users due to a bug fix detailed below.
edit: man pages were not properly updated in this package, and are still pointing to v1.5.6. If you want the v1.5.7 version, they are updated in dev branch, and can be cherry-picked using commit 6af3842118ea5325480b403213b2a9fbed3d3d74. Alternatively, they can also be built locally using make manual and make -C programs man.
Performance Enhancements
Enhanced Compression Speed for Small Data Blocks
The compression speed for small data blocks has been notably improved at fast compression levels, thanks to contributions from @TocarIP, further extended in #4165. Below are benchmark results illustrating the performance improvements for level 1 compression on the Silesia corpus, segmented into different block sizes:
These improvements are particularly beneficial in data centers and databases, where data is often divided into small segments to enhance seekability. A notable example is RocksDB, which is frequently configured with block sizes around 16 KB. Such systems will experience an immediate performance boost by updating their compression library to v1.5.7.
Additionally, the speed improvements partially extend to dictionary compression, with observed gains ranging from none to +15%.
Substantial --patch-from improvements
The --patch-from functionality of the zstd CLI allows for differential compression of a file given an older version, similar to bsdiff. While compression levels impact processing speed and compression ratio as expected, at high compression levels (18+), speed was previously reduced to uncomfortable levels.
v1.5.7 largely mitigates the speed impact of high compression levels 18+, as illustrated in the following scenario, which compresses linux-6.13.tar as a difference from linux-6.12.tar using 6 threads:
| level | v1.5.6 | v1.5.7 | Improv. |
| --- | --- | --- | --- |
| 18 | 210 s | 42.7 s | x4.9 |
| 19 | 273 s | 61.8 s | x4.4 |
| 20 | 310 s | 82.3 s | x3.8 |
| 21 | 365 s | 120 s | x3.1 |
Additionally, compression ratio of --patch-from is also substantially improved across the board, including higher compression levels:
The compression ratio has been enhanced slightly for large data across all compression levels, thanks to a refined approach in selecting block boundaries. This improvement is more pronounced at lower compression levels (≤ 15), where the previous boundary selection method was relatively basic. In contrast, higher compression levels (16+) already employed some analysis, but still benefit from the introduction of the new heuristic, resulting in marginally better compression ratios.
A comparison of the compression ratios between v1.5.6 and v1.5.7 is presented below:
Recognizing the prevalence of multi-core systems, zstd now employs multiple threads by default, thanks to #4211 by @daniellerozenblit
In v1.5.7, systems will see default threading scaled according to capability, but remains cautious to mitigate risks of overloading surprises on existing user base, with a maximum of 4 threads by default.
For example, a typical "low-end" system with 4 cores and hyperthreading will employ 2 threads by default. A "higher-end" system with 8 cores and hyperthreading will employ 4 threads by default.
These modifications only impact default settings. As usual, the number of threads that zstd uses can also be set manually, either via command line or via environment variables.
Additionally, a new zstd command --max is available, designed to maximize compression ratios by leveraging all parameters, at the expense of time and memory. It's slower and more resource hungry than --ultra -22, but achieves denser compressed archives if that's the goal. For instance, compressing enwik9 (1,000,000,000 bytes) at --max level on a i7-9700k platform yields:
v1.5.7 resolves a long standing and very rare compression issue in 32-bit mode, which can be triggered during long-lasting sessions (same ZSTD_CCtx* being continuously reused). This issue required several conditions to be met and a large amount of data to occur, but at the scale of zstd usage, it has been observed and reported (#4292).
In summary, if you use libzstd in 32-bit mode, this update is highly recommended.
Miscellaneous
The release also brings its lot of various improvements, impacting build scripts (cmake, meson, Apple Framework, Visual Studio), documentation and portability (QNX, FreeBSD, Hurd, icc/icx).
Notably, there are measurable performance gains for binaries produced with Visual Studio, as a cumulative result of several contributions.
Changelog:
fix: compression bug in 32-bit mode associated with long-lasting sessions
perf: substantial compression speed improvements (up to +30%) on small data, by @TocarIP (#4144) and @cyan4973 (#4165)
perf: improved compression speed (~+5%) for dictionary compression at low levels (#4170)
perf: much faster --patch-from at high compression levels (#4276)
perf: higher --patch-from compression ratios, notably at high levels (#4288)
perf: better speed for binaries on Windows (@pps83) and when compiled with Visual Studio (@MessyHack)
perf: slightly improve compression ratio across all levels thanks to better block boundaries (#4136, #4176, #4178)
perf: slight compression ratio improvement for dfast, aka levels 3 and 4 (#4171)
perf: runtime bmi2 detection now also enabled on x86 32-bit mode (#4251)
cli: multi-threading as default CLI setting, by @daniellerozenblit
cli: new --max command (#4290)
api: new method ZSTD_compressSequencesAndLiterals() (#4217, #4232)
api: ZSTD_getFrameHeader() now works on skippable frames (#4228)
build: improve msbuild version autodetection, support VS2022, by @ManuelBlanc
build: fix meson build by @artem and @Victor-C-Zhang, and on Windows by @bgilbert
build: compatibility with Apple Framework, by @Treata11
build: improve icc/icx compatibility, by @josepho0918 and @luau-project
build: improve compatibility with Android NDK, by Adenilson Cavalcanti
portability: linux kernel branch, with improved support for Sequence producers (@embg, @gcabiddu, @cyan4973)
portability: improved qnx compatibility, suggested by @rainbowball
portability: improved install script for FreeBSD, by @sunpoet
portability: fixed test suite compatibility with gnu hurd, by @diegonc
doc: clarify specification, by @elasota
misc: improved tests/decodecorpus validation tool (#4102), by antmicro
New Contributors
@ManuelBlanc made their first contribution in https://github.com/facebook/zstd/pull/3987
@sunpoet made their first contribution in https://github.com/facebook/zstd/pull/3994
@georgthegreat made their first contribution in https://github.com/facebook/zstd/pull/3998
@pstef made their first contribution in https://github.com/facebook/zstd/pull/4021
@Ansuel made their first contribution in https://github.com/facebook/zstd/pull/4020
@b7f7 made their first contribution in https://github.com/facebook/zstd/pull/4019
@bgilbert made their first contribution in https://github.com/facebook/zstd/pull/3931
@alexlnkp made their first contribution in https://github.com/facebook/zstd/pull/4025
@r-barnes made their first contribution in https://github.com/facebook/zstd/pull/4034
@Adenilson made their first contribution in https://github.com/facebook/zstd/pull/4050
@Federico-Maresca made their first contribution in https://github.com/facebook/zstd/pull/4039
@jbajic made their first contribution in https://github.com/facebook/zstd/pull/4054
@tpetazzoni made their first contribution in https://github.com/facebook/zstd/pull/4096
@uilianries made their first contribution in https://github.com/facebook/zstd/pull/4101
@jclab-joseph made their first contribution in https://github.com/facebook/zstd/pull/4087
@RubenKelevra made their first contribution in https://github.com/facebook/zstd/pull/4094
@proppy made their first contribution in https://github.com/facebook/zstd/pull/4102
@xionghul made their first contribution in https://github.com/facebook/zstd/pull/4119
@inventor500 made their first contribution in https://github.com/facebook/zstd/pull/4134
@dearblue made their first contribution in https://github.com/facebook/zstd/pull/4154
@artem made their first contribution in https://github.com/facebook/zstd/pull/4153
@lukaso made their first contribution in https://github.com/facebook/zstd/pull/4191
@jimis made their first contribution in https://github.com/facebook/zstd/pull/4199
@nhz2 made their first contribution in https://github.com/facebook/zstd/pull/4202
@rorosen made their first contribution in https://github.com/facebook/zstd/pull/4201
@medns made their first contribution in https://github.com/facebook/zstd/pull/4118
@Victor-C-Zhang made their first contribution in https://github.com/facebook/zstd/pull/4210
@kevinjzhang made their first contribution in https://github.com/facebook/zstd/pull/4213
@diegonc made their first contribution in https://github.com/facebook/zstd/pull/4212
@RicBent made their first contribution in https://github.com/facebook/zstd/pull/4220
@yotann made their first contribution in https://github.com/facebook/zstd/pull/4080
@MessyHack made their first contribution in https://github.com/facebook/zstd/pull/4234
@szsam made their first contribution in https://github.com/facebook/zstd/pull/4238
@lalakii made their first contribution in https://github.com/facebook/zstd/pull/4242
@pps83 made their first contribution in https://github.com/facebook/zstd/pull/4246
@Treata11 made their first contribution in https://github.com/facebook/zstd/pull/4259
@gcabiddu made their first contribution in https://github.com/facebook/zstd/pull/4260
@luau-project made their first contribution in https://github.com/facebook/zstd/pull/4269
@tsdgeos made their first contribution in https://github.com/facebook/zstd/pull/4271
Full Changelog: https://github.com/facebook/zstd/compare/v1.5.6...v1.5.7