zlib-ng is an optimized zlib library with better performance on most architectures (with contributions from the likes of Google, Cloudflare, and Intel). It is API compatible with zlib. https://github.com/zlib-ng/zlib-ng

I believe the only platform we don't use the OS's own zlib on is Windows so I'm tagging this issue Windows.

if this hasn't happened in the 3.11 betas this is presumably bumped to 3.12.

I started taking a look at this and it seems like we can build it without having to worry about their build system by renaming the zconf.h.in and zconf-ng.h.in files to remove the .in, and setting ZLIB_COMPAT preprocessor in pythoncore.vcxproj (as well as referencing the files).

Running test_zlib the only failures seem to be testing for certain failures that no longer fail, but I've got no idea how important they are. Also no indication of the performance impact, or anything else that may change (e.g. new DLL exports, etc.), but it certainly does seem like a feasible drop-in replacement.

@zooba which failures are these? I have accumulated quite some experience with the zlib/gzip formats due to working on python-isal, bindings for the ISA-L library that speeds up zlib-compatible compression by rewriting the algorithms in x86 Assembly language. It is quite good, but not suitable for a drop-in replacement, hence the python-isal project.

Having said that I'd love to help out with anything zlib related in CPython.

I knew I should've kept better track of the changed error messages 😄

Skimming through the tests, I'm pretty sure test_wbits failed in a few places because we expected some "invalid window size" errors that aren't errors with zlib-ng. I don't think any were critical, but I don't know the usage of the library well enough to be sure (e.g. do people expect these errors and change behaviour? or are they always just developer error and cause changes to just make things work?)

Checking internally for local patches to zlib-ng, this change zlib-ng/zlib-ng@ce6789c appears to fix that problem. I guess it hasn't landed in a zlib-ng release yet. (I don't see it in the 2.0.x branch)

Lets make sure that lands in zlib-ng before using it in our releases I guess.

Otherwise: I don't see any Python code using zlib.error that'd care (internally and in a large corpus of third party OSS python code). It is mostly only ever caught to bail out or follows this pattern to try the alternate meaning of wbits for zlib vs gzip format: https://github.com/urllib3/urllib3/blob/main/src/urllib3/response.py#L102

I started taking a look at this and it seems like we can build it without having to worry about their build system by renaming the zconf.h.in and zconf-ng.h.in files to remove the .in, and setting ZLIB_COMPAT preprocessor in pythoncore.vcxproj (as well as referencing the files).

I don't recommend bypassing the build system as it is used to determine features of the compiler and what optimizations to enable. If you only use Visual Studio then it is best to use cmake to generate your Visual Studio projects.

I don't recommend bypassing the build system

Yeah, I figured this would be a bad idea, but we're also not going to run a separate build system every time. We'll run it once and keep the generated files in our source mirror. (This extra work is why I didn't bother setting it all up earlier, and just hacked things enough to make it build.)