zipfile regression in 3.11 beta: ignores utf8 flag and removes it from existing entries #95463
Comments
|
NB: this regression also affects f-droid (which uses apksigcopier for reproducible builds). cc @eighthave |
|
CC: @gpshead as you reviewed the PR. Unless someone has a good reason not to, I plan to revert b5cf7374d79aae191c1e38d0959527e0bbb7a95e0df5b125a8b1056cc2c54851L483 tomorrow. |
|
@obfusk Thanks for the very detailed report |
|
Yes, good writeup. I'm fine with a revert of PR #32007's a25a985535ccbb7df8caddc0017550ff4eae5855. (I think you mispasted something else that isn't a commit id in your message pablogsal) It seems more is needed here in order to preserve existing bits. |
|
Oh I see, you were linking just to the particular one line of the change. :) If undoing just line itself is the fix, great. It sounds like we don't have test coverage for this situation though. |
Yes, reverting just the single line changed in I can confirm that monkey-patching Of course, adding additional test coverage to prevent regressions like this in the future would certainly be nice IMO. Thanks for getting this fixed before 3.11 is released! |
…ASK_UTF_FILENAME flags in bpo-28080
…ASK_UTF_FILENAME flags in bpo-28080 Signed-off-by: Pablo Galindo <pablogsal@gmail.com>
…ASK_UTF_FILENAME flags in bpo-28080 (pythonGH-96072) Automerge-Triggered-By: GH:pablogsal (cherry picked from commit 9d066e2) Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
|
This seems to be fixed, so I am closing it. Please re-open you think we missed something. |
|
apksigcopier CI is green again with 3.11rc2 and the workaround removed, so seems to be fixed — as expected — indeed. Thanks! |
obfusk commentedJul 30, 2022
Bug report
apksigcopier CI started failing on 3.11 beta because of this seemingly unnecessary change in #32007 (which added fallback encoding support):
def _encodeFilenameFlags(self): try: - return self.filename.encode('ascii'), self.flag_bits + return self.filename.encode('ascii'), self.flag_bits & ~_MASK_UTF_FILENAME except UnicodeEncodeError:which results in:
ZipInfo.flag_bits(whether set manually or from an existing entry that already had it) being completely ignored (i.e. set/unset based on whether the file name can be encoded in ascii or requires utf8, regardless of the current value) when file headers and the central directory are written, whereas before it was already being set when required but not unset if not required (and having ascii filenames with the utf8 flag set is completely fine and common, but now impossible);Which resulted in zip files that were no longer bitwise identical, making the existing lack of support for creating reproducible/deterministic zip files even worse than before.
Please revert this change; I'm happy to make a PR for that.
Your environment
The text was updated successfully, but these errors were encountered: