Python mtime timestamp logic does not work well with ostree force 0 mtime.
Use a small script to post-process all compiled '.pyc' and force the timestamp to 0.
See: https://github.com/ostreedev/ostree/issues/1469
Co-Authored-By: Miro Hrončok miro@hroncok.cz
Build failed. More information on how to proceed and troubleshoot errors available at https://fedoraproject.org/wiki/Zuul-based-ci https://fedora.softwarefactory-project.io/zuul/buildset/76269db66ce5463683a8c183e59c1b68
rebased onto 5aaeaa435986691e3133b69e5a0e3a432bb3d632
This function is missing return 0 at the end (when invalidation wasn't ZERO). It now implicitly returns None in that case and will blow up if such .pyc file is found because you cannot add None to an integer.
Build failed. More information on how to proceed and troubleshoot errors available at https://fedoraproject.org/wiki/Zuul-based-ci https://fedora.softwarefactory-project.io/zuul/buildset/6599d1a571744ce2be763456440d5450
The exact thing happens on the CI:
Traceback (most recent call last): File "<stdin>", line 34, in <module> TypeError: unsupported operand type(s) for +=: 'int' and 'NoneType'
Style nitpick: ALL_CAPS are usually used for constants, this isn't a constant (but regex is).
rebased onto 0728189946c058c007b53b03b143d47d07e434f3
rebased onto d085e9fecccaf5d0c164b8a67d3b046dc36df39f
Build succeeded. https://fedora.softwarefactory-project.io/zuul/buildset/78bd612d79104b49ad8957afce6d3d85
Processed 4855 pyc files \o/
On another look, I guess this len check is redundant because if the len is less than 4, it will never equal ZERO (which is len 4).
Let's give this one a try. I'll merge it so that it lands in Rawhide and when can check if it works as intended or fix it. Thanks a lot @churchyard !
Pull-Request has been merged by siosm
It looks "better" on latest Rawhide but not fully fixed:
$ python3 -v -c 'import mailbox' 2>&1 | grep '# bytecode is stale for' -A2 # bytecode is stale for 'urllib' # code object from /usr/lib64/python3.12/urllib/__init__.py # could not create '/usr/lib64/python3.12/urllib/__pycache__/__init__.cpython-312.pyc': OSError(30, 'Read-only file system')
On Rawhide.20240413.n.0:
Rawhide.20240413.n.0
$ sudo find /usr/ -ipath "*/__pycache__/*.cpython-*.pyc" | wc -l 4988
From https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20240413.n.0/logs/x86_64/Silverblue/ostree-7/create-ostree-repo.log:
Processed 4885 pyc files
Weird.
Shouldn't this be != ZERO ?
!= ZERO
Nah. The second 32bit word is a bitfield that encodes the type of invalidation. ZERO (empty bitfield) means this pyc file is timestamp-invalidated. Then, and only then, we know we can zero-out the third 32bit word, which carries the mtime.
Nothing special about that file on a regular rawhide system:
>>> REGEX.match(path) <re.Match object; span=(0, 65), match='/usr/lib64/python3.12/urllib/__pycache__/__init__> >>> f = open(path, "r+b") >>> w = f.read(4) >>> w b'\xcb\r\r\n' >>> magic = (w[0] + (w[1] << 8) + (w[2] << 16) + (w[3] << 24)) & 0xFFFF >>> magic 3531 >>> invalidation = f.read(4) >>> invalidation b'\x00\x00\x00\x00' >>> w = f.read(4) >>> mtime = (w[0] + (w[1] << 8) + (w[2] << 16) + (w[3] << 24)) >>> mtime 1712707200
:/
OK, this is weird then. Feels like we're missing something.
We can print the paths we are zeroing-mtimes for and see if we skip this one or if it is zeroed and later unzeroed by something else.
https://pagure.io/workstation-ostree-config/pull-request/507
Next step in https://pagure.io/workstation-ostree-config/pull-request/536
Python mtime timestamp logic does not work well with ostree force 0
mtime.
Use a small script to post-process all compiled '.pyc' and force the
timestamp to 0.
See: https://github.com/ostreedev/ostree/issues/1469
Co-Authored-By: Miro Hrončok miro@hroncok.cz