fedora-reproducible-builds / project

#5 .gnu_debuglink sections are non-reproducible
Closed by zbyszek. Opened by zbyszek.

Closed
Reopen Issue

Example from https://discussion.fedoraproject.org/t/report-from-the-reproducible-builds-hackfest-during-flock-2023/87469:

│ ├── ./usr/libexec/valgrind/cachegrind-amd64-linux
│ │┄ File has been modified after NT_GNU_BUILD_ID has been applied.
│ │ ├── readelf --wide --decompress --hex-dump=.gnu_debuglink {}
│ │ │ @@ -1,7 +1,7 @@
│ │ │  
│ │ │  Hex dump of section '.gnu_debuglink':
│ │ │    0x00000000 63616368 65677269 6e642d61 6d643634 cachegrind-amd64
│ │ │    0x00000010 2d6c696e 75782d33 2e32312e 302d382e -linux-3.21.0-8.
│ │ │    0x00000020 66633339 2e783836 5f36342e 64656275 fc39.x86_64.debu
│ │ │ -  0x00000030 67000000 919eb441                   g......A
│ │ │ +  0x00000030 67000000 3fdf2fd9                   g...?./.

https://sourceware.org/gdb/onlinedocs/gdb/Separate-Debug-Files.html says:

The executable contains a debug link that specifies the name of the separate debug info file. The separate debug file’s name is usually executable.debug, where executable is the name of the corresponding executable file without leading directories (e.g., ls.debug for /usr/bin/ls). In addition, the debug link specifies a 32-bit Cyclic Redundancy Check (CRC) checksum for the debug file, which GDB uses to validate that the executable and the debug file came from the same build.

The filename is terminated by NUL, and then the checksum is after that at the next 4-byte alignment.

It seems that the CRC is non-reproducible. This seems to affect many packages with compiled binaries.

$ rpmdiff original/valgrind-debuginfo-3.21.0-8.fc39.x86_64.rpm rebuilt/valgrind-debuginfo-3.21.0-8.fc39.x86_64.rpm
..5........ /usr/lib/debug/usr/libexec/valgrind/cachegrind-amd64-linux-3.21.0-8.fc39.x86_64.debug
..5........ /usr/lib/debug/usr/libexec/valgrind/callgrind-amd64-linux-3.21.0-8.fc39.x86_64.debug
..5........ /usr/lib/debug/usr/libexec/valgrind/dhat-amd64-linux-3.21.0-8.fc39.x86_64.debug
..5........ /usr/lib/debug/usr/libexec/valgrind/drd-amd64-linux-3.21.0-8.fc39.x86_64.debug
..5........ /usr/lib/debug/usr/libexec/valgrind/exp-bbv-amd64-linux-3.21.0-8.fc39.x86_64.debug
..5........ /usr/lib/debug/usr/libexec/valgrind/helgrind-amd64-linux-3.21.0-8.fc39.x86_64.debug
..5........ /usr/lib/debug/usr/libexec/valgrind/lackey-amd64-linux-3.21.0-8.fc39.x86_64.debug
..5........ /usr/lib/debug/usr/libexec/valgrind/massif-amd64-linux-3.21.0-8.fc39.x86_64.debug
..5........ /usr/lib/debug/usr/libexec/valgrind/memcheck-amd64-linux-3.21.0-8.fc39.x86_64.debug
..5........ /usr/lib/debug/usr/libexec/valgrind/none-amd64-linux-3.21.0-8.fc39.x86_64.debug

.gdb_index sections are different:

$ s=gdb_index; readelf --wide --debug-dump=$s root/usr/lib/debug/usr/libexec/valgrind/cachegrind-amd64-linux-3.21.0-8.fc39.x86_64.debug >$s.1 && readelf --wide --debug-dump=$s root2/usr/lib/debug/usr/libexec/valgrind/cachegrind-amd64-linux-3.21.0-8.fc39.x86_64.debug >$s.2
$ diff -u $s.{1,2} --color=always
@@ -1006,7 +1006,7 @@
 [1171] Iop_InterleaveOddLanes32x4: 677 [static, variable]
 [1174] alloc_DebugInfo: 704 [static, function]
 [1175] Iop_Ctz64x2: 677 [static, variable]
-[1185] handle_maybe_load_notifier: 742 [static, function]
+[1185] _abbv_state: 752 [static, type]
 [1202] init_preopened_fds_without_proc_self_fd: 724 [static, function]
 [1205] vgPlain_OSetGen_Size: 688 [global, function]
 [1210] add_var_to_arange: 707 [static, function]
@@ -1049,13 +1049,13 @@
 [1803] EmWarn_X86_daz: 721 [static, variable]
 [1818] Iex_GSPTR: 677 [static, variable]
 [1821] iselDblExpr: 784 [static, function]
-[1823] Iop_And1: 677 [static, variable]
+[1823] restore_rt_sigframe: 722 [static, function]
 [1830] Iop_And8: 677 [static, variable]
 [1855] isValidSector: 697 [static, function]
 [1859] Iop_I64UtoD64: 677 [static, variable]
 [1878] DW_AT_namelist_items: 749 [static, variable]
 [1881] IRExpr_GSPTR: 774 [global, function]
-[1886] vgPlain_track_pre_mem_read: 696 [global, function]
+[1886] VG_USERREQ__PRINTF: 682 [static, variable]
 [1905] D_PRINT_LONG_LONG: 771 [static, variable]
 [1931] vgModuleLocal_img_strdup: 750 [global, function]
 [1934] mon_out_buf: 713 [static, type]
@@ -1077,7 +1077,7 @@
 [2358] DW_TAG_structure_type: 749 [static, variable]
 [2364] h_generic_calc_HAdd8Ux4: 788 [global, function]
 [2372] DW_TAG_pointer_type: 749 [static, variable]
-[2381] Iop_Ctz32x4: 677 [static, variable]
+[2381] stepping: 717 [static, variable]
 [2393] Iop_CmpNEZ16x16: 677 [static, variable]
 [2399] Iop_Mul8x16: 677 [static, variable]
 [2436] VgSrc_ExitProcess: 681 [static, variable]
...

It's possible that this just an ordering issue (the list of functions is not sorted, or is sorted using a different seed?).

Screenshot_from_2023-08-15_10-03-36.png

→ https://bugzilla.redhat.com/show_bug.cgi?id=2232086

Hi,
are you rebuilding using rpmbuild --rebuild srpm?

are you rebuilding using rpmbuild --rebuild srpm?

I'm using mock. I initialize a mock environment with the package list gleaned from koji, and then do a more-or-less-normal mock build in that environment. mock does a rebuild of the srpm, but I'm not sure how it does that exactly.

Metadata Update from @zbyszek:
- Issue tagged with: problem

This is now fixed. The relevant commits:
https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=aff250145af
https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=3644f41dc80
and also interesting:
https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=aa19bc1d259

Metadata Update from @zbyszek:
- Issue status updated to: Closed (was: Open)
Metadata
None

irreproducibility

None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.