freeipa

#9453 Nightly test failure (f38 with updates-testing) in replica installation
Closed: fixed by frenaud. Opened by frenaud.

Closed: fixed
Reopen Issue

Issue

The nightly tests running on fedora38 with updates-testing repo enabled fail when installing a replica.

See for instance PR #3007 with the test simple_replication (Logs, report):

E           subprocess.CalledProcessError: Command '['ipa-replica-install', '--admin-password', 'Secret.123', '--password', 'Secret.123', '-U', '--setup-ca', '--ip-address', '192.168.122.233', '--realm', 'IPA.TEST', '--domain', 'ipa.test', '--dirsrv-config-file', '/ipatests/ipatests_dse.ldif']' returned non-zero exit status 1.
pytest_ipa/integration/host.py:202: CalledProcessError
 -----------------------------Captured stderr setup------------------------------ 
ipa: ERROR: stderr: Lookup failed: Preferred host replica0.ipa.test does not provide DNS.
Reverse DNS resolution of address 192.168.122.76 (master.ipa.test) failed. Clients may not function properly. Please check your DNS setup. (Note that this check queries IPA DNS directly and ignores /etc/hosts.)
Validate installation settings ...
Create file system structures ...
Perform SELinux labeling ...
Create database backend: dc=ipa,dc=test ...
Perform post-installation tasks ...
CalledProcessError(Command ['/usr/sbin/ipa-getkeytab', '-k', '/etc/dirsrv/ds.keytab', '-p', 'ldap/replica0.ipa.test@IPA.TEST', '-H', 'ldaps://master.ipa.test'] returned non-zero exit status -6: "Keytab successfully retrieved and stored in: /etc/dirsrv/ds.keytab\nk5_mutex_lock: Received error 22 (Invalid argument)\nipa-getkeytab: ../../include/k5-thread.h:376: k5_mutex_lock: Assertion `r == 0' failed.\n")
The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information

Actual behavior

The command ipa-replica-install fails in the step obtaining the machine keytab:

2023-09-21T06:50:15Z DEBUG   [27/43]: creating DS keytab
2023-09-21T06:50:15Z DEBUG raw: service_add('ldap/replica0.ipa.test@IPA.TEST', force=True, version='2.253')
2023-09-21T06:50:15Z DEBUG service_add(ipapython.kerberos.Principal('ldap/replica0.ipa.test@IPA.TEST'), force=True, skip_host_check=False, all=False, raw=False, version='2.253', no_members=False)
2023-09-21T06:50:15Z DEBUG raw: host_show('replica0.ipa.test', version='2.253')
2023-09-21T06:50:15Z DEBUG host_show('replica0.ipa.test', rights=False, all=False, raw=False, version='2.253', no_members=False)
2023-09-21T06:50:15Z DEBUG Backing up system configuration file '/etc/dirsrv/ds.keytab'
2023-09-21T06:50:15Z DEBUG   -> Not backing up - '/etc/dirsrv/ds.keytab' doesn't exist
2023-09-21T06:50:15Z DEBUG Starting external process
2023-09-21T06:50:15Z DEBUG args=['/usr/sbin/ipa-getkeytab', '-k', '/etc/dirsrv/ds.keytab', '-p', 'ldap/replica0.ipa.test@IPA.TEST', '-H', 'ldaps://master.ipa.test']
2023-09-21T06:50:16Z DEBUG Process finished, return code=-6
2023-09-21T06:50:16Z DEBUG stdout=
2023-09-21T06:50:16Z DEBUG stderr=Keytab successfully retrieved and stored in: /etc/dirsrv/ds.keytab
k5_mutex_lock: Received error 22 (Invalid argument)
ipa-getkeytab: ../../include/k5-thread.h:376: k5_mutex_lock: Assertion `r == 0' failed.
2023-09-21T06:50:16Z DEBUG Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/ipaserver/install/service.py", line 686, in start_creation
    run_step(full_msg, method)
  File "/usr/lib/python3.11/site-packages/ipaserver/install/service.py", line 672, in run_step
    method()
  File "/usr/lib/python3.11/site-packages/ipaserver/install/service.py", line 862, in request_service_keytab
    self.run_getkeytab(self.api.env.ldap_uri, self.keytab, self.principal)
  File "/usr/lib/python3.11/site-packages/ipaserver/install/service.py", line 852, in run_getkeytab
    ipautil.run(args, nolog=nolog)
  File "/usr/lib/python3.11/site-packages/ipapython/ipautil.py", line 599, in run
    raise CalledProcessError(
ipapython.ipautil.CalledProcessError: CalledProcessError(Command ['/usr/sbin/ipa-getkeytab', '-k', '/etc/dirsrv/ds.keytab', '-p', 'ldap/replica0.ipa.test@IPA.TEST', '-H', 'ldaps://master.ipa.test'] returned non-zero exit status -6: "Keytab successfully retrieved and stored in: /etc/dirsrv/ds.keytab\nk5_mutex_lock: Received error 22 (Invalid argument)\nipa-getkeytab: ../../include/k5-thread.h:376: k5_mutex_lock: Assertion `r == 0' failed.\n")
2023-09-21T06:50:16Z DEBUG   [error] CalledProcessError: CalledProcessError(Command ['/usr/sbin/ipa-getkeytab', '-k', '/etc/dirsrv/ds.keytab', '-p', 'ldap/replica0.ipa.test@IPA.TEST', '-H', 'ldaps://master.ipa.test'] returned non-zero exit status -6: "Keytab successfully retrieved and stored in: /etc/dirsrv/ds.keytab\nk5_mutex_lock: Received error 22 (Invalid argument)\nipa-getkeytab: ../../include/k5-thread.h:376: k5_mutex_lock: Assertion `r == 0' failed.\n")

The journal shows a coredump:

Sep 21 06:50:15 replica0.ipa.test systemd[1]: Started systemd-coredump@1-19900-0.service - Process Core Dump (PID 19900/UID 0).
Sep 21 06:50:16 replica0.ipa.test systemd-coredump[19901]: Process 19899 (ipa-getkeytab) of user 0 dumped core.
                                                           Module libgssrpc.so.4 from rpm krb5-1.21-3.fc38.x86_64
                                                           Module libverto.so.1 from rpm libverto-0.3.2-5.fc38.x86_64
                                                           Module proxymech.so from rpm gssproxy-0.9.1-5.fc38.x86_64
                                                           Module libdigestmd5.so from rpm cyrus-sasl-2.1.28-9.fc38.x86_64
                                                           Module libcrammd5.so from rpm cyrus-sasl-2.1.28-9.fc38.x86_64
                                                           Module libplain.so from rpm cyrus-sasl-2.1.28-9.fc38.x86_64
                                                           Module liblogin.so from rpm cyrus-sasl-2.1.28-9.fc38.x86_64
                                                           Module libgssapi_krb5.so.2 from rpm krb5-1.21-3.fc38.x86_64
                                                           Module libgssapiv2.so from rpm cyrus-sasl-2.1.28-9.fc38.x86_64
                                                           Module libsasldb.so from rpm cyrus-sasl-2.1.28-9.fc38.x86_64
                                                           Module libanonymous.so from rpm cyrus-sasl-2.1.28-9.fc38.x86_64
                                                           Module libpcre2-8.so.0 from rpm pcre2-10.42-1.fc38.1.x86_64
                                                           Module libcrypt.so.2 from rpm libxcrypt-4.4.36-1.fc38.x86_64
                                                           Module libselinux.so.1 from rpm libselinux-3.5-1.fc38.x86_64
                                                           Module libbasicobjects.so.0 from rpm ding-libs-0.6.2-53.fc38.x86_64
                                                           Module libref_array.so.1 from rpm ding-libs-0.6.2-53.fc38.x86_64
                                                           Module libpath_utils.so.1 from rpm ding-libs-0.6.2-53.fc38.x86_64
                                                           Module libcollection.so.4 from rpm ding-libs-0.6.2-53.fc38.x86_64
                                                           Module libssl.so.3 from rpm openssl-3.0.9-2.fc38.x86_64
                                                           Module libsasl2.so.3 from rpm cyrus-sasl-2.1.28-9.fc38.x86_64
                                                           Module libevent-2.1.so.7 from rpm libevent-2.1.12-8.fc38.x86_64
                                                           Module libkeyutils.so.1 from rpm keyutils-1.6.1-6.fc38.x86_64
                                                           Module libkrb5support.so.0 from rpm krb5-1.21-3.fc38.x86_64
                                                           Module libz.so.1 from rpm zlib-1.2.13-3.fc38.x86_64
                                                           Module libini_config.so.5 from rpm ding-libs-0.6.2-53.fc38.x86_64
                                                           Module libpopt.so.0 from rpm popt-1.19-2.fc38.x86_64
                                                           Module libcom_err.so.2 from rpm e2fsprogs-1.46.5-4.fc38.x86_64
                                                           Module libk5crypto.so.3 from rpm krb5-1.21-3.fc38.x86_64
                                                           Module libkrb5.so.3 from rpm krb5-1.21-3.fc38.x86_64
                                                           Module libcrypto.so.3 from rpm openssl-3.0.9-2.fc38.x86_64
                                                           Module ipa-getkeytab from rpm freeipa-4.12.0.dev-0.fc38.x86_64
                                                           Stack trace of thread 19899:
                                                           #0  0x00007f2780d09884 __pthread_kill_implementation (libc.so.6 + 0x8e884)
                                                           #1  0x00007f2780cb8afe raise (libc.so.6 + 0x3dafe)
                                                           #2  0x00007f2780ca187f abort (libc.so.6 + 0x2687f)
                                                           #3  0x00007f2780ca179b __assert_fail_base.cold (libc.so.6 + 0x2679b)
                                                           #4  0x00007f2780cb1187 __assert_fail (libc.so.6 + 0x36187)
                                                           #5  0x00007f2780c57323 krb5int_key_delete (libkrb5support.so.0 + 0x6323)
                                                           #6  0x00007f278092ae8b gssint_mechglue_fini (libgssapi_krb5.so.2 + 0xee8b)
                                                           #7  0x00007f278143e0f2 _dl_call_fini (ld-linux-x86-64.so.2 + 0x10f2)
                                                           #8  0x00007f2781441e5e _dl_fini (ld-linux-x86-64.so.2 + 0x4e5e)
                                                           #9  0x00007f2780cbb1e6 __run_exit_handlers (libc.so.6 + 0x401e6)
                                                           #10 0x00007f2780cbb32e exit (libc.so.6 + 0x4032e)
                                                           #11 0x000055f6ffa303bc main (ipa-getkeytab + 0x63bc)
                                                           #12 0x00007f2780ca2b8a __libc_start_call_main (libc.so.6 + 0x27b8a)
                                                           #13 0x00007f2780ca2c4b __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x27c4b)
                                                           #14 0x000055f6ffa31bb5 _start (ipa-getkeytab + 0x7bb5)
                                                           ELF object binary architecture: AMD x86-64
Sep 21 06:50:16 replica0.ipa.test systemd[1]: systemd-coredump@1-19900-0.service: Deactivated successfully.

The issue seems linked to the update of glibc to 2.37-6.f38.

Reproducer:
- on the first machine (the server): dnf install -y freeipa-server; ipa-server-install --domain ipa.test --realm IPA.TEST -a Secret123 -p Secret123 -U
- on the second machine (the replica): dnf install -y freeipa-server; dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-7f0a294b1a; ipa-replica-install --domain ipa.test --realm IPA.TEST --server server.ipa.test --principal admin --password Secret123 -U

Closing as fixed, glibc update to glibc-2.37-7.fc38 has resolved the issue as can be seen in simple_replication: logs, report

Metadata Update from @frenaud:
- Issue close_status updated to: fixed
- Issue status updated to: Closed (was: Open)
Metadata
None

test-failure tracker

None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.