freeipa

#8869 Nightly test failure in test_integration/test_ipa_cert_fix.py::TestCertFixReplica::test_renew_expired_cert_replica
Closed: duplicate by frenaud. Opened by sorlov.

Closed: duplicate
Reopen Issue

The test tries to resubmit all existing certificates using command getcert resubmit -i .... Resubmitting one of the certificates failed with error message

Please verify that the certmonger service is still running.

[testing_master_latest_selinux] Nightly PR 942

artifacts

report.html

Metadata Update from @sorlov:
- Issue tagged with: test-failure, tests

I don't know if this is the RCA or not but this test isn't following the ipa-cert-fix man page. It states that the shared certs need to be renewed first and then ipa-cert-fix is to be run on a replica.

This is generating an entire new set of CA certificates which is not expected nor desired.

The tracking goes from CA_WORKING (expected) to trying to pull in the newly generated certificates.

It doesn't explain why the dbus connection went away. certmonger is socket-activated so even if it dies it will restart on the next request., but there is no indication in the journal that anything bad happened at all.

So I think the test needs to be restructured and perhaps ipa-cert-fix needs to detect a newly renewed situation and refuse to renew them. Otherwise customers could end up renewing their CA certificates over and over on each server if they hop from machine to machine.

I fully believe that certmonger would have renewed all the certificates if given enough time.

Metadata Update from @rcritten:
- Custom field cc adjusted to myusuf@redhat.com

Failure seen in testing_master_previous

Failure seen at:

self = <ipatests.test_integration.test_ipa_cert_fix.TestCertFixReplica object at 0x7fdf2f5b00a0>
    def test_renew_expired_cert_replica(self):
        """Test renewal of certificates on replica with ipa-cert-fix
        This is to check that ipa-cert-fix renews the certificates
        on replica
        related: https://pagure.io/freeipa/issue/7885
        """
        move_date(self.master, 'stop', '+3years+1days')
        # wait for cert expiry
        check_status(self.master, 8, "CA_UNREACHABLE")
        self.master.run_command(['ipa-cert-fix', '-v'], stdin_text='yes\n')
        check_status(self.master, 9, "MONITORING")
        # move system date to expire cert on replica
        move_date(self.replicas[0], 'stop', '+3years+1days')
        # RA agent cert will be expired and in CA_UNREACHABLE state
        check_status(self.replicas[0], 1, "CA_UNREACHABLE")
        # renew RA agent cert
>       self.replicas[0].run_command(
            ['ipa-cert-fix', '-v'], stdin_text='yes\n'
        )

Error seen:

INFO: Restoring LDAP connection for CA
INFO: Storing subsystem config: /var/lib/pki/pki-tomcat/ca/conf/CS.cfg
INFO: Storing registry config: /var/lib/pki/pki-tomcat/ca/conf/registry.cfg
ipapython.admintool: DEBUG:   File "/usr/lib/python3.9/site-packages/ipapython/admintool.py", line 180, in execute
    return_value = self.run()
  File "/usr/lib/python3.9/site-packages/ipaserver/install/ipa_cert_fix.py", line 150, in run
    run_cert_fix(certs, extra_certs)
  File "/usr/lib/python3.9/site-packages/ipaserver/install/ipa_cert_fix.py", line 373, in run_cert_fix
    ipautil.run(cmd, raiseonerr=True)
  File "/usr/lib/python3.9/site-packages/ipapython/ipautil.py", line 598, in run
    raise CalledProcessError(
ipapython.admintool: DEBUG: The ipa-cert-fix command failed, exception: CalledProcessError: CalledProcessError(Command ['pki-server', 'cert-fix', '--ldapi-socket', '/run/slapd-IPA-TEST.socket', '--agent-uid', 'ipara', '--cert', 'sslserver', '--cert', 'ca_ocsp_signing', '--cert', 'ca_audit_signing', '--extra-cert', '7', '--extra-cert', '14'] returned non-zero exit status 1: "INFO: Loading instance: pki-tomcat\nINFO: Loading global Tomcat config: /etc/tomcat/tomcat.conf\nINFO: Loading PKI Tomcat config: /usr/share/pki/etc/tomcat.conf\nINFO: Loading instance Tomcat config: /etc/pki/pki-tomcat/tomcat.conf\nINFO: Loading password config: /etc/pki/pki-tomcat/password.conf\nINFO: Loading subsystem config: /var/lib/pki/pki-tomcat/ca/conf/CS.cfg\nINFO: Loading subsystem registry: /var/lib/pki/pki-tomcat/ca/conf/registry.cfg\nINFO: Loading instance registry: /etc/sysconfig/pki/tomcat/pki-tomcat/pki-tomcat\nINFO: Fixing the following system certs: ['sslserver', 'ca_ocsp_signing', 'ca_audit_signing']\nINFO: Renewing the following additional certs: ['7', '14']\nINFO: Stopping the instance to proceed with system cert renewal\nINFO: Configuring LDAP connection for CA\nINFO: Setting pkidbuser password via ldappasswd\nSASL/EXTERNAL authentication started\nSASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth\nSASL SSF: 0\nINFO: Storing subsystem config: /var/lib/pki/pki-tomcat/ca/conf/CS.cfg\nINFO: Storing registry config: /var/lib/pki/pki-tomcat/ca/conf/registry.cfg\nINFO: Storing subsystem config: /var/lib/pki/pki-tomcat/ca/conf/CS.cfg\nINFO: Storing registry config: /var/lib/pki/pki-tomcat/ca/conf/registry.cfg\nINFO: Selftests disabled for subsystems: ca\nSASL/EXTERNAL authentication started\nSASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth\nSASL SSF: 0\nINFO: Resetting password for uid=ipara,ou=people,o=ipaca\nSASL/EXTERNAL authentication started\nSASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth\nSASL SSF: 0\nINFO: Creating a temporary sslserver cert\nINFO: Getting sslserver cert info from CS.cfg\nINFO: Getting sslserver cert info from NSS database\nINFO: Trying to create a new temp cert for sslserver.\nINFO: Generate temp SSL certificate\nINFO: Getting sslserver cert info from CS.cfg\nINFO: Getting sslserver cert info from NSS database\nINFO: CSR for sslserver has been written to /tmp/tmp6kr60t2e/sslserver.csr\nINFO: Getting signing cert info from CS.cfg\nINFO: Getting signing cert info from NSS database\nINFO: CA cert written to /tmp/tmp6kr60t2e/ca_certificate.crt\nINFO: AKI: 0xB3E8FC4EA454A844210C156FC966D32C0964A607\nINFO: Temp cert for sslserver is available at /etc/pki/pki-tomcat/certs/sslserver.crt.\nINFO: Getting sslserver cert info from CS.cfg\nINFO: Getting sslserver cert info from NSS database\nINFO: Getting sslserver cert info from CS.cfg\nINFO: Getting sslserver cert info from NSS database\nINFO: Updating CS.cfg with the new certificate\nINFO: Storing subsystem config: /var/lib/pki/pki-tomcat/ca/conf/CS.cfg\nINFO: Storing registry config: /var/lib/pki/pki-tomcat/ca/conf/registry.cfg\nINFO: Starting the instance\nJob for pki-tomcatd@pki-tomcat.service canceled.\nINFO: Storing subsystem config: /var/lib/pki/pki-tomcat/ca/conf/CS.cfg\nINFO: Storing registry config: /var/lib/pki/pki-tomcat/ca/conf/registry.cfg\nINFO: Selftests enabled for subsystems: ca\nINFO: Restoring LDAP connection for CA\nINFO: Storing subsystem config: /var/lib/pki/pki-tomcat/ca/conf/CS.cfg\nINFO: Storing registry config: /var/lib/pki/pki-tomcat/ca/conf/registry.cfg\n")
ipapython.admintool: ERROR: CalledProcessError(Command ['pki-server', 'cert-fix', '--ldapi-socket', '/run/slapd-IPA-TEST.socket', '--agent-uid', 'ipara', '--cert', 'sslserver', '--cert', 'ca_ocsp_signing', '--cert', 'ca_audit_signing', '--extra-cert', '7', '--extra-cert', '14'] returned non-zero exit status 1: "INFO: Loading instance: pki-tomcat\nINFO: Loading global Tomcat config: /etc/tomcat/tomcat.conf\nINFO: Loading PKI Tomcat config: /usr/share/pki/etc/tomcat.conf\nINFO: Loading instance Tomcat config: /etc/pki/pki-tomcat/tomcat.conf\nINFO: Loading password config: /etc/pki/pki-tomcat/password.conf\nINFO: Loading subsystem config: /var/lib/pki/pki-tomcat/ca/conf/CS.cfg\nINFO: Loading subsystem registry: /var/lib/pki/pki-tomcat/ca/conf/registry.cfg\nINFO: Loading instance registry: /etc/sysconfig/pki/tomcat/pki-tomcat/pki-tomcat\nINFO: Fixing the following system certs: ['sslserver', 'ca_ocsp_signing', 'ca_audit_signing']\nINFO: Renewing the following additional certs: ['7', '14']\nINFO: Stopping the instance to proceed with system cert renewal\nINFO: Configuring LDAP connection for CA\nINFO: Setting pkidbuser password via ldappasswd\nSASL/EXTERNAL authentication started\nSASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth\nSASL SSF: 0\nINFO: Storing subsystem config: /var/lib/pki/pki-tomcat/ca/conf/CS.cfg\nINFO: Storing registry config: /var/lib/pki/pki-tomcat/ca/conf/registry.cfg\nINFO: Storing subsystem config: /var/lib/pki/pki-tomcat/ca/conf/CS.cfg\nINFO: Storing registry config: /var/lib/pki/pki-tomcat/ca/conf/registry.cfg\nINFO: Selftests disabled for subsystems: ca\nSASL/EXTERNAL authentication started\nSASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth\nSASL SSF: 0\nINFO: Resetting password for uid=ipara,ou=people,o=ipaca\nSASL/EXTERNAL authentication started\nSASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth\nSASL SSF: 0\nINFO: Creating a temporary sslserver cert\nINFO: Getting sslserver cert info from CS.cfg\nINFO: Getting sslserver cert info from NSS database\nINFO: Trying to create a new temp cert for sslserver.\nINFO: Generate temp SSL certificate\nINFO: Getting sslserver cert info from CS.cfg\nINFO: Getting sslserver cert info from NSS database\nINFO: CSR for sslserver has been written to /tmp/tmp6kr60t2e/sslserver.csr\nINFO: Getting signing cert info from CS.cfg\nINFO: Getting signing cert info from NSS database\nINFO: CA cert written to /tmp/tmp6kr60t2e/ca_certificate.crt\nINFO: AKI: 0xB3E8FC4EA454A844210C156FC966D32C0964A607\nINFO: Temp cert for sslserver is available at /etc/pki/pki-tomcat/certs/sslserver.crt.\nINFO: Getting sslserver cert info from CS.cfg\nINFO: Getting sslserver cert info from NSS database\nINFO: Getting sslserver cert info from CS.cfg\nINFO: Getting sslserver cert info from NSS database\nINFO: Updating CS.cfg with the new certificate\nINFO: Storing subsystem config: /var/lib/pki/pki-tomcat/ca/conf/CS.cfg\nINFO: Storing registry config: /var/lib/pki/pki-tomcat/ca/conf/registry.cfg\nINFO: Starting the instance\nJob for pki-tomcatd@pki-tomcat.service canceled.\nINFO: Storing subsystem config: /var/lib/pki/pki-tomcat/ca/conf/CS.cfg\nINFO: Storing registry config: /var/lib/pki/pki-tomcat/ca/conf/registry.cfg\nINFO: Selftests enabled for subsystems: ca\nINFO: Restoring LDAP connection for CA\nINFO: Storing subsystem config: /var/lib/pki/pki-tomcat/ca/conf/CS.cfg\nINFO: Storing registry config: /var/lib/pki/pki-tomcat/ca/conf/registry.cfg\n")
ipapython.admintool: ERROR: The ipa-cert-fix command failed.

PR 964
Logs
Edited by sumedhs

@sumedhs
the issue you saw is probably a different one, reported at https://pagure.io/freeipa/issue/8815

In the journal there is no trace of certmonger failing, and we can even see the last call to getcert (trying to resubmit 20210531205920 which is the subsystemCert):

Jun 01 21:17:55 replica0.ipa.test certmonger[25122]: 2024-06-01 21:17:55 [25122] Setting "CERTMONGER_REQ_SUBJECT" to "CN=CA Subsystem,O=IPA.TEST" for child.

The answer is not received by getcert, maybe simply because the system was overloaded. The call to getcert resubmit just before this one was extremely slow (more than 8seconds):

2021-05-31 21:17:40,332    DEBUG  [ipatests.pytest_ipa.integration.host.Host.replica0.cmd119] RUN ['getcert', 'resubmit', '-i', '20210531205915']
2021-05-31 21:17:52,694    DEBUG  [ipatests.pytest_ipa.integration.host.Host.replica0.cmd119] Resubmitting "20210531205915" to "dogtag-ipa-ca-renew-agent".

(Please ignore the date in the timestamps as they are taken on the controller, while the replica's date has been moved in the future).

I agree with Rob, there is nothing pointing in the direction of a certmonger issue. Since the test needs to be restructured (see https://pagure.io/freeipa/issue/8815), I think we can close this one and focus on 8815 instead.

Metadata Update from @frenaud:
- Issue close_status updated to: duplicate
- Issue status updated to: Closed (was: Open)
Metadata
None

test-failure tests

None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.