freeipa

#9463 Nightly test failure in test_replica_promotion.py::TestHiddenReplicaPromotion::test_ipahealthcheck_hidden_replica
Closed: fixed by frenaud. Opened by frenaud.

Closed: fixed
Reopen Issue

Issue

The nightly test test_replica_promotion.py::TestHiddenReplicaPromotion::test_ipahealthcheck_hidden_replica is failing when calling ipa-healthcheck, because it detects that ipa-ods-exporter service is not running.

Example of failing run: PR #3058 with the following logs and report:

self = <ipatests.test_integration.test_replica_promotion.TestHiddenReplicaPromotion object at 0x7f9c0e15e550>
    def test_ipahealthcheck_hidden_replica(self):
        """Ensure that ipa-healthcheck runs successfully on all members
        of an IPA cluster that includes a hidden replica.
        """
        os_version = (tasks.get_platform(self.master),
                      tasks.get_platform_version(self.master))
        pki_version = tasks.get_pki_version(self.master)
        # verify state
        self._check_config([self.master], [self.replicas[0]])
        # A DNA range is needed on the replica for ipa-healthcheck to work.
        # Create a user so that the replica gets a range.
        tasks.user_add(self.replicas[0], 'testuser')
        tasks.user_del(self.replicas[0], 'testuser')
        for srv in (self.master, self.replicas[0]):
            returncode, _unused = run_healthcheck(
                srv,
                failures_only=True
            )
            pki_too_old = \
                (os_version[0] == 'fedora'
                    and pki_version < tasks.parse_version('11.1.0'))\
                or (os_version[0] == 'rhel'
                    and os_version[1][0] == 8
                    and pki_version < tasks.parse_version('10.12.0'))\
                or (os_version[0] == 'rhel'
                    and os_version[1][0] == 9
                    and pki_version < tasks.parse_version('11.0.4'))
            with xfail_context(pki_too_old,
                               'https://pagure.io/freeipa/issue/8582'):
>               assert returncode == 0
E               assert 1 == 0

The output of ipa-healthcheck:

RUN ['ipa-healthcheck', '--output-type', 'json', '--failures-only']
Invalid IP address fe80::2f4b:11e:f987:f2e0 for master.ipa.test.: cannot use link-local IP address fe80::2f4b:11e:f987:f2e0
[
  {
    "source": "ipahealthcheck.meta.services",
    "check": "ipa_ods_exporter",
    "result": "ERROR",
    "uuid": "68ba7a76-6421-440c-8467-09a2dad6eaf5",
    "when": "20231010194227Z",
    "duration": "0.008166",
    "kw": {
      "status": false,
      "msg": "ipa-ods-exporter: not running"
    }
  }
]
Exit code: 1

The journal shows that the service exited:

Oct 10 19:36:34 master.ipa.test ipa-ods-exporter[15142]: ipa-ods-exporter: INFO     new replica keys in LDAP: set()
Oct 10 19:36:34 master.ipa.test ipa-ods-exporter[15142]: ipa-ods-exporter: INFO     obsolete replica keys in local HSM: set()
Oct 10 19:36:34 master.ipa.test ipa-ods-exporter[15142]: ipa-ods-exporter: INFO     ldap2master_replica: keys in local HSM & LDAP: {'0xb43a36af81059208e0972ebc8d427c1a', '0x8cc021ce10bc0e827123fc8f627306ec'}
Oct 10 19:36:34 master.ipa.test ipa-ods-exporter[15142]: ipa-ods-exporter: INFO     master2ldap_zone_keys: keys in local HSM & LDAP: set()
Oct 10 19:36:34 master.ipa.test ipa-ods-exporter[15142]: ipa-ods-exporter: INFO     master2ldap_zone_keys: keys in local HSM & LDAP: set()
Oct 10 19:36:34 master.ipa.test ipa-ods-exporter[15142]: ipa-ods-exporter: INFO     HSM synchronization finished, skipping zone synchronization.
Oct 10 19:36:35 master.ipa.test ipa-ods-exporter[15142]: Exception ignored in: <function LocalHSM.__del__ at 0x7f9c079672e0>
Oct 10 19:36:35 master.ipa.test ipa-ods-exporter[15142]: Traceback (most recent call last):
Oct 10 19:36:35 master.ipa.test ipa-ods-exporter[15142]:   File "/usr/lib/python3.11/site-packages/ipaserver/dnssec/localhsm.py", line 99, in __del__
Oct 10 19:36:35 master.ipa.test ipa-ods-exporter[15142]:   File "/usr/lib/python3.11/site-packages/ipaserver/p11helper.py", line 939, in finalize
Oct 10 19:36:35 master.ipa.test ipa-ods-exporter[15142]: TypeError: 'NoneType' object is not callable
Oct 10 19:36:35 master.ipa.test systemd[1]: ipa-ods-exporter.service: Deactivated successfully.
Oct 10 19:36:35 master.ipa.test audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=ipa-ods-exporter comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Oct 10 19:36:35 master.ipa.test systemd[1]: ipa-ods-exporter.service: Consumed 2.017s CPU time.

Test scenario:
1. install the server with ipa-server-install ... --setup-dns. At this point ipa-ods-exporter is inactive and ipa-healthcheck successful.
2. install the server as dnssec-master with ipa-dns-install --dnssec-master. ipa-healthcheck reports the issue
3. install the replica with ipa-replica-install ... --setup-dns --hidden-replica
4. run healthcheck on the server

The check for ipa-ods-exporter process has been added recently in ipa-healthcheck and the issue may have been present and unnoticed for a long time.

freeipa-healthcheck-0.16-2 release disables the ipa-ods-exporter check. This test error should not happen any more.

The test is now green, see the logs for the pipeline using master branch + fedora 39 in PR#3364.

Closing as fixed

Metadata Update from @frenaud:
- Issue close_status updated to: fixed
- Issue status updated to: Closed (was: Open)
Metadata
None

test-failure

None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.