#8793 [Tracker] Nightly failure (rawhide/f34) in test_dnssec.py::TestInstallDNSSECFirst::test_chain_of_trust
Closed: fixed by abbra. Opened by frenaud.

The nightly test test_dnssec.py::TestInstallDNSSECFirst::test_chain_of_trust fails on rawhide, see for instance PR #824 with the following logs and report:

self = <ipatests.test_integration.test_dnssec.TestInstallDNSSECFirst object at 0x7f9cf0226e50>
    def test_chain_of_trust(self):
        """
        Validate signed DNS records, using our own signed root zone
        :return:
        """
        dnszone_add_dnssec(self.master, example_test_zone)
        # delegation
        args = [
            "ipa", "dnsrecord-add", root_zone, example_test_zone,
            "--ns-rec=" + self.master.hostname
        ]
        self.master.run_command(args)
        # TODO: test require restart
        tasks.restart_named(self.master, self.replicas[0])
        # wait until zone is signed
        assert wait_until_record_is_signed(
            self.master.ip, example_test_zone, timeout=100
        ), "Zone %s is not signed (master)" % example_test_zone
        # wait until zone is signed
        assert wait_until_record_is_signed(
            self.replicas[0].ip, example_test_zone, timeout=200
        ), "Zone %s is not signed (replica)" % example_test_zone
        # GET DNSKEY records from zone
        ans = resolve_with_dnssec(self.master.ip, example_test_zone,
                                  rtype="DNSKEY")
        dnskey_rrset = ans.response.get_rrset(
            ans.response.answer,
            dns.name.from_text(example_test_zone),
            dns.rdataclass.IN,
            dns.rdatatype.DNSKEY)
        assert dnskey_rrset, "No DNSKEY records received"
        logger.debug("DNSKEY records returned: %s", dnskey_rrset.to_text())
        # generate DS records
        ds_records = []
        for key_rdata in dnskey_rrset:
            if key_rdata.flags != 257:
                continue  # it is not KSK
            ds_records.append(dns.dnssec.make_ds(example_test_zone, key_rdata,
                                                 'sha256'))
        assert ds_records, ("No KSK returned from the %s zone" %
                            example_test_zone)
        logger.debug("DS records for %s created: %r", example_test_zone,
                     ds_records)
        # add DS records to root zone
        args = [
            "ipa", "dnsrecord-add", root_zone, example_test_zone,
            # DS record requires to coexists with NS
            "--ns-rec", self.master.hostname,
        ]
        for ds in ds_records:
            args.append("--ds-rec")
            args.append(ds.to_text())
        self.master.run_command(args)
        # wait until DS records it replicated
        assert wait_until_record_is_signed(
            self.replicas[0].ip, example_test_zone, timeout=100,
            rtype="DS"
        ), "No DS record of '%s' returned from replica" % example_test_zone
        # extract DSKEY from root zone
        ans = resolve_with_dnssec(self.master.ip, root_zone,
                                  rtype="DNSKEY")
        dnskey_rrset = ans.response.get_rrset(ans.response.answer,
                                              dns.name.from_text(root_zone),
                                              dns.rdataclass.IN,
                                              dns.rdatatype.DNSKEY)
        assert dnskey_rrset, "No DNSKEY records received"
        logger.debug("DNSKEY records returned: %s", dnskey_rrset.to_text())
        # export trust keys for root zone
        root_key_rdatas = []
        for key_rdata in dnskey_rrset:
            if key_rdata.flags != 257:
                continue  # it is not KSK
            root_key_rdatas.append(key_rdata)
        assert root_key_rdatas, "No KSK returned from the root zone"
        root_keys_rrset = dns.rrset.from_rdata_list(dnskey_rrset.name,
                                                    dnskey_rrset.ttl,
                                                    root_key_rdatas)
        logger.debug("Root zone trusted key: %s", root_keys_rrset.to_text())
        # set trusted key for our root zone
        self.master.put_file_contents(paths.DNSSEC_TRUSTED_KEY,
                                      root_keys_rrset.to_text() + '\n')
        self.replicas[0].put_file_contents(paths.DNSSEC_TRUSTED_KEY,
                                           root_keys_rrset.to_text() + '\n')
        # verify signatures
        time.sleep(DNSSEC_SLEEP)
        args = [
            "drill", "@localhost", "-k",
            paths.DNSSEC_TRUSTED_KEY, "-S",
            example_test_zone, "SOA"
        ]
        # test if signature chains are valid
>       self.master.run_command(args)

The test scenario is the following:
- configure master as DNSSEC master
- add a zone example.test. with dnssec enabled
- add NS record and DS record on the zone "." for example.test
- wait until the DS record is signed and replicated
- get the DNSKEY and put it into /etc/trusted.key on master and replica
- call drill on the master to validate the chain of trust

The drill output is the following:

RUN ['drill', '@localhost', '-k', '/etc/trusted-key.key', '-S', 'example.test.', 'SOA']
;; Number of trusted keys: 1
;; Chasing: example.test. SOA
DNSSEC Trust tree:
example.test. (SOA)
|---example.test. (DNSKEY keytag: 26662 alg: 8 flags: 257)
No trusted keys found in tree: first error was: No DNSSEC public key(s)
;; Chase failed.
Exit code: 29

Metadata Update from @frenaud:
- Issue priority set to: important

Metadata Update from @frenaud:
- Issue set to the milestone: DNSSEC

it is also failing on Fedora 34: http://freeipa-org-pr-ci.s3-website.eu-central-1.amazonaws.com/jobs/c5011154-b289-11eb-bcce-fa163e67bee2/

Metadata Update from @slev:
- Issue assigned to slev

I faced this issue during migration Azure F32->F34.
I made debugging and there is a mix of issues.

Failed in [testing_master_latest]

PR #913

report

The first part of issue was triggered by GCC11(F34 switched to).
The ldns code breaks strict aliasing and should be built with -fno-strict-aliasing.
I hope upstream reacts, otherwise Fedora packaging bug should be open against ldns(I will open this week).

Upstream ticket:
https://github.com/NLnetLabs/ldns/issues/131

Downstream ticket:
https://bugzilla.redhat.com/show_bug.cgi?id=1962010

@slev
thanks for investigating. Adding the tracker label to display the dependency on another component.

master:

  • 6fe8620668c1bf94bab0f908560af1a1a9a8aa31 azure: bump F32->F34
  • a8fd65be655469f79ba508f10d81a48c495386c4 freeipa.spec: do not use jsl for linting on Fedora 34+
  • be2f659aa7f746446aa521d4a2bf996e00596181 azure: Collect systemd boot log
  • 09a4918f03b1ed3782d7f2430d661e17564f44cc azure: Enforce multi-user.target as default systemd's target
  • 1c82895c20aeaf963b456f60789c2638d6bc6c11 azure: Wait for systemd booted
  • 7ed21f158cf700e8aeacf54835163552ef3f27eb azure: Remove no longer needed repo
  • 06d7c7f1d3f5f9dd64e9739c60ed92a9bd52bbc8 azure: Mask systemd-resolved
  • 1aff24e8913b4ce876abefa5de557117057ffdcb ipatests: Update expectations for test_detect_container
  • 9148ca2e1ece4298ee399fecd5409c45d49bd8d0 azure: Add workaround for PhantomJS against OpenSSL 1.1.1
  • b5fdba7a72009c52d4e2ffaf0dfe51afa66dc420 azure: Warn about memory issues
  • 07c423afd51b9664733e941486cf13d039bd2e9c BIND: Setup logging
  • 65700bf7436b82ad83d9d69d00eaa789f7a9c1ad ipatests: Setup and collect BIND logs
  • fc0c6b44a807eb4e2cee1a6f68ba9e283372ec88 azure: Run Base and XMLRPC tests is isolated network
  • cc72a988d6c1d483bd8f643397963ce4ebcfa406 ipatests: Handle network-isolated mode
  • 4709aefb83e1bf396b2fd919c8c2ebb77d6df159 dnsutil: Improvements for IPA DNS Resolver
  • 269c61a9a63321182e1b209bd506286f7b417038 dns: get_reverse_zone: Ignore resolver's timeout
  • 3889d8654af56d2f0bf9e249ce4b6bc711a73b6b pytest: Show extra summary information for all except passed tests
  • afef09ccbaa19f63cb8d498b37d7431abda8201f ipatests: Ignore warnings on failed to read files on tarring
  • 4a4c078b806a31e313d4dc02f7d0bf1ceb4011a4 ipatests: Suppress list trust or certificates
  • 611b49e42b0546f142707972e1290ee68f360fda azure: Collect installed packages
  • 26ee44bcfd5d5ddaca226d5c1bf73729bfcf0dd9 ipatests: dnssec: Add alternative approach for checking chain of trust
  • a893852b4f6178c14260593bf3b7f8cc344b41ce azure: Warn about extra and missing gating tests compared to PR-CI
  • 01553572d4a775bc10b4205e3d6d9f3715dd8e1b azure: Re-balance tests envs
  • 6c2db326f804f2268883be2714ad258c51536ea8 azure: coredump: Wait for systemd fully booted
  • 10461b7091678e4950b57c2a6ef7fcef89792ae7 azure: Make it possible to adjust Docker resources per test env

ipa-4-9:

  • b9fd47a7ae58085be49bcf6772d688779b492748 azure: bump F32->F34
  • 18563bc87b07fd7aca6f9d3af534fdc2c42bb8aa freeipa.spec: do not use jsl for linting on Fedora 34+
  • c711292bcf5a0cb2ce02cd470be6a37b67200b25 azure: Collect systemd boot log
  • c26907bc020afc4d54826606f2c00889a15bf06f azure: Enforce multi-user.target as default systemd's target
  • eb0a5db3043ead31194143926a56923dec593349 azure: Wait for systemd booted
  • e243b956f4fb04adbbee80a8ffdaca92f9eabf7c azure: Remove no longer needed repo
  • 4d53d9fdf2339307033a14e2214bfb98923f4e9d azure: Mask systemd-resolved
  • c90a3636a4bb1b424f916a4498dd8aeef5d37dde ipatests: Update expectations for test_detect_container
  • aa0c8c832350cad0de1bfa99e7b23f8c6cadb4de azure: Add workaround for PhantomJS against OpenSSL 1.1.1
  • 6164bfb56ad6d3acce9ea778cebf6694a15dd3d8 azure: Warn about memory issues
  • 0932c9217ff343d788b13ee1845fecb9666cbea8 BIND: Setup logging
  • 64c0f900306295d1e9832ed58844bb9b2338a339 ipatests: Setup and collect BIND logs
  • 5501fda5617739d1514e09eb0edebbda1fcf77b1 azure: Run Base and XMLRPC tests is isolated network
  • a192c21b2ce7e6167aba4132ed9504a46aeebf4e ipatests: Handle network-isolated mode
  • b487629262671714b5e36c74c8dbc193b008ce7d dnsutil: Improvements for IPA DNS Resolver
  • 9e1531180336083fe1b873045a06647ae9074633 dns: get_reverse_zone: Ignore resolver's timeout
  • 645f90a835d8ba8080ffdd9a7a3f98f40a3ac550 pytest: Show extra summary information for all except passed tests
  • 535131d633f80f3dd07206df9082b200b1d257c5 ipatests: Ignore warnings on failed to read files on tarring
  • c92f10029b08b31969d1e1da9dfe6f4b6c1b7dff ipatests: Suppress list trust or certificates
  • 3049b9587f0ded768ba3a61ec0e98368a6316a0a azure: Collect installed packages
  • 3ada2d983fa136016c94cf827f3c3ef1d65a8323 ipatests: dnssec: Add alternative approach for checking chain of trust
  • 0dd0631b21ce768d5624128983e7eb1c953297a3 azure: Warn about extra and missing gating tests compared to PR-CI
  • d4d27947a80c8b530ae8b4b962543929db4a8999 azure: Re-balance tests envs
  • 692f42dc7d887d3f89a0915529aed4d5380079ea azure: coredump: Wait for systemd fully booted
  • 391ca8b90b6610b16023602a30b451bdcb5a1e6f azure: Make it possible to adjust Docker resources per test env

Metadata Update from @abbra:
- Issue close_status updated to: fixed
- Issue status updated to: Closed (was: Open)

Reopening as test_chain_of_trust_drill introduced in 26ee44b should be marked as xfail

Metadata Update from @mpolovka:
- Issue status updated to: Open (was: Closed)

ipa-4-8:

  • c02544c07d040e4e96ed17233b479c958a68b8fa azure: bump F32->F34
  • 7802e14f5e209512fc736af56972f572bc599bec freeipa.spec: do not use jsl for linting on Fedora 34+
  • 7433be926c6e084f7f1f3bead060544a1334c86e azure: Collect systemd boot log
  • 523a9f863815cccfa05f0f87269d6b448a5ab12a azure: Enforce multi-user.target as default systemd's target
  • 677df148c7d606242ae14665e3d0580f12e972b3 azure: Wait for systemd booted
  • 04c90fb4f2fb16e932f5d198cacb31c75dc9b955 azure: Remove no longer needed repo
  • 8fea2f6f3573cd18f6d1cf2af5f95f0f5da2a643 azure: Mask systemd-resolved
  • 976a3bf4c388e71c3dc85c8c50b9c727bc07de65 ipatests: Update expectations for test_detect_container
  • e5731634a17aa96f4efa235f12bcca993726e6fc azure: Add workaround for PhantomJS against OpenSSL 1.1.1
  • 01237953a21b80219ebd6665e0eb67558a951a83 azure: Warn about memory issues
  • 835df314d65f42c2e36ac52737eecc1f4e9536e6 BIND: Setup logging
  • 2a9dea81254accb08a114a0e853a4c485d1082e4 ipatests: Setup and collect BIND logs
  • e23f9767ce2d509c00efa3c0fa5684e7a1fe1973 azure: Run Base and XMLRPC tests is isolated network
  • 34e1f6ab3971468a0fa485fe7588e469632d3d0f ipatests: Handle network-isolated mode
  • c8e5867d01e4a3a75f7abebf08caf29bbeae5eb6 dnsutil: Improvements for IPA DNS Resolver
  • fe0b5ff449da473a69449f7c3267e4a17245a753 dns: get_reverse_zone: Ignore resolver's timeout
  • d40306b90527337e504d019797ffb41b7c13d035 pytest: Show extra summary information for all except passed tests
  • ff70aacbf8f9035ec496c5e8ea8b23ac6724a8c6 ipatests: Ignore warnings on failed to read files on tarring
  • cb3b396fbbbf11462597a7289d5a89cb9fe2cbc9 ipatests: Suppress list trust or certificates
  • 21a520181f0e8c2a1e19b3a9e559fd77d9df4b04 azure: Collect installed packages
  • c65c7eb05779425355266486c02a30d9cc1f9659 ipatests: dnssec: Add alternative approach for checking chain of trust
  • 6710ff42ea853a5dcee96ab64a72051fb4e3f6f7 azure: Warn about extra and missing gating tests compared to PR-CI
  • a5730f5053a706c9ce93ef7ec4c52f44b32ca432 azure: Re-balance tests envs
  • e66eb48ac43ec513b1e787dcb0da3ee7f7ea3908 azure: coredump: Wait for systemd fully booted
  • 6561fc689800c390b8a05cec01e3e69e563edebb ipatests: re-add test_dnssec.py::TestInstallDNSSECFirst in gating
  • 8bf95380f1e3608be993e90c0005742db1cb9090 azure: Make it possible to adjust Docker resources per test env
  • 2a7f21a9c308cba33edf9ffa43c4aaf271e3a612 ipa-kdb: fix gcc complaints in kdb tests
  • e94261f9d9a09dbeca581dfe77f0ae94d2cb1c0b Set client keytab location for 389ds
  • ba6eb857ee7abf0fdead07a86e60b0308935ed38 dnssec: fix the key type with OpenDNSSEC 2.1
  • 7daf47c83a11ee9cf63462bef8c34a4899054623 ipatests: add a test for ZSK/KSK keytype in DNSKEY record
  • b8242e64ee7765d17746281192eaefc4f8fd7dcc handle Y2038 in timestamp to datetime conversions
  • 5bfe16a8182f2a2903d12ddd93c9d86a20cae198 OpenDNSSEC: fix timezone in key creation date
  • 56746ec0055803b8f6c73c382bed1132a950c0df freeipa.spec: bump the required version of 389ds
  • 2b8ccc8a1ec00fe2b4054f169233092c12d78522 freeipa.spec: synchronize with Fedora for 389-ds and PKI versions
  • a8686043a521ab115e21700c7782f3fad8ee9752 ipatests: collect config files for NetworkManager and systemd-resolved
  • bc9ca47fbd006a533a5f520fe303913b56b13712 ipatests: add utility for managing domain name resolvers
  • cdc78af9d77e22b27fc13d3a82e511f7693dff89 ipatests: setup resolvers during replica and client installations
  • 549ef48c4bf749919fcd93a7ee71641c180b1687 ipatests: do not manually modify /etc/resolv.conf in tests
  • 324ba203ebdd5e8ca5837d40a549c6f43e214d13 ipatests: disable systemd-resolved cache
  • 9a28022e4792a905137da9b8a463057473a86499 ipatests: mock resolver factory
  • 63a3cfff967eadc8b8ca07ab82ea47d876cc8491 ipatests: always try to create A records for hosts in IPA domain
  • d9744e7fa3a9ada6af44b993047fde6521f9bac3 ipatests: do not configure nameserver when installing client and replica
  • 47e9df18cd85295e02dd96c95d5888a84aaceaae ipatests: fix TestInstalDNSSECFirst::test_resolvconf logic
  • bca86ced8dd4fc36685137b32361e311ba34c04f pr-ci: Run tests on F34
  • 0b8517d6f1f09ce246ef2e6c5305099ceec8ac48 Revert "ipatests: configure client to use IPA server as DNS resolver"
  • d43d9ca8afa20c83010a3e862e806936414631fc ipatests: Fetch sudo rules without time offset

Metadata Update from @abbra:
- Issue close_status updated to: fixed
- Issue status updated to: Closed (was: Open)

@mpolovka, it was decided to not bump the required version of ldns, but update the test image.

https://github.com/freeipa/freeipa/pull/5800

Metadata