#9002 Nightly failure in test_fips.py::TestInstallFIPS::test_basic::setup
Closed: fixed by frenaud. Opened by frenaud.

The nightly test test_fips.py::TestInstallFIPS::test_basic is failing in rawhide, see PR #1195 with the following logs and report:

cls = <class 'ipatests.test_integration.test_fips.TestInstallFIPS'>
mh = <pytest_multihost.plugin.MultihostFixture object at 0x7f874ff29ff0>
    @classmethod
    def install(cls, mh):
>       super(TestInstallFIPS, cls).install(mh)
test_integration/test_fips.py:28: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
test_integration/base.py:83: in install
    cls.enable_fips_mode()
test_integration/base.py:65: in enable_fips_mode
    host.enable_userspace_fips()
pytest_ipa/integration/host.py:106: in enable_userspace_fips
    enable_userspace_fips(self)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
host = <ipatests.pytest_ipa.integration.host.Host master.ipa.test (master)>
    def enable_userspace_fips(host):
        # create /etc/system-fips
        host.put_file_contents(SYSTEM_FIPS, "# userspace fips\n")
        # fake Kernel FIPS mode with bind mount
        host.run_command(["mkdir", "-p", FIPS_OVERLAY_DIR])
        host.put_file_contents(FIPS_OVERLAY, "1\n")
        host.run_command(
            ["chcon", "-t", "sysctl_crypto_t", "-u", "system_u", FIPS_OVERLAY]
        )
        host.run_command(
            ["mount", "--bind", FIPS_OVERLAY, paths.PROC_FIPS_ENABLED]
        )
        # set crypto policy to FIPS mode
        host.run_command(["update-crypto-policies", "--show"])
        host.run_command(["update-crypto-policies", "--set", "FIPS"])
        # sanity check
        assert is_fips_enabled(host)
        result = host.run_command(
            ["openssl", "md5", "/dev/null"], raiseonerr=False
        )
>       assert result.returncode == 1
E       assert 0 == 1
E         +0
E         -1

Test scenario: the test is faking userspace fips mode with commands equivalent to the following:

echo -e "#userspace fips\n" > /etc/system-fips
mkdir -p /var/tmp/userspace-fips
echo -e "1\n" > /var/tmp/userspace-fips/fips-enabled
chcon -t sysctl_crypto_t -u system_u /var/tmp/userspace-fips/fips-enabled
mount --bind /var/tmp/userspace-fips/fips-enabled /proc/sys/crypto/fips_enabled
update-crypto-policies --set FIPS

and then it checks that md5 cannot be used with openssl md5 /dev/null, expecting a result code =1 (error).

Apparently this fake FIPS mode does not disable md5 on rawhide, with openssl-3.0.0-1.fc36.x86_64 (it used to work with openssl-1.1.1k-2.fc35.x86_64).


Metadata Update from @mpolovka:
- Issue assigned to mpolovka

test failure seen in testing_master_latest PR 1734 Report

In RHEL 9 we suggest trying a different approach to userspace FIPS mode emulation in OpenSSL

openssl.cnf like this should work

openssl_conf = openssl_init
# Comment out the next line to ignore configuration errors
config_diagnostics = 1
[openssl_init]
providers = provider_sect
alg_section = algorithm_sect
[provider_sect]
fips = fips_sect
base = base_sect
[base_sect]
activate = 1
[fips_sect]
activate = 1
[algorithm_sect]
default_properties = fips=yes

test failure seen in testing_master_latest PR 1743 Report

Reproducible in testing_master_latest PR 1752 [Report](http://freeipa-org-pr-ci.s3-website.eu-central-1.amazonaws.com/jobs/5891a444-e077-11ec-9a78-fa163ea04eff/report.html

Crypto team tells me that openssl 3.x in Fedora has no been yet updated to include FIPS patches on par with RHEL 9. This means we cannot use the same method for 'userspace FIPS' testing as it was with openssl 1.x and as works in RHEL 8/9. An alternative is what @dbelyavs provided -- it would need to be done on top of existing 'fake FIPS' setup. The latter is still needed for the other libraries (gnutls, etc).

An alternative is to run FIPS pipelines against CentOS 9 Stream which has the same openssl version as in RHEL. The downside is that we cannot test and guarantee Fedora in FIPS mode at all.

Reproducible in testing_master_latest PR 1760 Report

Reproducible in testing_master_latest PR 1768 Report

Reproducible in testing_master_latest PR 1778 Report

Reproducible in testing_master_latest PR 1786 Report

Could the issue be cause by a regression in Rawhide's OpenSSL build? RHEL's OpenSSL supports OPENSSL_FORCE_FIPS_MODE=1 env var and /proc/sys/crypto/fips_enabled file. Fedora seems to ignore both the proc file and the env var.

RHEL 9.0

# rpm -qa openssl
openssl-3.0.1-20.el9_0.x86_64
# OPENSSL_FORCE_FIPS_MODE=1 openssl md5 < /dev/null
Error setting digest
000C1650DC7F0000:error:0308010C:digital envelope routines:inner_evp_generic_fetch:unsupported:crypto/evp/evp_fetch.c:349:Global default library context, Algorithm (MD5 : 97), Properties ()
000C1650DC7F0000:error:03000086:digital envelope routines:evp_md_init_internal:initialization error:crypto/evp/digest.c:237:

Fake FIPS mode:

# echo 1 > /fips_enabled
# mount --bind /fips_enabled /proc/sys/crypto/fips_enabled
# cat /proc/sys/crypto/fips_enabled
1
# openssl md5 < /dev/null
Error setting digest
001C27E02E7F0000:error:0308010C:digital envelope routines:inner_evp_generic_fetch:unsupported:crypto/evp/evp_fetch.c:349:Global default library context, Algorithm (MD5 : 97), Properties ()
001C27E02E7F0000:error:03000086:digital envelope routines:evp_md_init_internal:initialization error:crypto/evp/digest.c:237

strace

# strace -P /proc/sys/crypto/fips_enabled openssl md5
openat(AT_FDCWD, "/proc/sys/crypto/fips_enabled", O_RDONLY) = 3
read(3, "1\n", 2)                       = 2
close(3)                                = 0
Error setting digest
...

Fedora Rawhide

# rpm -qa openssl
openssl-3.0.3-1.fc37.x86_64
# OPENSSL_FORCE_FIPS_MODE=1 openssl md5 < /dev/null
MD5(stdin)= d41d8cd98f00b204e9800998ecf8427e

Fake fips mode

# echo 1 > /fips_enabled
# mount --bind /fips_enabled /proc/sys/crypto/fips_enabled
# cat /proc/sys/crypto/fips_enabled 
1
# openssl md5 < /dev/null
MD5(stdin)= d41d8cd98f00b204e9800998ecf8427e
# strace -P /proc/sys/crypto/fips_enabled openssl md5 < /dev/null
MD5(stdin)= d41d8cd98f00b204e9800998ecf8427e
+++ exited with 0 +++

Unfortunately, Fedora currently does not support this mode at all. I recommend using the solution from https://pagure.io/freeipa/issue/9002#comment-799339

We have plans to sync Fedora FIPS patches with RHEL, but it will not happen until RHEL patches will be sort of finalized.

reproduced in testing_master_latest_selinux report

Reproducible in testing_master_latest PR 1813 Report

Reproducible in testing_master_pki report

Reproducible in testing_master_latest PR 1821 Report

Reproducible in testing_master_latest PR 1838 Report

Reproducible in testing_master_latest PR 1856 Report

Reproducible in testing_master_latest PR 1866 Report

Reproducible in testing_master_389ds PR 1891 Report

Reproducible in testing_master_latest PR 1887 Report

Reproducible in testing_master_latest PR Report

Reproducible in testing_master_389ds PR 1910 Report

Reproducible in `testing_master_latest PR 1906 Report

Reproducible in testing_master_latest PR 1916 Report

Reproducible in testing_master_latest PR 1927 Report

Reproducible in testing_master_latest PR 1937 Report

Reproducible in testing_master_pki PR 1937 Report

Reproducible in testing_master_latest PR 1965 Report

Reproducible in testing_master_latest PR 1975 Report

Reproducible in testing_master_latest PR 1985 Report

Reproducible in testing_master_latest PR 1995 Report

Reproducible in testing_master_latest PR 2014 Report

Reproducible in testing_master_latest PR 2024 Report

On rawhide, the failure is slightly different since the update of openssl to openssl-3.0.5-3:

        result = host.run_command(
            ["openssl", "md5", "/dev/null"], raiseonerr=False
        )
        assert result.returncode == 1
>       assert "EVP_DigestInit_ex:disabled for FIPS" in result.stderr_text
E       AssertionError: assert 'EVP_DigestInit_ex:disabled for FIPS' in 'Error setting digest\n0022EA66477F0000:error:0308010C:digital envelope routines:inner_evp_generic_fetch:unsupported:crypto/evp/evp_fetch.c:349:Global default library context, Algorithm (MD5 : 97), Properties ()\n0022EA66477F0000:error:03000086:digital envelope routines:evp_md_init_internal:initialization error:crypto/evp/digest.c:252:\n'
E        +  where 'Error setting digest\n0022EA66477F0000:error:0308010C:digital envelope routines:inner_evp_generic_fetch:unsupported:crypto/evp/evp_fetch.c:349:Global default library context, Algorithm (MD5 : 97), Properties ()\n0022EA66477F0000:error:03000086:digital envelope routines:evp_md_init_internal:initialization error:crypto/evp/digest.c:252:\n' = <pytest_multihost.transport.SSHCommand object at 0x7f5339df0290>.stderr_text

See PR #2026 with the following report.

This update of openssl brings the same set of patches as on RHEL.

The last error message is completely normal and matching the intended FIPS behavior.

Reproducible in testing_master_latest PR 2034 Report

Reproducible in testing_master_latest PR 2043 Report

Reproducible in testing_master_latest PR 2053 Report

master:

  • b8947b829bcd68c901b853ec7492a8c0f9376d48 ipatests: update the fake fips mode expected message
  • 6bd9d156e05c6dd0d4f9ece2aa3df34e77c58749 cert utilities: MAC verification is incompatible with FIPS mode
  • 17a5d5bff1df5e12899e9316f4a4364d2512a64f FIPS setup: fix typo filtering camellia encryption
  • be21cabad48395f48f123c5041c858608de52d38 Spec file: bump krb5_kdb_version on rawhide

ipa-4-9:

  • 1d01692cf241645ca59b7f3d3e2096ce738d6a05 ipatests: update the fake fips mode expected message
  • 42381ebd036feee63fab2bbf8579b7a385624bf7 cert utilities: MAC verification is incompatible with FIPS mode
  • f2a337caaf82fca4a8d7c347454b412ba2b4a0dd FIPS setup: fix typo filtering camellia encryption
  • f2b4d019881232833e915fedba48537548d2ef60 Spec file: bump krb5_kdb_version on rawhide

ipa-4-10:

  • 68f6574cb2bcf0b04840b4f62a8ac70b4d45cb1a ipatests: update the fake fips mode expected message
  • c853cfde56fb56798424bd402012d78ed47647c0 cert utilities: MAC verification is incompatible with FIPS mode
  • dfba6ebf9ab7b7d17e65f928c90ae63b31d9cae7 FIPS setup: fix typo filtering camellia encryption
  • 2904b15a94eebbb37ca6a289eccd6b95f063d7ca Spec file: bump krb5_kdb_version on rawhide

Metadata Update from @frenaud:
- Issue tagged with: fips

Metadata Update from @frenaud:
- Custom field on_review adjusted to https://github.com/freeipa/freeipa/pull/7684
- Issue assigned to frenaud (was: mpolovka)

master:

  • 50e8c4a1273dc5ba9dace14df8743821127b37fd ipatests: simulate FIPS mode and install replica

Metadata Update from @frenaud:
- Issue close_status updated to: fixed
- Issue status updated to: Closed (was: Open)

ipa-4-12:

  • 232169cc00e0d66bef1cc27997e45944d4d3cd2c ipatests: simulate FIPS mode and install replica
  • 599e208225d9a878af28e069d480dc4ee9d9bdea ipatests: test_fips: Remove obsolete patch
Metadata