freeipa

#7613 PR-CI considers unexpected passes as success
Closed: fixed Opened by cheimes.

Closed: fixed
Reopen Issue

PR-CI doesn't mark a test run with unexpected passes as failed. For example test_installation.py::TestInstallWithCA_DNS1::()::test_replica0_ca_less_install from http://freeipa-org-pr-ci.s3-website.eu-central-1.amazonaws.com/jobs/9369fcca-7f18-11e8-aae8-fa163ef99e01/report.html is a XPassed.

Another example: http://freeipa-org-pr-ci.s3-website.eu-central-1.amazonaws.com/jobs/59805778-83d4-11e8-9987-fa163e430211/report.html for fedora-28/test_installation_TestInstallWithCA1 has 8 unexpected passes, e.g. test_installation.py::TestInstallWithCA1::()::test_replica2_ipa_dns_install.

FreeIPA's test suite is using xfail to mark tests as known failures. A failing test doesn't cause the test suite to fail. Instead it's recored as XFAIL (expected failure). It's also possible that an xfail test passes, Pytest calls this XPASS (unexpected pass). PR-CI doesn't consider an unexpected pass as an error.

With @pytest.mark.xfail(strict=True), pytest turns an XPASS into an error. See https://pytest.readthedocs.io/en/reorganize-docs/new-docs/user/xfail.html for more details

Metadata Update from @cheimes:
- Issue assigned to cheimes

Metadata Update from @cheimes:
- Custom field on_review adjusted to https://github.com/freeipa/freeipa/pull/2131

master:

  • 0c1010d6f7585095402f47c111b8bb6cde3068b3 Mark all expected failures as strict
  • ec65590c9f2e00d0e527e7bbf521a0b99997788c Fix XPASS in test_installation

Metadata Update from @cheimes:
- Issue close_status updated to: fixed
- Issue status updated to: Closed (was: Open)
Metadata

tests

None
None
None
https://github.com/freeipa/freeipa/pull/2131
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.