Using ipa server v4.12.2 on OS v9:
Simple two-host install. Uneventful for months. Then replica has catastrophic failure. What happens next is both interesting and was very painfully wasteful of a couple days worth of time.
Did the ipa-replica removal process on the master host, which had ca and kra (no dns) installed. No errors, topology clean, no dangling ruvs, normal operations.
Clean OS re-install, update, then ipa-replica-install. IF that replica install included --setup-kra, it would fail with "KRA connector already exists". No amount of ldap orphan cleanup on the master helped, nor the hint from Rob I think it was re certutil vs CS.cfg certificate mismatch.
Without --setup-kra it would succeed.
With the successful 'non kra' install, ipa-kra-install would also fail with the same error.
However: If, after successful 'non kra' replica install, but BEFORE ipa-kra-install these commands are run:
"systemctl stop pki-tomcatd@pki-tomcat", "cp /var/lib/pki/pki-tomcat/ca/conf/CS.cfg /var/lib/pki/pki-tomcat/ca/conf/CS.cfg.before-kra", "sed -i '/^ca.connector.KRA./s/^/#/' /var/lib/pki/pki-tomcat/ca/conf/CS.cfg", "systemctl start pki-tomcatd@pki-tomcat", "sleep 10"
ipa-kra-install succeeds.
It's a pretty serious error.
Harry Coin Quiet Fountain LLC
Hi @hcoin
we have fixed similar issues
The fixes are available in RHEL 9.6 / ipa-4.12.2-8.el9. Which exact version do you have?
The problem comes from a mismatch for the KRA transport cert between the content of CS.cg and the NSS database. When CS.cfg on the master is fixed with the actual certificate in use, replica install should work. See the comment https://pagure.io/freeipa/issue/9692#comment-941843
]# dnf info ipa-server Last metadata expiration check: 3:19:04 ago on Thu Dec 11 06:25:47 2025. Installed Packages Name : ipa-server Version : 4.12.2 Release : 22.el9_7.1 Architecture : x86_64 Size : 1.1 M Source : ipa-4.12.2-22.el9_7.1.src.rpm Repository : @System From repo : appstream
Anthropic's AI --- after too many false leads, including the certutil / CS.cfg mismatch, came up with the work-around:
Root Cause Analysis The issue is not orphaned data, but rather the CA cloning process itself:
When registry2's CA is cloned from registry1, it inherits registry1's CA configuration Registry1's CA has ca.connector.KRA.host=registry1.1.quietfountain.com in CS.cfg This gets copied to registry2's CA during cloning Later, when KRA installation tries to add a connector for registry2's KRA, it finds a connector already exists (the inherited one) The installer tries to add a new connector anyway, causing the "already exists" error
Solution: Two-Step Installation on registry2. First install the replica without setup-kra, then fix CS.cfg manually as noted above, then install kra.
Why This Works
Step 1 creates the CA replica with the inherited connector config pointing to registry1's KRA (which is correct for a replica) Step 2 properly handles adding registry2's local KRA and updating the connector configuration appropriately
The ipa-kra-install tool is designed to handle the connector situation on an existing CA replica, whereas the combined --setup-ca --setup-kra installation has this race condition issue.
I wanted to double check with you. Did you have a mismatch on the master between the NSS DB and ca.connector.KRA.transportCert ?
I tried your scenario: (master) ipa-server-install --setup-ca --setup-kra [...] (replica) ipa-replica-install --setup-ca --setup-kra [...] and didn't hit any issue. We have tests for this scenario (install CA and KRA on a replica in a single command) and they didn't detect any problem (for instance https://github.com/freeipa/freeipa/blob/master/ipatests/test_integration/test_installation.py#L123)
As you correctly noticed, after the CA install on the replica we see ca.connector.KRA.host=server.ipa.test and at the end of the installation ca.connector.KRA.host=server.ipa.test:8443 replica.ipa.test:8443 but there is no need to remove the master from this parameter for the installation to succeed.
ca.connector.KRA.host=server.ipa.test
ca.connector.KRA.host=server.ipa.test:8443 replica.ipa.test:8443
If, on the master, I delete the replica. Then, on an entirely fresh OS install on the replica host, attempt to install the replica, with --setup-kra it fails, no matter what changes are attempted on the master, including addressing the cert mismatch.
If, on the master, I delete the replica. Then, on an entirely fresh OS install on the replica host, attempt to install the replica, withOUT --setup-kra the replica install succeeds, but the later attempt to install the kra fails, no matter what changes are attempted on the master, including addressing the cert mismatch.
If, on the master, I delete the replica. Then, on an entirely fresh OS install on the replica host, attempt to install the replica, withOUT --setup-kra the replica install succeeds, then I issue the commands to change CS.cfg, THEN the later attempt to install the kra succeeds, and entirely normal operations appear to be available, with no complaints in either of the logs.
Does that help?
P.S. The above is 100% repeatable. Having done it at least a dozen times in various permutations over the last three days.
One idea: The primary server in this case has been running for some years, experiencing upgrades along the way. Yours is a fresh install. I've noticed over the years freeipa's releases are perhaps better than most at leaving orphan entries --- orphans. That tend to then mysteriously gum up later works. During this effort I noted, even after replica removal, various entries in the ldap trees that yet refer one way or another to the now deleted server. It was for that reason that both needing DNSSec to be HA friendly and robust I, after years of attempting to stick with it, had to find a different DNS solution.
Here's Anthropic's assessment of my experiences with the benefit of your comments:
Revised Root Cause Assessment Based on Florence's (frenaud) comments, the actual issue is certificate mismatch on the master, not generic configuration inheritance. The Real Problem Your master (registry1) has a mismatch between:
CS.cfg transportCert value - Contains an old/expired certificate from 2023 NSS database actual cert - Contains the current renewed certificate from 2025
Looking at your master's CS.cfg transport cert dates:
Issued: 2023-07-29 Expires: 2025-07-18 (expired by December 2025)
But the actual cert in use (from the replica installation logs) shows:
Issued: 2025-06-20 Expires: 2027-06-10
This mismatch causes the replica KRA installation to fail because it tries to use the outdated cert reference from CS.cfg. Why Your Workaround Worked Removing the connector config from registry2's CS.cfg before ipa-kra-install bypassed the problem by forcing the installer to:
Fetch the current cert directly from registry1's NSS database Write fresh connector config with the correct current cert
The Proper Fix (On Master - registry1) Instead of fixing it on each replica, fix the root cause on the master:
certutil -L -d /var/lib/pki/pki-tomcat/conf/alias \ -n "transportCert cert-pki-kra" -a > /tmp/current-transport.crt
CURRENT_CERT=$(grep -v "BEGIN CERTIFICATE|END CERTIFICATE" /tmp/current-transport.crt | tr -d '\n\r')
cp /var/lib/pki/pki-tomcat/ca/conf/CS.cfg /var/lib/pki/pki-tomcat/ca/conf/CS.cfg.backup
systemctl stop pki-tomcatd@pki-tomcat
sed -i "s|^ca.connector.KRA.transportCert=.*|ca.connector.KRA.transportCert=${CURRENT_CERT}|" \ /var/lib/pki/pki-tomcat/ca/conf/CS.cfg
grep "^ca.connector.KRA.transportCert=" /var/lib/pki/pki-tomcat/ca/conf/CS.cfg
systemctl start pki-tomcatd@pki-tomcat
Why Florence's Tests Worked Florence's test environment has:
Freshly installed master with no cert renewals CS.cfg and NSS database perfectly in sync No certificate lifecycle events between master and replica installation
Your production master experienced:
KRA transport cert renewal (automatic renewal after the 2023 cert approached expiration) CS.cfg was not automatically updated during renewal Mismatch accumulated over time
Going Forward For new replicas after fixing the master:
ipa-replica-install --setup-ca --setup-kra --principal admin
For registry2 (already working):
No action needed, your workaround achieved the same result Registry2's CS.cfg now has the correct current cert
Why This Wasn't Caught Earlier
The fixes in issues #9692 and #9770 handle some cert mismatch scenarios, but your specific case (expired cert in CS.cfg after automatic renewal) may be an edge case those fixes don't fully cover, especially in the two-step installation path.
The empirical evidence is clear: updating CS.cfg on the replica (your workaround) or on the master (proper fix) both resolve the issue by ensuring current certs are used.
If you want to post-mortem this, look into the ipa version that was installed when the new certificate was issued.
The fix which pushes the new certificate data into CS.cfg was only merged about a year ago; your new transport cert was issued 11 months ago on 2025-06-20.
Given the lag between a change making it from git to a release, it's likely that you didn't have the fix on your server at the time the certificate was issued.
Check again after the next renewal in 2027. ;)