freeipa

#9926 different database generation ID than the local database
Closed: invalid by abbra. Opened by feiliang.

Closed: invalid
Reopen Issue

Issue

[desI noticed that some nodes in my freeipa cluster were having replication issues. I used AI to execute some repair commands for me, but the terrifying thing was that all the CA nodes were completely cleared.
The other non-CA nodes did fix the replication problem. I exported the 389ds data from the non-CA nodes and imported it to the CA for successful import and normal startup. However, I found that their topology segments and replication could not be unified.
On the web interface, I could see the topology segment information, but it was not available in the commands. This has caused the inability to replicate data at present. How can I fix this problem? Thank you.

Steps to Reproduce

1: hostname:ipa-65-189.hiido.host.xx.com Export full 389ds slapd data

db2ldif -D /etc/dirsrv/slapd-YYDEVOPS-COM -n userRoot -a /tmp/userRoot.ldif

2: fs-hiido-kerberos-ca-02.hiido.host.xx.com Import ipa-65-189.hiido.host.xx.com full data

stop-dirsrv
rm -rf /var/lib/dirsrv/slapd-YYDEVOPS-COM/db/*
rm -rf /var/lib/dirsrv/slapd-YYDEVOPS-COM/changelog/*
ldif2db -D /etc/dirsrv/slapd-YYDEVOPS-COM -n userRoot -i /tmp/userRoot.ldif 
start-dirsrv

n)

Version/Release/Distribution

ubuntu16.04
freeipa 4.3

ns-slapd -v
389 Project
389-Directory/1.3.4.9 B2016.109.158

Additional info:

hostname:ipa-65-189.hiido.host.xx.com
tailf dirsrv/slapd-YYDEVOPS-COM/error

[26/Jan/2026:09:56:52 +0800] NSMMReplicationPlugin - agmt="cn=ipa-65-189.hiido.host.xx.com-to-fs-hiido-kerberos-ca-02.hiido.host.xx.com" (fs-hiido-kerberos-ca-02:389): The remote replica has a different database generation ID than the local database.  You may have to reinitialize the remote replica, or the local replica.
[26/Jan/2026:09:56:55 +0800] NSMMReplicationPlugin - agmt="cn=ipa-65-189.hiido.host.xx.com-to-fs-hiido-kerberos-ca-02.hiido.host.xx.com" (fs-hiido-kerberos-ca-02:389): The remote replica has a different database generation ID than the local database.  You may have to reinitialize the remote replica, or the local replica.
[26/Jan/2026:09:56:59 +0800] NSMMReplicationPlugin - agmt="cn=ipa-65-189.hiido.host.xx.com-to-fs-hiido-kerberos-ca-02.hiido.host.xx.com" (fs-hiido-kerberos-ca-02:389): The remote replica has a different database generation ID than the local database.  You may have to reinitialize the remote replica, or the local replica.
[26/Jan/2026:09:57:01 +0800] NSMMReplicationPlugin - agmt="cn=meTofs-hiido-kerberos-21-117-149.hiido.host.xx.com" (fs-hiido-kerberos-21-117-149:389): The remote replica has a different database generation ID than the local database.  You may have to reinitialize the remote replica, or the local replica.

root@ipa-65-189:/var/log/dirsrv/slapd-YYDEVOPS-COM# ipa-replica-manage list

fs-hiido-kerberos-server02.hiido.host.xx.com: master
ipa-65-189.hiido.host.xx.com: master
fs-hiido-ipa-65-155.hiido.host.xx.com: master
fs-hiido-kerberos-server03.hiido.host.xx.com: master
fs-hiido-kerberos-21-117-149.hiido.host.xx.com: master
fs-hiido-kerberos-ca-02.hiido.host.xx.com: master
fs-hiido-kerveros-test08.hiido.host.xx.com: master

root@ipa-65-189:/var/log/dirsrv/slapd-YYDEVOPS-COM# ipa-replica-manage re-initialize --from fs-hiido-kerberos-ca-02.hiido.host.xx.com

'ipa-65-189.hiido.host.xx.com' has no replication agreement for 'fs-hiido-kerberos-ca-02.hiido.host.xx.com'

root@ipa-65-189:/var/log/dirsrv/slapd-YYDEVOPS-COM# ipa topologysegment-find domain

------------------
0 segments matched
------------------
----------------------------
Number of entries returned 0
----------------------------

root@ipa-65-189:/var/log/dirsrv/slapd-YYDEVOPS-COM# ipa-replica-manage re-initialize --from fs-hiido-kerberos-ca-02.hiido.host.xx.com

'ipa-65-189.hiido.host.xx.com' has no replication agreement for 'fs-hiido-kerberos-ca-02.hiido.host.xx.com'
root@ipa-65-189:/var/log/dirsrv/slapd-YYDEVOPS-COM# 
root@ipa-65-189:/var/log/dirsrv/slapd-YYDEVOPS-COM# ipa topologysegment-add domain fs-hiido-kerberos-ca-02.hiido.host.xx.com-to-ipa-65-189.hiido.host.xx.com \
>   --leftnode=fs-hiido-kerberos-ca-02.hiido.host.xx.com \
>  --rightnode=ipa-65-189.hiido.host.xx.com
ipa: ERROR: Server is unwilling to perform: Segment already exists in topology. Add rejected.

root@ipa-65-189:/var/log/dirsrv/slapd-YYDEVOPS-COM# ipactl status

Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: STOPPED
ipa_memcached Service: RUNNING
httpd Service: RUNNING
ipa-custodia Service: RUNNING
ipa-otpd Service: RUNNING
ipa: INFO: The ipactl command was successful

root@ipa-65-189:/var/log/dirsrv/slapd-YYDEVOPS-COM# ldapsearch -LLL -x -D "cn=Directory Manager" -W \

-b "cn=topology,cn=ipa,cn=etc,dc=xx,dc=com" \
"(objectClass=ipaTopologySegment)" dn ipaTopologySegmentLeftNode ipaTopologySegmentRightNode
Enter LDAP Password:
null

root@ipa-65-189:/var/log/dirsrv/slapd-YYDEVOPS-COM# ipa-replica-manage re-initialize --from fs-hiido-kerberos-21-117-149.hiido.host.xx.com

'ipa-65-189.hiido.host.xx.com' has no replication agreement for 'fs-hiido-kerberos-21-117-149.hiido.host.xx.com'
root@ipa-65-189:/var/log/dirsrv/slapd-YYDEVOPS-COM#

hostname:fs-hiido-kerberos-ca-02

root@fs-hiido-kerberos-ca-02:/var/log/dirsrv/slapd-YYDEVOPS-COM# tailf errors | grep 189
[26/Jan/2026:10:01:22 +0800] NSMMReplicationPlugin - agmt="cn=fs-hiido-kerberos-ca-02.hiido.host.xx.com-to-ipa-65-189.hiido.host.xx.com" (ipa-65-189:389): The remote replica has a different database generation ID than the local database.  You may have to reinitialize the remote replica, or the local replica.
[26/Jan/2026:10:01:25 +0800] NSMMReplicationPlugin - agmt="cn=fs-hiido-kerberos-ca-02.hiido.host.xx.com-to-ipa-65-189.hiido.host.xx.com" (ipa-65-189:389): The remote replica has a different database generation ID than the local database.  You may have to reinitialize the remote replica, or the local replica.
[26/Jan/2026:10:01:29 +0800] NSMMReplicationPlugin - agmt="cn=fs-hiido-kerberos-ca-02.hiido.host.xx.com-to-ipa-65-189.hiido.host.xx.com" (ipa-65-189:389): The remote replica has a different database generation ID than the local database.  You may have to reinitialize the remote replica, or the local replica.

[infoflow_2026-1-26_10-18-43-topo.jpg]

389 ds ldapsearch Directory Manager info

root@ipa-65-189:/var/log/dirsrv/slapd-YYDEVOPS-COM# ldapsearch -LLL -x -D "cn=Directory Manager" -W -b "cn=ipa-65-189.hiido.host.xx.com-to-fs-hiido-kerberos-ca-02.hiido.host.xx.com,cn=replica,cn=dc\3Dxx\2Cdc\3Dcom,cn=mapping tree,cn=config" "(objectClass=*)"
Enter LDAP Password: 

dn: cn=ipa-65-189.hiido.host.xx.com-to-fs-hiido-kerberos-ca-02.hiido.hos
 t.xx.com,cn=replica,cn=dc\3Dxx\2Cdc\3Dcom,cn=mapping tree,cn=conf
 ig
objectClass: nsds5replicationagreement
objectClass: ipaReplTopoManagedAgreement
objectClass: top
cn: ipa-65-189.hiido.host.xx.com-to-fs-hiido-kerberos-ca-02.hiido.host.y
 ydevops.com
nsDS5ReplicaHost: fs-hiido-kerberos-ca-02.hiido.host.xx.com
nsDS5ReplicaPort: 389
nsds5replicaTimeout: 300
nsDS5ReplicaRoot: dc=xx,dc=com
description: ipa-65-189.hiido.host.xx.com to fs-hiido-kerberos-ca-02.hii
 do.host.xx.com
ipaReplTopoManagedAgreementState: managed agreement - generated by topology pl
 ugin
nsDS5ReplicaTransportInfo: LDAP
nsDS5ReplicaBindMethod: SASL/GSSAPI
nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial
  entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount
nsds5ReplicaStripAttrs: modifiersName modifyTimestamp internalModifiersName in
 ternalModifyTimestamp
nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblasts
 uccessfulauth krblastfailedauth krbloginfailedcount
nsds50ruv: {replicageneration} 69734f310000003f0000
nsds50ruv: {replica 63 ldap://fs-hiido-kerberos-ca-02.hiido.host.xx.com:
 389} 69734f330000003f0000 6973530e0002003f0000
nsds50ruv: {replica 62 ldap://ipa-65-189.hiido.host.xx.com:389}
nsds50ruv: {replica 4 ldap://fs-hiido-kerberos-21-117-149.hiido.host.xx.
 com:389}
nsds50ruv: {replica 60 ldap://fs-hiido-ipa-65-155.hiido.host.xx.com:389}
nsruvReplicaLastModified: {replica 63 ldap://fs-hiido-kerberos-ca-02.hiido.hos
 t.xx.com:389} 00000000
nsruvReplicaLastModified: {replica 62 ldap://ipa-65-189.hiido.host.xx.co
 m:389} 00000000
nsruvReplicaLastModified: {replica 4 ldap://fs-hiido-kerberos-21-117-149.hiido
 .host.xx.com:389} 00000000
nsruvReplicaLastModified: {replica 60 ldap://fs-hiido-ipa-65-155.hiido.host.yy
 devops.com:389} 00000000
nsds5ReplicaEnabled: on
nsds5replicareapactive: 0
nsds5replicaLastUpdateStart: 19700101000000Z
nsds5replicaLastUpdateEnd: 19700101000000Z
nsds5replicaChangesSentSinceStartup:
nsds5replicaLastUpdateStatus: 402 Replication error acquiring replica: unknown
  error - Replica has different database generation ID, remote replica may nee
 d to be initialized
nsds5replicaUpdateInProgress: FALSE
nsds5replicaLastInitStart: 19700101000000Z
nsds5replicaLastInitEnd: 19700101000000Z
  • ca-02

root@fs-hiido-kerberos-ca-02:/var/log/dirsrv/slapd-YYDEVOPS-COM# ldapsearch -LLL -x -D "cn=Directory Manager" -W -b "cn=fs-hiido-kerberos-ca-02.hiido.host.xx.com-to-ipa-65-189.hiido.host.xx.com,cn=replica,cn=dc\3Dxx\2Cdc\3Dcom,cn=mapping tree,cn=config" "(objectClass=*)"
Enter LDAP Password: 

dn: cn=fs-hiido-kerberos-ca-02.hiido.host.xx.com-to-ipa-65-189.hiido.hos
 t.xx.com,cn=replica,cn=dc\3Dxx\2Cdc\3Dcom,cn=mapping tree,cn=conf
 ig
objectClass: nsds5replicationagreement
objectClass: ipaReplTopoManagedAgreement
objectClass: top
cn: fs-hiido-kerberos-ca-02.hiido.host.xx.com-to-ipa-65-189.hiido.host.y
 ydevops.com
nsDS5ReplicaHost: ipa-65-189.hiido.host.xx.com
nsDS5ReplicaPort: 389
nsds5replicaTimeout: 300
nsDS5ReplicaRoot: dc=xx,dc=com
description: fs-hiido-kerberos-ca-02.hiido.host.xx.com to ipa-65-189.hii
 do.host.xx.com
ipaReplTopoManagedAgreementState: managed agreement - generated by topology pl
 ugin
nsDS5ReplicaTransportInfo: LDAP
nsDS5ReplicaBindMethod: SASL/GSSAPI
nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial
  entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount
nsds5ReplicaStripAttrs: modifiersName modifyTimestamp internalModifiersName in
 ternalModifyTimestamp
nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblasts
 uccessfulauth krblastfailedauth krbloginfailedcount
nsds50ruv: {replicageneration} 5d778f4c000000040000
nsds50ruv: {replica 62 ldap://ipa-65-189.hiido.host.xx.com:389} 631096ad
 0001003e0000 697367ed0003003e0000
nsds50ruv: {replica 4 ldap://fs-hiido-kerberos-21-117-149.hiido.host.xx.
 com:389} 5d778f4c000100040000 69731625000800040000
nsds50ruv: {replica 48 ldap://fs-hiido-kerberos-server03.hiido.host.xx.c
 om:389} 5ecf811a000000300000 5ed0be57001300300000
nsds50ruv: {replica 7} 5e732d1c000200070000 5e732d1c000200070000
nsds50ruv: {replica 22 ldap://fs-hiido-kerberos-server01.hiido.host.xx.c
 om:389} 5ec7b6b1000000160000 5ec7b718000800160000
nsds50ruv: {replica 44 ldap://fs-hiido-kerveros-test08.hiido.host.xx.com
 :389} 5ecf60220000002c0000 6973199637ee002c0000
nsds50ruv: {replica 45} 5ecf755e0000002d0000 5ecf75850000002d0000
nsds50ruv: {replica 46 ldap://fs-hiido-kerberos-server02.hiido.host.xx.c
 om:389} 5ecf76ac0000002e0000 5ecf8bc40006002e0000
nsds50ruv: {replica 43 ldap://fs-hiido-hadoop-assit-21-33-33.hiido.host.yydevo
 ps.com:389} 5ecf67650002002b0000 5ed0bd6a0004002b0000
nsds50ruv: {replica 38 ldap://fs-hiido-kerveros-test06.hiido.host.xx.com
 :389} 5ecf6e77001100260000 5ed4d8ff002500260000
nsds50ruv: {replica 39 ldap://fs-hiido-kerveros-test07.hiido.host.xx.com
 :389} 5ecf6e77001600270000 5ed4d70b000200270000
nsds50ruv: {replica 54 ldap://fs-hiido-kerberos-mytest01.hiido.host.xx.c
 om:389} 5ed711b4000000360000 5ed7247b000200360000
nsds50ruv: {replica 60 ldap://fs-hiido-ipa-65-155.hiido.host.xx.com:389}
  63108d650000003c0000 69731aae0000003c0000
nsds50ruv: {replica 63 ldap://fs-hiido-kerberos-ca-02.hiido.host.xx.com:
 389} 631565ee0000003f0000 697321d90003003f0000
nsruvReplicaLastModified: {replica 62 ldap://ipa-65-189.hiido.host.xx.co
 m:389} 00000000
nsruvReplicaLastModified: {replica 4 ldap://fs-hiido-kerberos-21-117-149.hiido
 .host.xx.com:389} 00000000
nsruvReplicaLastModified: {replica 48 ldap://fs-hiido-kerberos-server03.hiido.
 host.xx.com:389} 00000000
nsruvReplicaLastModified: {replica 7} 00000000
nsruvReplicaLastModified: {replica 22 ldap://fs-hiido-kerberos-server01.hiido.
 host.xx.com:389} 00000000
nsruvReplicaLastModified: {replica 44 ldap://fs-hiido-kerveros-test08.hiido.ho
 st.xx.com:389} 00000000
nsruvReplicaLastModified: {replica 45} 00000000
nsruvReplicaLastModified: {replica 46 ldap://fs-hiido-kerberos-server02.hiido.
 host.xx.com:389} 00000000
nsruvReplicaLastModified: {replica 43 ldap://fs-hiido-hadoop-assit-21-33-33.hi
 ido.host.xx.com:389} 00000000
nsruvReplicaLastModified: {replica 38 ldap://fs-hiido-kerveros-test06.hiido.ho
 st.xx.com:389} 00000000
nsruvReplicaLastModified: {replica 39 ldap://fs-hiido-kerveros-test07.hiido.ho
 st.xx.com:389} 00000000
nsruvReplicaLastModified: {replica 54 ldap://fs-hiido-kerberos-mytest01.hiido.
 host.xx.com:389} 00000000
nsruvReplicaLastModified: {replica 60 ldap://fs-hiido-ipa-65-155.hiido.host.yy
 devops.com:389} 00000000
nsruvReplicaLastModified: {replica 63 ldap://fs-hiido-kerberos-ca-02.hiido.hos
 t.xx.com:389} 00000000
nsds5ReplicaEnabled: on
nsds5replicareapactive: 0
nsds5replicaLastUpdateStart: 19700101000000Z
nsds5replicaLastUpdateEnd: 19700101000000Z
nsds5replicaChangesSentSinceStartup:
nsds5replicaLastUpdateStatus: 402 Replication error acquiring replica: unknown
  error - Replica has different database generation ID, remote replica may nee
 d to be initialized
nsds5replicaUpdateInProgress: FALSE
nsds5replicaLastInitStart: 19700101000000Z
nsds5replicaLastInitEnd: 19700101000000Z
Edited by feiliang

cat reinit.ldif

dn: cn=ipa-65-189.hiido.host.yydevops.com-to-fs-hiido-kerberos-ca-02.hiido.host.yydevops.com,cn=replica,cn=dc\3Dyydevops\2Cdc\3Dcom,cn=mapping tree,cn=config
changetype: modify
add: nsds5BeginReplicaRefresh
nsds5BeginReplicaRefresh: start

ldapmodify -x -D "cn=Directory Manager" -W -f reinit.ldif

error log 

[26/Jan/2026:14:39:01 +0800] NSMMReplicationPlugin - Beginning total update of replica "agmt="cn=ipa-65-189.hiido.host.yydevops.com-to-fs-hiido-kerberos-ca-02.hiido.host.yydevops.com" (ipa-65-189:389)".
[26/Jan/2026:14:39:01 +0800] NSMMReplicationPlugin - conn=1 op=923 replica="dc=yydevops,dc=com": Unable to acquire replica: error: permission denied

I verified that all the /etc/dirsrv/ds.keytab files work properly for kinit to klist operations. KNVO also does not match. Could you please clarify which specific permissions are being referred to here?

It was the AI that messed up the permissions. It has been attempting to perform the copy operation using the "/etc/dirsrv/ds.key" or the "cn=Directory Manager" -W account of LDAP.
The correct way is to use "kinit admin"
to perform the "ipa-replica-manage re-initialize" operation

This doesn't look like a code problem, rather an operational mistake. We do not use the project issues for this purpose. If you have operational problems, please consider using freeipa-users@ mailing list instead.

I'm closing this issue.

Metadata Update from @abbra:
- Issue close_status updated to: invalid
- Issue status updated to: Closed (was: Open)
Metadata
None
None
None
None
None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.