#8392 nsslapd-unhashed-pw-switch reverted to 'nolog' every upgrade, breaks winsync
Opened by nhinds. Modified

Issue

I have 3 FreeIPA servers replicating with one another, and one of those servers has a winsync replication with Active Directory. Every time I upgrade FreeIPA, the 2 servers without the winsync agreement revert nsslapd-unhashed-pw-switch back to 'nolog', and password changes on those servers no longer sync with Active Directory.

Steps to Reproduce

  1. Deploy multiple FreeIPA servers replicating with one another
  2. Create a winsync agreement on one of the servers with an Active Directory server; this sets nsslapd-unhashed-pw-switch: on for that single FreeIPA server
  3. Set nsslapd-unhashed-pw-switch: on for all other FreeIPA servers, so that password syncing works, e.g.
ldapmodify -D 'cn=directory manager' -W
dn: cn=config
changetype: modify
replace: nsslapd-unhashed-pw-switch
nsslapd-unhashed-pw-switch: on
  1. Change user passwords on each of the FreeIPA servers
  2. Run ipa-server-upgrade on each of the FreeIPA servers
  3. Change user passwords on each of the FreeIPA servers

Actual behavior

Password changes are synced to AD correctly before the upgrade, but after the upgrade only password changes on the FreeIPA server with the winsync agreement work. Password changes on other FreeIPA servers are synced to all FreeIPA servers, but they are not synced to AD.

Expected behavior

Password changes on all FreeIPA servers should sync to AD.

Running the upgrade tool should not revert my configuration settings.

Version/Release/Distribution

$ rpm -q freeipa-server freeipa-client ipa-server ipa-client 389-ds-base pki-ca krb5-server
package freeipa-server is not installed
package freeipa-client is not installed
ipa-server-4.6.6-11.el7.centos.x86_64
ipa-client-4.6.6-11.el7.centos.x86_64
389-ds-base-1.3.10.1-9.el7_8.x86_64
pki-ca-10.5.17-6.el7.noarch
krb5-server-1.15.1-46.el7.x86_64

Additional info:

The upgrade script appears to have been added in https://pagure.io/freeipa/issue/4812, which has a comment explaining "As we found out, this option needs to be set on all FreeIPA servers when winsync is enabled". Unfortunately the upgrade script does not match the comment.


Can you please provide ipaupgrade.log? The upgrade plugin does force on value if there is at least one winsync agreement enabled.

The detection of the winsync agreement is done by looking for an entry with (objectclass=nsDSWindowsReplicationAgreement) below cn=config (see the code). As cn=config is not replicated, only the node with a winsync agreement will get the on value forced.

Right, it doesn't find the winsync agreements on other servers.

I'd rather not provide the full ipaupgrade.log because it seems to contain details about all users in my directory, but the pieces related to the update_unhashed_password upgrade plugin are:
On the server with the winsync agreement:

2020-07-02T23:13:57Z DEBUG Executing upgrade plugin: update_unhashed_password
2020-07-02T23:13:57Z DEBUG raw: update_unhashed_password
2020-07-02T23:13:57Z DEBUG Upgrading unhashed password configuration
2020-07-02T23:13:57Z WARNING This server is configured for winsync, the changelog files under /var/lib/dirsrv/slapd-MY-DOMAIN/cldb may contain clear text passwords.
Please ensure that these files can be accessed only by trusted accounts.
2020-07-02T23:13:57Z DEBUG Destroyed connection context.ldap2_139761033427216
2020-07-02T23:13:57Z DEBUG   duration: 16 seconds

On another server:

2020-07-02T23:15:28Z DEBUG Executing upgrade plugin: update_unhashed_password
2020-07-02T23:15:28Z DEBUG raw: update_unhashed_password
2020-07-02T23:15:28Z DEBUG Upgrading unhashed password configuration
2020-07-02T23:15:28Z DEBUG Unhashed password this is not a winsync deployment
2020-07-02T23:15:28Z DEBUG Updating existing entry: cn=config
2020-07-02T23:15:28Z DEBUG ---------------------------------------------
2020-07-02T23:15:28Z DEBUG Initial value
2020-07-02T23:15:28Z DEBUG dn: cn=config
...
2020-07-02T23:15:28Z DEBUG nsslapd-unhashed-pw-switch:
2020-07-02T23:15:28Z DEBUG  on
...
2020-07-02T23:15:28Z DEBUG remove: 'on' from nsslapd-unhashed-pw-switch, current value [u'on']
2020-07-02T23:15:28Z DEBUG remove: updated value []
2020-07-02T23:15:28Z DEBUG add: 'nolog' to nsslapd-unhashed-pw-switch, current value []
2020-07-02T23:15:28Z DEBUG add: updated value [u'nolog']
2020-07-02T23:15:28Z DEBUG ---------------------------------------------
2020-07-02T23:15:28Z DEBUG Final value after applying updates
2020-07-02T23:15:28Z DEBUG dn: cn=config
...
2020-07-02T23:15:28Z DEBUG nsslapd-unhashed-pw-switch:
2020-07-02T23:15:28Z DEBUG  nolog
...
2020-07-02T23:15:28Z DEBUG [(2, u'nsslapd-unhashed-pw-switch', [u'nolog'])]
2020-07-02T23:15:28Z DEBUG Updated 1
2020-07-02T23:15:28Z DEBUG Done
2020-07-02T23:15:28Z DEBUG Destroyed connection context.ldap2_140476404646992
2020-07-02T23:15:28Z DEBUG   duration: 14 seconds

If there is more information you need from somewhere else in the ipaupgrade.log files please let me know, but I don't have the time to sanitize the full ~39,000 lines of logs that each ipa-server-upgrade command writes.

The output of searching for (objectclass=nsDSWindowsReplicationAgreement) on those 2 servers:
On the server with the winsync agreement:

$ ldapsearch -D cn=directory\ manager -W -b cn=config '(objectclass=nsDSWindowsReplicationAgreement)' -o ldif-wrap=no dn
# extended LDIF
#
# LDAPv3
# base <cn=config> with scope subtree
# filter: (objectclass=nsDSWindowsReplicationAgreement)
# requesting: dn 
#
# meTowindows.server.fqdn, replica, dc\3Dmy\2Cdc\3Ddomain, mapping tree, config
dn: cn=meTowindows.server.fqdn,cn=replica,cn=dc\3Dmy\2Cdc\3Ddomain,cn=mapping tree,cn=config
# search result
search: 2
result: 0 Success
# numResponses: 2
# numEntries: 1

On another server:

$ ldapsearch -D cn=directory\ manager -W -b cn=config '(objectclass=nsDSWindowsReplicationAgreement)' -o ldif-wrap=no dn
# extended LDIF
#
# LDAPv3
# base <cn=config> with scope subtree
# filter: (objectclass=nsDSWindowsReplicationAgreement)
# requesting: dn 
#
# search result
search: 2
result: 0 Success
# numResponses: 1

These extracts are helpful, thank you.

We need to amend the logic of detecting winsync agreements with something that would work on all replicas. One possible solution is to check whether one of object classes specific to winsync-ed users exists in at least one user entry. Users are replicated everywhere so they should be a good sign of a working agreement for the purpose of this check.

Something like (&(objectclass=ntUser)(ntUserDomainId=*)) should capture the fact that there are users brought in via winsync.

Metadata Update from @frenaud:
- Custom field rhbz adjusted to https://bugzilla.redhat.com/show_bug.cgi?id=1989944

Metadata