ping @tbordaz @mreynolds do you mind checking this please :)

@firstyear the patch looks good but running rfc2307compt.py (https://pagure.io/389-ds-base/issue/50933#comment-642253) it looks some definitions are still problematic.

I removed (nisNetgroupTriple, ,nisMapName, nisMapEntryb, nisNetgroup, nisMap and nisObject) the test went further but still has a problem of replication (where the added definition (objectcategory) is not propagated)

In addition to this test, I think we need to run upgrade test and also a test where an instance got standard definitions in 99user.ldif (https://pagure.io/389-ds-base/issue/50933#comment-642326). This last bug can be hit with an update of the schema on a replica, the consumer should get std def in 99user.ldif

How did you test the replication in this case @tbordaz? That way I can test it myself to try and work out some of the issues.

I made a python test case in the Ticket #50933 (https://pagure.io/389-ds-base/issue/50933#comment-642253). It creates 2 masters one with rfc2307 and one with rfc2307compat. Then it updates the schema (add a new definition) on one master and checks if the definition gets replicated. This is what happens if a new instance (rfc2307compat) joins a topology with rfc2307 deployed.

Thanks, I'll give that a go today. Appreciate it.

Hey @tbordaz I had to adjust your test to get it to work properly, but it ... works? I can't reproduce what's going on here.

It looks like the schema replication works, even though the instances have different rfc2307 instances (master1 compat, master2 the original). I even then extended the test to "simulate" updating the rfc2307 to compat on M2 after the first test, and running again. Again, it all works. So I don't understand why your setup is failing? I'm gonna need more info on this, and maybe you could try with my copy of the test (I've put it into the PR).

Thanks,

Saying that, something I did notice is that basically the entire schema is duplicated into 99user.ldif on master2 which seems ... like a bug. The schemas should be same except the replicated objectCategory and the rfc2307 diffs. There shouldn't be a need to replicate the whole thing :|

@tbordaz @mreynolds ping on this one :)

@tbordaz ping again, it has been a month ....

@firstyear, sorry for the late feedback. It hits same issue (schema not pushed) with the provided testcase (https://pagure.io/389-ds-base/pull-request/51090#comment-120258). I had to slightly update the testcase and attached the new version of the testcase in https://pagure.io/389-ds-base/issue/50933#comment-665860

All good. I'll check the updated test case and see what's going on. Thanks for that.

So I was really stumped to why your test and my test are giving different results - In my test, which is very very similar it works, and schema replication has no issues, but in your test it fails.

I noticed a very odd behaviour in your test. In M1 we have 10rfc2307compat.ldif which contains definitions for automounts and gecos. But it looks like in 99user.ldif these were over-loaded with definitions from rfc2307.ldif from M2.

attributeTypes: (
  1.3.6.1.1.1.1.2 NAME 'gecos'
  DESC 'The GECOS field; the common name'
  EQUALITY caseIgnoreIA5Match
  SUBSTR caseIgnoreIA5SubstringsMatch
  SYNTAX 1.3.6.1.4.1.1466.115.121.1.26
  SINGLE-VALUE
  )
VS
attributeTypes: ( 1.3.6.1.1.1.1.2 NAME 'gecos' DESC 'Standard LDAP attribute t
 ype' SYNTAX 1.3.6.1.4.1.1466.115.121.1.15 SINGLE-VALUE X-ORIGIN ( 'RFC 2307'
 'user defined' ) )

I'm wondering if the difference between our tests is not a methodology, but a race condition - your test has M2 -> M1 first, where my test may have M1 -> M2 first. It could explain why my test works and yours does not, but I'm not 100% clear on why this should be different at all.

I'm going to re-run my test and check the replication logs also there to try to understand this, but I now have a reproducer of what you are seeing at least. Thanks again for your patience @tbordaz, but finally I'm making some progress!

I have to say thank you to you @firstyear for such frustrating investigations. Schema enhancement are always looking simple until diving in various upgrade scenario. I just hope we will find a solution more flexible than asking for a full upgrade of the topology at the same time.

The definitions you are seeing in 99user.ldif (gecos/automount) are likely the result of schema learning. THe server where the schema is updated (add 'objectCategory') evaluates if it can push its schema (because it has a higher schemacsn). So comparing the definitions I guess it detected extended definitions from the other server and updated its schema (99user) with them.
Unfortunately it found some definition that it was not able to learn (incompatible) and then decided it can not push its schema to the other server.

You need to enable replication logging to get verbose explanation what it is doing and why it is failing.

I do not really understand what differs from your testcase with mine. They look similar. Thank for your patience on this. How urgent/important it that ticket for you ?

You need to enable replication logging to get verbose explanation what it is doing and why it is failing.

Yep I'm going to add this to both tests and compare to understand this.

I do not really understand what differs from your testcase with mine. They look similar. Thank for your patience on this. How urgent/important it that ticket for you ?

It's "important" but there is a few months before it's "urgent". So I'm happy to work on it now to understand it, so I will continue my investigation. I'll let you know what I find :)

@tbordaz, Okay I understand the issue now. In my test, I move all the schema into /etc which means that the schema is loaded where 10rfc2307compat.ldif is before 60autofs.ldif. But in your test you only move the 10rfc2307 files, and the system schema remains as is. This means that 10rfc2307 is loaded after which means the definition of automountInformation came from 10rfc2307compat. As automountInformation and others from autofs and compat were different, this is what caused the schema replication to fail as they were conflicting.

60autofs appears to be a "very early" copy of the rfc2307bis automount content, but early enough that it does not match 2307bis.

To resolve this I have modify 60autofs.ldif to be a "super set" combining the two. It retains the 60autofs.ldif may/must rules, and extends the "may" rules to allow a 2307bis object to be added with some slight modifications). This means that your test and my test now both pass with schema replication :)

Ps: I haven't adjust the commit messages yet, I'll let you review it first.

https://tools.ietf.org/html/draft-howard-rfc2307bis-01 says it is dirstring not IA5. What RFC do you want do you want to follow ?

idem for the MR. I agree that coming from rfc2307compat it was IA5 but https://tools.ietf.org/html/draft-howard-rfc2307bis-01 says it is dirstring.
dirstring including IA5 and if we want to follow rfcbis we could define it as dirstring.

Just for recording Ticket is https://pagure.io/389-ds-base/issue/50933

https://tools.ietf.org/html/draft-howard-rfc2307bis-01 says automountKey MUST but I agree with your change because automoukey was neither required/allowed making it allowed makes sense
RFC does not mention 'cn' shouldn't we relax the requirement of 'cn' moving it to MAY ?

RFC does not mention 'ou' shouldn't we relax the requirement of 'ou' moving it to MAY ?

@firstyear thanks for the patch, IMHO remains MR and attributes to move(or not) must/may. Did you try your patch with freeipa ipa-server/replica-install ?

@tbordaz I thought about changing the types in 60autofs to match what is in rfc2307bis, but I think i'd rather leave them as they are. They may already be in use, so I don't want to disturb things too much.

You're right about the possibility of moving ou to may as well. It allows more configurations to be valid, but it also then removes a "strictness". Saying this, I think automount may not be a very popular schema, so I think moving this to MAY is the correct move to "allow many configs".

So I think you're right that we should be as "forgiving" as possible here. Honestly, this whole processhas been a mess and really highlights how poor-past decisions can be hard to undo in the future :(

@mreynolds I've updated this based @tbordaz's suggestions, so I think it's good to go through a freeipa test now. Thanks!

Hey @mreynolds did we ever get this tested by FreeIPA?

Hey @mreynolds did we ever get this tested by FreeIPA?

They are testing it now, I should have the results tomorrow!

Thank you!

Hey @mreynolds did we ever get this tested by FreeIPA?

They are testing it now, I should have the results tomorrow!

IPA tests passed! Ack

Thank you very much @mreynolds and @tbordaz ! This is going to make it a lot easier to allow openldap migrations into 389 so I really appreciate your patience with this!

As an FYI @mreynolds I'm leaving this as two commits, as there is a an "enable" commit we can quickly revert if we find (yet another) problem here.

Thanks!

Well the IPA tests passed, but I just ran a test with two different versions of DS with IPA (one with your fix and one without) , and it broke IPA replica install:

The ipa-replica-install command failed, exception: NameNotUnique: NAME not unique for b"( 1.3.6.1.1.1.2.13 NAME 'nisMap' DESC 'Standard LDAP objectclass' SUP top STRUCTURAL MUST nisMapName MAY description X-ORIGIN ( 'RFC 2307' 'user defined' ) )"

I do not know if this is specific to IPA or not. There needs to be more testing. But we need this to work with mixed versions of DS or else we need to look at other options. Maybe we do need to support different/auxiliary schema directories? I'll do some more testing later today...

I don't see nisMap or OID 1.3.6.1.1.1.2.13 in IPA at all, so this is something coming from the 389-ds.

If you look at various 10rfc23097*.ldif files, you can see that there are two different OIDs associated with nisMap:

$ git grep -i nismap
ldap/schema/10rfc2307.ldif:attributeTypes: ( 1.3.6.1.1.1.1.26 NAME 'nisMapName' DESC 'Standard LDAP attribute type' SYNTAX 1.3.6.1.4.1.1466.115.121.1.15 X-ORIGIN 'RFC 2307' )
ldap/schema/10rfc2307.ldif:attributeTypes: ( 1.3.6.1.1.1.1.27 NAME 'nisMapEntry' DESC 'Standard LDAP attribute type' SYNTAX 1.3.6.1.4.1.1466.115.121.1.26 SINGLE-VALUE X-ORIGIN 'RFC 2307' )
ldap/schema/10rfc2307.ldif:objectClasses: ( 1.3.6.1.1.1.2.10 NAME 'nisObject' DESC 'Standard LDAP objectclass' SUP top STRUCTURAL MUST ( cn $ nisMapEntry $ nisMapName ) MAY ( description ) X-ORIGIN 'RFC 2307' )
ldap/schema/10rfc2307.ldif:objectClasses: ( 1.3.6.1.1.1.2.13 NAME 'nisMap' DESC 'Standard LDAP objectclass' SUP top STRUCTURAL MUST ( nisMapName ) MAY ( description ) X-ORIGIN 'RFC 2307' )
ldap/schema/10rfc2307bis.ldif:  1.3.6.1.1.1.1.26 NAME 'nisMapName'
ldap/schema/10rfc2307bis.ldif:  1.3.6.1.1.1.1.27 NAME 'nisMapEntry'
ldap/schema/10rfc2307bis.ldif:  1.3.6.1.1.1.2.9 NAME 'nisMap' SUP top STRUCTURAL
ldap/schema/10rfc2307bis.ldif:  MUST nisMapName
ldap/schema/10rfc2307bis.ldif:  MUST ( cn $ nisMapEntry $ nisMapName )
ldap/schema/10rfc2307compat.ldif:  1.3.6.1.1.1.1.26 NAME 'nisMapName'
ldap/schema/10rfc2307compat.ldif:  1.3.6.1.1.1.1.27 NAME 'nisMapEntry'
ldap/schema/10rfc2307compat.ldif:  1.3.6.1.1.1.2.9 NAME 'nisMap' SUP top STRUCTURAL
ldap/schema/10rfc2307compat.ldif:  MUST nisMapName
ldap/schema/10rfc2307compat.ldif:  MUST ( cn $ nisMapEntry $ nisMapName )

1.3.6.1.1.1.2.13 oid comes from rfc2307.ldif. With this patch rfc2307.ldif is no longer a default schema, the new default schema is rfc2307compat.ldif that delivers the same objectclass but with a different OID (IIRC 1.3.6.1.1.1.2.9)

With mixed DS version, it creates an issue having same objectclass with different OID.

IIRC, it was decided to remove nis* definitions and to move them in data/60nis.ldif

389-ds-base is moving from Pagure to Github. This means that new issues and pull requests
will be accepted only in 389-ds-base's github repository.

This pull request has been cloned to Github as issue and is available here:
- https://github.com/389ds/389-ds-base/issues/4143

If you want to continue to work on the PR, please navigate to the github issue,
download the patch from the attachments and file a new pull request.

Thank you for understanding. We apologize for all inconvenience.