freeipa

#7797 SSSD's getservby*() causes performance issues
Closed: fixed Opened by cheimes.

Closed: fixed
Reopen Issue

issue

SSSD provides nsswitch feature for services as defined in https://tools.ietf.org/html/rfc2307#section-5.2 . For example the sss plugin provides a hook for getservbyport(57672, "tcp") that will query an IPA service for (&(ipserviceport=57672)(ipserviceprotocol=tcp)(objectclass=ipservice)).

IPA does not provide any facilities for ipService definitions out of the box. Although the query always returns 0, it can really costly. There is no index on ipServiceProtocol and ipServicePort. In ticket https://pagure.io/freeipa/issue/7786 a user reported query times of over a second:

  Unindexed Component #2 (notes=U)
  -  Date/Time:             14/Nov/2018:18:31:11
  -  Connection Number:     3866581
  -  Operation Number:      27
  -  Etime:                 1.1235918834
  -  Nentries:              0
  -  IP Address:            Unknown_Host
  -  Search Base:           cn=accounts,dc=example,dc=local
  -  Search Scope:          2 (subtree)
  -  Search Filter:         (&(ipserviceport=57672)(ipserviceprotocol=tcp)(objectclass=ipservice))

Why is the query so slow?

  • subtree scan on cn=accounts, which can easily contain 100k objects in a large installation (all users, groups, services, hosts, and more)
  • No index on ipServiceProtocol and ipServicePort
  • Possibly order of filter elements: (&(objectclass=ipservice)(ipserviceport=57672)(ipserviceprotocol=tcp)) might be faster than (&(ipserviceport=57672)(ipserviceprotocol=tcp)(objectclass=ipservice)), because 389-DS can first filter by an indexed attribute.

possible solutions

  1. create indices for both attributes
  2. define and create a subtree for IP services (e.g. cn=ipservices,cn=accounts) and have SSSD search this subtree with sub ONE. IPA could just create an container and leave it empty for future use.
  3. Reorder filter expression in SSSD to first filter on objectClass. (I don't know if this change would actually make a difference.)
  4. or any combination of above

Solution 1 is FreeIPA only but has the draw back of two additional indices. I don't know enough about 389-DS and LDBM internals to predict performance issues for general operations.

The other solutions require changes to SSSD and FreeIPA.

index creation

dn: cn=ipServicePort,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
changetype: add
cn: ipServicePort
objectClass:top
objectClass:nsIndex
nsSystemIndex: false
nsIndexType: eq
dn: cn=ipServiceProtocol,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
changetype: add
cn: ipServiceProtocol
objectClass:top
objectClass:nsIndex
nsSystemIndex: false
nsIndexType: eq

index update

dn: cn=ipServicePort,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
default: cn: ipServicePort
default: objectClass:top
default: objectClass:nsIndex
default: nsSystemIndex: false
default: nsIndexType: eq
dn: cn=ipServiceProtocol,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
default: cn: ipServiceProtocol
default: objectClass:top
default: objectClass:nsIndex
default: nsSystemIndex: false
default: nsIndexType: eq

@jhrozek @sbose @mreynolds request for comment, please.

From the DS side of things, adding the indexes would be the best approach. Rearranging the filter "could" help a little, but not as much as adding the indexes would. There is also very little overhead for these indexes since they are equality types.

On a side note, in DS we were automatically rearranging(optimizing) filter components for you, but this caused a regression in FreeIPA and we had to back it out . We are planning on adding it back once it gets stabilized. See https://pagure.io/389-ds-base/issue/50073

Do we actually know if anyone is using the ipServices in IPA at all? I did that manually a long time ago to test something on the client side, so I was OK with just stuffing objects into the tree with ldapadd..but I doubt many people are doing this in production.

I wonder if we could internally set ldap_service_search_base in sssd to something like cn=ipservices,cn=accounts but just don't create any container on the IPA side, so the search would at least not troll the whole db? And document that if you do have some services stored in IPA you need to reset it back?

About the filter ordering, I found https://pagure.io/SSSD/sssd/issue/3409 which actually suggests to go from the most specific item to the least specific one, IIRC this was filed after some discussion with either Thierry or Ludwig, but I forgot the details in the meantime, sorry..

@jhrozek, the ticket https://pagure.io/389-ds-base/issue/49372 was completed to optimize filter components. In some specific cases (see below) it showed perf improvements but we had to revert because it created some regressions. The changes were in low level mechanisms that it was difficult to evaluate all impacts and be sure all regressions were fixed.

In short the improved cases were

  • in union it moves the most likely unindexed component (substring) first (to be evaluated)
  • in intersection, it moves objectclass component to the end

@jhrozek ipService is not officially supported by IPA. There is zero mentioning of ipservice in the FreeIPA source code. The only for ipservice on FreeIPA website is a profile for HPUX, https://www.freeipa.org/page/HpuxNonTlsProfile. An internet search did not reveal anything conclusive either. It's safe to assume that IPA never claimed to support ipService and nobody uses it either.

It might be a good idea to create the cn=ipservices,cn=accounts container entry in IPA by default. The entry would serve as self-documenting element in case somebody wants to have ip services.

@tbordaz @mreynolds I don't have sufficient experience with indices in 389-DS to make any performance predictions. I don't even know how indexing work in detail. Would two additional indices make any impact on daily operations? I would assume that each index costs a couple of CPU cycles in each ADD, MOD or RDN op.

Creating the container seems like a nice idea, then we could patch the clients to also set the service base to this empty container. I guess we need a corresponding sssd ticket?

Search is done in two steps:

  • evaluating filter to get a list of entries (candidates) that could be returned
  • test the matching of the filter against each candidate before returning entries/attributes

Regarding evaluation of the filter to build a candidate list "(&(ipserviceport=57672)(ipserviceprotocol=tcp)(objectclass=ipservice))". If ipserviceport and ipserviceprotocol are not indexed, the two first components are ignored an the candidates are the entries that match '(objectclass=ipservice)'. Objectclass being indexed in equality it is fast to compute and moreover faster as there is no ipservice entry. Whatever the order of the components the evaluation of the filter should be fast.

Indexing ipserviceport and ipserviceprotocol is likely a good idea. But in that specific case it would slightly slowdown the search. Indeed instead of skipping the 2 first components, now it requires access to ipserviceport index page, retrieval of entriesID that contain ipserviceport=57672, then the same for ipserviceprotocol=tcp, intersection of both, this before processing objectclass.
Updating those index on update should be fast.
IMHO indexing ipserviceport should be the most beneficial as it will return very few entries, making shortcut of the filter evaluation.

Regarding the performance of the search, I think we need to confirm in access log if 'conn=3866581 op=27' last more that 1s. And if it did, then check what operations occurred at the same time.

Thanks @tbordaz

I agree with your that an index on ipServiceProtocol would have a very poor selectivity. A search for a port would return at max two entries for TCP and UDP variant. For this special case, a string match is going to faster than an index lookup.

I created a PR that creates the container and just an index on the service port. I left a comment that explains why IPA doesn't have an index on service protocol, too.

Metadata Update from @cheimes:
- Custom field on_review adjusted to https://github.com/freeipa/freeipa/pull/2661

Corresponding sssd issue: https://pagure.io/SSSD/sssd/issue/3899

master:

  • 39eaf2fab5e27bd12edfb2a24c439a8ea5fb26f0 Add index and container for RFC 2307 IP services

Metadata Update from @cheimes:
- Issue assigned to cheimes

ipa-4-6:

  • df606fff97af464b1b724efce55f9803d977251b Add index and container for RFC 2307 IP services

ipa-4-7:

  • 76052d5ac5b1c53d2586d3c48e2faacc6963004e Add index and container for RFC 2307 IP services

Metadata Update from @cheimes:
- Issue close_status updated to: fixed
- Issue status updated to: Closed (was: Open)
Metadata

performance

None
None
None
https://github.com/freeipa/freeipa/pull/2661
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.