freeipa

#6280 Deadlock in schema compat between a SRCH and ADD (from dna plugin) operation
Opened by tbordaz. Modified

Open
Close Issue

389-ds-base-1.3.5.13-1.fc24.x86_64

IPA 4.4 git build

The deadlock occurs because of two locks (DB and Schema Compat map) taken in the opposite order. It occurs between a ADD and a SRCH thread:

DNA plugin creates a shared host entry:

        "dnaHostname=vm-140-013.<domain>+dnaPortNum=389,cn=posix-ids,cn=dna,cn=ipa,cn=etc,<suffix>"
        This ADD acquires some DB pages (entryrdn and objectclass) than calling
        post_txnbe schema compat callback (backend_shr_betxn_post_add_cb) it acquires the
        schema compat map lock
Thread 12 (Thread 0x7f870b7fe700 (LWP 24045)):
#0  0x00007f872a152f68 in __pthread_rwlock_wrlock_slow () from /lib64/libpthread.so.0
#1  0x00007f871aba9f8e in backend_shr_post_add_cb.part () from /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#2  0x00007f871abaa167 in backend_shr_betxn_post_add_cb () from /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#3  0x00007f872c5e8da9 in plugin_call_func () from /usr/lib64/dirsrv/libslapd.so.0
#4  0x00007f872c5e9014 in plugin_call_plugins () from /usr/lib64/dirsrv/libslapd.so.0
#5  0x00007f871cb5de17 in ldbm_back_add () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
#6  0x00007f872c58bf6b in op_shared_add () from /usr/lib64/dirsrv/libslapd.so.0
#7  0x00007f872c58c7f3 in add_internal_pb () from /usr/lib64/dirsrv/libslapd.so.0
#8  0x00007f872c58d4fe in slapi_add_internal_pb () from /usr/lib64/dirsrv/libslapd.so.0
#9  0x00007f87212864c2 in dna_update_shared_config () from /usr/lib64/dirsrv/plugins/libdna-plugin.so
#10 0x00007f872128931b in dna_update_config_event () from /usr/lib64/dirsrv/plugins/libdna-plugin.so
#11 0x00007f872c5aeacc in eq_loop () from /usr/lib64/dirsrv/libslapd.so.0
#12 0x00007f872a7af7df in _pt_root () from /lib64/libnspr4.so
#13 0x00007f872a14e5ca in start_thread () from /lib64/libpthread.so.0
#14 0x00007f8729e87ead in clone () from /lib64/libc.so.6

and a search request

 * base:   "<suffix>"
 * scope:  sub
 * filter: (|(&(gidNumber=505600000)(objectClass=ipaNTGroupAttrs))(&(uidNumber=505600000)(objectClass=posixAccount)(objectClass=ipaNTUserAttrs)))"
 * attrs:  ipaNTSecurityIdentifier objectClass
        that call schema compat callback backend_search_cb that acquire
        schema compat map lock and then need to access some db pages (here entryrdn index)
Thread 18 (Thread 0x7f86ff7fe700 (LWP 24052)):
#0  0x00007f872a153bd0 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f8723857643 in __db_hybrid_mutex_suspend () from /lib64/libdb-5.3.so
#2  0x00007f8723856a27 in __db_tas_mutex_lock () from /lib64/libdb-5.3.so
#3  0x00007f8723904070 in __lock_get_internal () from /lib64/libdb-5.3.so
#4  0x00007f8723904ba7 in __lock_get () from /lib64/libdb-5.3.so
#5  0x00007f872393143f in __db_lget () from /lib64/libdb-5.3.so
#6  0x00007f872387585a in __bam_search () from /lib64/libdb-5.3.so
#7  0x00007f872385f79b in __bamc_search () from /lib64/libdb-5.3.so
#8  0x00007f8723861dff in __bamc_get () from /lib64/libdb-5.3.so
#9  0x00007f872391d8a3 in __dbc_iget () from /lib64/libdb-5.3.so
#10 0x00007f872392d1d6 in __dbc_get_pp () from /lib64/libdb-5.3.so
#11 0x00007f871cb6ce84 in _entryrdn_get_elem () from /usr/lib64/dirsrv/plugins/libback-ldbm.so                                         
#12 0x00007f871cb703b8 in _entryrdn_index_read () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
#13 0x00007f871cb718d9 in entryrdn_index_read_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
#14 0x00007f871cb40ea9 in dn2entry_ext () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
#15 0x00007f871cb43d38 in find_entry_internal.isra () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
#16 0x00007f871cb441eb in find_entry () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
#17 0x00007f871cb7f496 in ldbm_back_search () from /usr/lib64/dirsrv/plugins/libback-ldbm.so
#18 0x00007f872c5dd420 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0
#19 0x00007f872c5ee00e in search_internal_callback_pb () from /usr/lib64/dirsrv/libslapd.so.0
#20 0x00007f871abb97e0 in wrap_search_internal_get_entry () from /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#21 0x00007f871abb1154 in format_deref_rx.isra () from /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#22 0x00007f871abb18bf in format_deref_r () from /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#23 0x00007f871abafc20 in format_expand () from /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#24 0x00007f871abb0aa4 in format_format () from /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#25 0x00007f871abb24f3 in format_get_data_set () from /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#26 0x00007f871aba59cd in backend_set_entry_from () from /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#27 0x00007f871aba6ab0 in backend_search_cb () from /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#28 0x00007f872c5e8da9 in plugin_call_func () from /usr/lib64/dirsrv/libslapd.so.0
#29 0x00007f872c5e9014 in plugin_call_plugins () from /usr/lib64/dirsrv/libslapd.so.0
#30 0x00007f872c5dd0b2 in op_shared_search () from /usr/lib64/dirsrv/libslapd.so.0
#31 0x000055f9b7b3d577 in do_search ()
#32 0x000055f9b7b2b2b5 in connection_threadmain ()
#33 0x00007f872a7af7df in _pt_root () from /lib64/libnspr4.so
#34 0x00007f872a14e5ca in start_thread () from /lib64/libpthread.so.0
#35 0x00007f8729e87ead in clone () from /lib64/libc.so.6

Additional data about the deadlock

Deadlock threads
     410 dd=120 locks held 0    write locks 0    pid/thread 24032/ 140217788917504 flags 0    priority 100        7f86ff7fe700
     410 READ          1 WAIT    userRoot/entryrdn.db      page         23
80006524 dd=45 locks held 61   write locks 25   pid/thread 24032/ 140217990244096 flags 0    priority 100        7f870b7fe700
80006524 READ         11 HELD    userRoot/entryrdn.db      page         23
80006524 READ         25 HELD    userRoot/objectclass.db   page          5
80006524 WRITE         2 HELD    userRoot/objectclass.db   page          5
80006524 READ          1 HELD    userRoot/objectclass.db   page          2
80006524 WRITE         2 HELD    userRoot/objectclass.db   page          2
Threads indirect victim of the deadlock                                                                                                
     41c dd=108 locks held 0    write locks 0    pid/thread 24032/ 140217654634240 flags 0    priority 100        7f86f77ee700
     41c READ          1 WAIT    userRoot/objectclass.db   page          5
     46c dd=27 locks held 0    write locks 0    pid/thread 24032/ 140217663026944 flags 0    priority 100        7f86f7fef700
     46c READ          1 WAIT    userRoot/objectclass.db   page          5
     46d dd=26 locks held 0    write locks 0    pid/thread 24032/ 140217604278016 flags 0    priority 100        7f86f47e8700
     46d READ          1 WAIT    userRoot/objectclass.db   page          5
     46e dd=25 locks held 0    write locks 0    pid/thread 24032/ 140217629456128 flags 0    priority 100        7f86f5feb700
     46e READ          1 WAIT    userRoot/objectclass.db   page          5
     470 dd=23 locks held 0    write locks 0    pid/thread 24032/ 140217704990464 flags 0    priority 100        7f86fa7f4700
     470 READ          1 WAIT    userRoot/objectclass.db   page          5
     472 dd=21 locks held 0    write locks 0    pid/thread 24032/ 140217646241536 flags 0    priority 100        7f86f6fed700
     472 READ          1 WAIT    userRoot/objectclass.db   page          5
     474 dd=19 locks held 0    write locks 0    pid/thread 24032/ 140217696597760 flags 0    priority 100        7f86f9ff3700
     474 READ          1 WAIT    userRoot/objectclass.db   page          5
     476 dd=17 locks held 0    write locks 0    pid/thread 24032/ 140217612670720 flags 0    priority 100        7f86f4fe9700
     476 READ          1 WAIT    userRoot/objectclass.db   page          5
     478 dd=15 locks held 0    write locks 0    pid/thread 24032/ 140217637848832 flags 0    priority 100        7f86f67ec700
     478 READ          1 WAIT    userRoot/objectclass.db   page          5
     47a dd=13 locks held 0    write locks 0    pid/thread 24032/ 140217713383168 flags 0    priority 100        7f86faff5700
     47a READ          1 WAIT    userRoot/objectclass.db   page          5
     47c dd=11 locks held 0    write locks 0    pid/thread 24032/ 140217746953984 flags 0    priority 100        7f86fcff9700
     47c READ          1 WAIT    userRoot/objectclass.db   page          5
     47e dd=9 locks held 0    write locks 0    pid/thread 24032/ 140217621063424 flags 0    priority 100        7f86f57ea700
     47e READ          1 WAIT    userRoot/objectclass.db   page          5
     480 dd=7 locks held 0    write locks 0    pid/thread 24032/ 140217948280576 flags 0    priority 100        7f8708ff9700
     480 READ          1 WAIT    userRoot/objectclass.db   page          5
     482 dd=5 locks held 0    write locks 0    pid/thread 24032/ 140217763739392 flags 0    priority 100        7f86fdffb700
     482 READ          1 WAIT    userRoot/objectclass.db   page          5
     484 dd=3 locks held 0    write locks 0    pid/thread 24032/ 140217738561280 flags 0    priority 100        7f86fc7f8700
     484 READ          1 WAIT    userRoot/objectclass.db   page          5
     486 dd=1 locks held 0    write locks 0    pid/thread 24032/ 140217688205056 flags 0    priority 100        7f86f97f2700
     486 READ          1 WAIT    userRoot/objectclass.db   page          5
     487 dd=0 locks held 0    write locks 0    pid/thread 24032/ 140217973458688 flags 0    priority 100        7f870a7fc700

I wonder if it would be an option to check ignore_subtrees or restrict_subtrees before acquiring the map lock.

Writing down a POSSIBLE reproducible test case:

Install an IPA server

Provision some users/groups.... Then get uidNumber/gidNumber of one user.

The search part is:

ldapsearch -LLL -D "cn=directory manager" -w Secret123 -x -b "<suffix>" "(|(&(gidNumber=<gidNumber>)(objectClass=ipasshuser))(&(uidNumber=<uidNumber>)(objectClass=posixAccount)(objectClass=ipasshuser)))"

The update part is ADD/DEL a dummy entry

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
#!/bin/sh
SUFFIX="<suffix>
HOST_DUMMY="dummy.<domain>"
PASSWORD=<dm_passwd>
while :
do
DN="dnaHostname=${HOST_DUMMY}+dnaPortNum=389,cn=posix-ids,cn=dna,cn=ipa,cn=etc,${SUFFIX}"
ldapadd -D "cn=directory manager" -w ${PASSWORD} <<!EOF
dn: ${DN}
objectClass: dnaSharedConfig
objectClass: top
dnaHostname: ${HOST_DUMMY}
dnaPortNum: 389
dnaSecurePortNum: 636
dnaRemainingValues: 192193
dnaRemoteBindMethod: SASL/GSSAPI
dnaRemoteConnProtocol: LDAP
!EOF
ldapdelete -D "cn=directory manager" -w ${PASSWORD} ${DN}
done

Run once the update part in background. Run N times in parallel the search part

I have been unsuccessful to reproduce with the above test case. I created a trust with AD, provisioned a AD user then search (ldapsearch) it. I did not get to the point where the search goes backend_search_cb->backend_set_entry_from. It could be related to some configuration specificity or entry being cache...

Anyway the deadlock does not occur in a easy test or simple configuration. So the frequency of the deadlock should be low.

We ran into this issue again in our CI during a legacy clients test. According to Thierry the stack traces looked the same on the affected VM.

Packages:
389-ds-base-1.3.5.13-1.fc24.x86_64
freeipa built from ipa-4-4 branch

Attached is the jenkins job output. The job is stuck on uninstallation but there are earlier kerberos failures probably caused by the deadlock.

attachment
legacy-clients.log

The problem appears again while testing test_integration/test_legacy_clients (domain level 0)

The problem happens rarely because there is a short window for dna_update_config_event (spawn few secs (30) after startup). When dna_update_config_event run, IMHO it should be very easy to reproduce the hang with SRCHs like: scope subtree, base ""cn=ad,cn=trusts," '(objectclass=*)', and load that trigger dna_update_config.

The key element is the configuration and I failed to reproduce this configuration :-(.
The configuration should lead to searches ending in 'format_deref_rx.isra' in schema compat.

Metadata Update from @tbordaz:
- Issue assigned to someone
- Issue set to the milestone: FreeIPA 4.5 backlog
Metadata
None
None
None
Normal
defect
NIS plugin
0
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.