freeipa

#6379 Deadlock while updating replication managers
Opened by tbordaz. Modified

Open
Close Issue

ipa 4.4 (git) - 389-ds-base-1.3.5.13-1.fc24.x86_64

Deadlock occurs with two threads acquiring locks in the opposite order.
one lock is DB lock (member index) and the other is Schema compat map lock

There is no identified reproducible test case. The deadlock is not systematic.

- Deadlock occurs with two threads acquiring locks in the opposite order
    - db lock of an updated page ('member' index)
    - Schema compat map lock
                [Current thread is 41 (Thread 0x7fabe4fe9700 (LWP 66373))]
                80000101 dd= 7 locks held 20   write locks 10   pid/thread 66331/140376258025216 flags 0    priority 100
                80000101 READ          7 HELD    userRoot/member.db        page          4
                ...
                [Current thread is 22 (Thread 0x7fabee7fc700 (LWP 66354))]
                      80 dd= 9 locks held 0    write locks 0    pid/thread 66331/140376417486592 flags 0    priority 100
                      80 READ          1 WAIT    userRoot/member.db        page          4
- first thread is applying a replicated operation to add a new member to 'replication managers'
  second thread is applying a direct operation to delete the entry "cn=replication manager,cn=config" (that is no longer used
  for replication)
  The first thread hold some 'member' index page lock under a txn and need to access to the SC map.
  The second holds the SC map lock and does a search using 'member' index
        MOD:
            replicated update
            entry: cn=replication managers,cn=sysaccounts,cn=etc,<suffix>
            MOD ADD: member: "krbprincipalname=ldap/<host_fqdn>@<domain>,cn=services,cn=accounts,<suffix>"
                -> memberof updates "krbprincipalname=ldap/<host_fqdn>@<domain>,cn=services,cn=accounts,<suffix>"
                    to MOD replace 'memberof' with 'cn=replication managers,cn=sysaccounts,cn=etc,<suffix>'
                -> slapi-nis MAP lock
        DEL "cn=replication manager,cn=config"
            slapi-nis - internal search to retrieve its direct groups owner
                    -b "cn=groups,cn=accounts,<suffix>"
                    -s one
                    "(member=cn=replication manager,cn=config)"
- It is likely that those two operations are related to the same admin operation
  that setup 'replication manager' entries, switching from individual entry (cn=replication manager,cn=config)
  to the group of 'cn=replication managers'
- There is no known workaround for that deadlock

Details of the threads

Thread updating cn=replication managers,cn=sysaccounts

[Current thread is 41 (Thread 0x7fabe4fe9700 (LWP 66373))]
#0  0x00007fac2048b8d1 in futex_wait (private=<optimized out>, expected=0, futex_word=0x5635f190fbf8)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:61
#1  futex_wait_simple (private=<optimized out>, expected=0, futex_word=0x5635f190fbf8) at ../sysdeps/nptl/futex-internal.h:135
#2  __pthread_rwlock_rdlock_slow (rwlock=0x5635f190fbf0) at pthread_rwlock_rdlock.c:68
#3  0x00007fac2048bb19 in __GI___pthread_rwlock_rdlock (rwlock=<optimized out>) at pthread_rwlock_rdlock.c:177
#4  0x00007fac22947aba in slapi_rwlock_rdlock (rwlock=<optimized out>) at ldap/servers/slapd/slapi2nspr.c:222
#5  0x00007fac13550728 in wrap_rwlock_rdlock (rwlock=<optimized out>) at wrap.c:313
#6  0x00007fac1354fa2c in map_rdlock () at map.c:1224
#7  0x00007fac1353ded4 in backend_write_cb (pb=pb@entry=0x7fab6c00a660, state=<optimized out>, state=<optimized out>) at back-sch.c:2083
#8  0x00007fac1353e62c in backend_write_cb (state=0x5635f1901bc0, state=0x5635f1901bc0, pb=0x7fab6c00a660) at back-sch.c:2077
#9  backend_betxn_pre_write_cb (pb=0x7fab6c00a660) at back-sch.c:2116
#10 0x00007fac22923da9 in plugin_call_func (list=0x5635f1911370, operation=operation@entry=461, pb=pb@entry=0x7fab6c00a660, 
    call_one=call_one@entry=0) at ldap/servers/slapd/plugin.c:2049
#11 0x00007fac22924014 in plugin_call_list (pb=0x7fab6c00a660, operation=461, list=<optimized out>) at ldap/servers/slapd/plugin.c:1993
#12 plugin_call_plugins (pb=pb@entry=0x7fab6c00a660, whichfunction=whichfunction@entry=461) at ldap/servers/slapd/plugin.c:445
#13 0x00007fac1550f0ae in ldbm_back_modify (pb=0x7fab6c00a660) at ldap/servers/slapd/back-ldbm/ldbm_modify.c:660
#14 0x00007fac22911632 in op_shared_modify (pb=pb@entry=0x7fab6c00a660, pw_change=pw_change@entry=0, old_pw=0x0)
    at ldap/servers/slapd/modify.c:1055
#15 0x00007fac229121c5 in modify_internal_pb (pb=pb@entry=0x7fab6c00a660) at ldap/servers/slapd/modify.c:599
#16 0x00007fac22912cdb in slapi_modify_internal_pb (pb=pb@entry=0x7fab6c00a660) at ldap/servers/slapd/modify.c:454
#17 0x00007fac14be40f9 in memberof_add_memberof_attr (mods=mods@entry=0x7fab6c0076f0, 
    dn=0x7fab6c008910 "krbprincipalname=ldap/<host_fqdn>@<domain>,cn=services,cn=accounts,<suffix>", add_oc=add_oc@entry=0x0)
    at ldap/servers/plugins/memberof/memberof.c:2936
#18 0x00007fac14be450c in memberof_fix_memberof_callback (e=<optimized out>, callback_data=callback_data@entry=0x7fabe4fe7e20)
    at ldap/servers/plugins/memberof/memberof.c:2904
#19 0x00007fac14be4f8f in memberof_modop_one_replace_r (pb=pb@entry=0x7fabe4fe8a30, config=config@entry=0x7fabe4fe7e20, 
    mod_op=mod_op@entry=0, group_sdn=group_sdn@entry=0x7fab6c000df0, op_this_sdn=op_this_sdn@entry=0x7fab6c000df0, 
    replace_with_sdn=replace_with_sdn@entry=0x0, op_to_sdn=0x7fab6c0083d0, stack=0x0) at ldap/servers/plugins/memberof/memberof.c:1690
#20 0x00007fac14be52b6 in memberof_modop_one_r (stack=0x0, op_to_sdn=<optimized out>, op_this_sdn=0x7fab6c000df0, 
    group_sdn=0x7fab6c000df0, mod_op=0, config=0x7fabe4fe7e20, pb=0x7fabe4fe8a30) at ldap/servers/plugins/memberof/memberof.c:1434
#21 memberof_modop_one (op_to_sdn=<optimized out>, op_this_sdn=0x7fab6c000df0, mod_op=0, config=0x7fabe4fe7e20, pb=0x7fabe4fe8a30)
    at ldap/servers/plugins/memberof/memberof.c:1421
#22 memberof_mod_smod_list (pb=pb@entry=0x7fabe4fe8a30, config=config@entry=0x7fabe4fe7e20, mod=mod@entry=0, group_sdn=0x7fab6c000df0, 
    smod=smod@entry=0x7fab6c007550) at ldap/servers/plugins/memberof/memberof.c:1798
#23 0x00007fac14be6605 in memberof_add_smod_list (smod=0x7fab6c007550, group_sdn=<optimized out>, config=0x7fabe4fe7e20, 
    pb=0x7fabe4fe8a30) at ldap/servers/plugins/memberof/memberof.c:1821
#24 memberof_postop_modify (pb=0x7fabe4fe8a30) at ldap/servers/plugins/memberof/memberof.c:1136
#25 0x00007fac22923da9 in plugin_call_func (list=0x5635f18e3a60, operation=operation@entry=561, pb=pb@entry=0x7fabe4fe8a30, 
    call_one=call_one@entry=0) at ldap/servers/slapd/plugin.c:2049
#26 0x00007fac22924014 in plugin_call_list (pb=0x7fabe4fe8a30, operation=561, list=<optimized out>) at ldap/servers/slapd/plugin.c:1993
#27 plugin_call_plugins (pb=pb@entry=0x7fabe4fe8a30, whichfunction=whichfunction@entry=561) at ldap/servers/slapd/plugin.c:445
#28 0x00007fac1550fc61 in ldbm_back_modify (pb=0x7fabe4fe8a30) at ldap/servers/slapd/back-ldbm/ldbm_modify.c:832
#29 0x00007fac22911632 in op_shared_modify (pb=pb@entry=0x7fabe4fe8a30, pw_change=pw_change@entry=0, old_pw=0x0)
    at ldap/servers/slapd/modify.c:1055
#30 0x00007fac22912a3c in do_modify (pb=pb@entry=0x7fabe4fe8a30) at ldap/servers/slapd/modify.c:387
#31 0x00005635f040e39d in connection_dispatch_operation (pb=0x7fabe4fe8a30, op=0x5635f1d5ede0, conn=0x7fac04329bd0)
    at ldap/servers/slapd/connection.c:627
#32 connection_threadmain () at ldap/servers/slapd/connection.c:1759
#33 0x00007fac20ae87df in _pt_root (arg=0x5635f1e53a10) at ../../../nspr/pr/src/pthreads/ptthread.c:216
#34 0x00007fac204875ca in start_thread (arg=0x7fabe4fe9700) at pthread_create.c:333
#35 0x00007fac201c0f6d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread deleting "cn=replication manager,cn=config"

[Current thread is 22 (Thread 0x7fabee7fc700 (LWP 66354))]
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007fac1998b643 in __db_pthread_mutex_condwait (env=0x5635f1b38650, mutex=140377252744070, timespec=0x0, mutexp=<optimized out>)
    at ../../src/mutex/mut_pthread.c:321
#2  __db_hybrid_mutex_suspend (env=env@entry=0x5635f1b38650, mutex=mutex@entry=2785, timespec=timespec@entry=0x0, 
    exclusive=exclusive@entry=1) at ../../src/mutex/mut_pthread.c:577
#3  0x00007fac1998aa27 in __db_tas_mutex_lock_int (nowait=0, timeout=0, mutex=2785, env=0x5635f1b38650) at ../../src/mutex/mut_tas.c:255
#4  __db_tas_mutex_lock (env=env@entry=0x5635f1b38650, mutex=2785, timeout=timeout@entry=0) at ../../src/mutex/mut_tas.c:286
#5  0x00007fac19a38070 in __lock_get_internal (lt=lt@entry=0x5635f1a590e0, sh_locker=sh_locker@entry=0x7fac0f6a49a8, 
    flags=flags@entry=0, obj=<optimized out>, lock_mode=<optimized out>, timeout=timeout@entry=0, lock=<optimized out>)
    at ../../src/lock/lock.c:989
#6  0x00007fac19a38ba7 in __lock_get (env=env@entry=0x5635f1b38650, locker=0x7fac0f6a49a8, flags=0, obj=obj@entry=0x7fab80015db0, 
    lock_mode=lock_mode@entry=DB_LOCK_READ, lock=lock@entry=0x7fabee7ed690) at ../../src/lock/lock.c:469
#7  0x00007fac19a6543f in __db_lget (dbc=dbc@entry=0x7fab80015cc0, action=action@entry=0, pgno=4, mode=mode@entry=DB_LOCK_READ, 
    lkflags=lkflags@entry=0, lockp=lockp@entry=0x7fabee7ed690) at ../../src/db/db_meta.c:1257
#8  0x00007fac199a985a in __bam_search (dbc=dbc@entry=0x7fab80015cc0, root_pgno=1, root_pgno@entry=0, key=key@entry=0x7fabee7ed960, 
    flags=<optimized out>, slevel=slevel@entry=1, recnop=recnop@entry=0x0, exactp=0x7fabee7ed7c4) at ../../src/btree/bt_search.c:723
#9  0x00007fac1999379b in __bamc_search (dbc=dbc@entry=0x7fab80015cc0, root_pgno=root_pgno@entry=0, key=0x7fabee7ed960, 
    flags=<optimized out>, exactp=<optimized out>) at ../../src/btree/bt_cursor.c:2804
#10 0x00007fac1999590f in __bamc_get (dbc=0x7fab80015cc0, key=<optimized out>, data=<optimized out>, flags=<optimized out>, 
    pgnop=0x7fabee7ed854) at ../../src/btree/bt_cursor.c:1099
#11 0x00007fac19a518a3 in __dbc_iget (dbc=0x7fab80015710, key=0x7fabee7ed960, data=0x7fabee7ed990, flags=26) at ../../src/db/db_cam.c:952
#12 0x00007fac19a5214d in __dbc_get (dbc=dbc@entry=0x7fab80015710, key=key@entry=0x7fabee7ed960, data=data@entry=0x7fabee7ed990, 
    flags=flags@entry=2074) at ../../src/db/db_cam.c:770
#13 0x00007fac19a611d6 in __dbc_get_pp (dbc=0x7fab80015710, key=0x7fabee7ed960, data=0x7fabee7ed990, flags=2074)
    at ../../src/db/db_iface.c:2361
#14 0x00007fac154dfa02 in idl_new_fetch (be=0x5635f19353e0, db=0x7fab80014e50, inkey=0x7fabee7efaf0, txn=0x0, a=0x5635f1b248d0, 
    flag_err=0x7fabee7f680c, allidslimit=4000) at ldap/servers/slapd/back-ldbm/idl_new.c:202
#15 0x00007fac154df6a5 in idl_fetch_ext (be=be@entry=0x5635f19353e0, db=<optimized out>, key=key@entry=0x7fabee7efaf0, 
    txn=txn@entry=0x0, a=<optimized out>, err=err@entry=0x7fabee7f680c, allidslimit=4000) at ldap/servers/slapd/back-ldbm/idl_shim.c:101
#16 0x00007fac154ee4f0 in index_read_ext_allids (pb=pb@entry=0x7fabc0001df0, be=be@entry=0x5635f19353e0, 
    type=type@entry=0x7fabc0007770 "member", indextype=indextype@entry=0x7fac1553674f "eq", val=<optimized out>, 
    txn=txn@entry=0x7fabee7f3dd0, err=0x7fabee7f680c, unindexed=0x7fabee7f3dc4, allidslimit=4000)
    at ldap/servers/slapd/back-ldbm/index.c:1028
#17 0x00007fac154d844d in keys2idl (pb=pb@entry=0x7fabc0001df0, be=be@entry=0x5635f19353e0, type=0x7fabc0007770 "member", 
    indextype=indextype@entry=0x7fac1553674f "eq", ivals=0x7fabee7f3eb0, err=err@entry=0x7fabee7f680c, unindexed=0x7fabee7f3dc4, 
    txn=0x7fabee7f3dd0, allidslimit=4000) at ldap/servers/slapd/back-ldbm/filterindex.c:986
#18 0x00007fac154d8cea in ava_candidates (pb=pb@entry=0x7fabc0001df0, be=be@entry=0x5635f19353e0, f=f@entry=0x7fabc00018b0, 
    ftype=<optimized out>, err=0x7fabee7f680c, allidslimit=4000, range=0, nextf=0x0) at ldap/servers/slapd/back-ldbm/filterindex.c:288
#19 0x00007fac154d92a9 in filter_candidates_ext (pb=pb@entry=0x7fabc0001df0, be=be@entry=0x5635f19353e0, 
    base=base@entry=0x7fabc00029c0 "cn=groups,cn=accounts,<suffix>", 
    f=f@entry=0x7fabc00018b0, nextf=nextf@entry=0x0, range=range@entry=0, err=0x7fabee7f680c, allidslimit=4000)
    at ldap/servers/slapd/back-ldbm/filterindex.c:111
#20 0x00007fac154da25f in list_candidates (pb=pb@entry=0x7fabc0001df0, be=be@entry=0x5635f19353e0, 
    base=base@entry=0x7fabc00029c0 "cn=groups,cn=accounts,<suffix>", 
    flist=flist@entry=0x7fabc0003110, ftype=161, err=0x7fabee7f680c, allidslimit=4000) at ldap/servers/slapd/back-ldbm/filterindex.c:817
#21 0x00007fac154d921a in filter_candidates_ext (pb=pb@entry=0x7fabc0001df0, be=be@entry=0x5635f19353e0, 
    n=groups,cn=accounts,<suffix>", 
    f=f@entry=0x7fabc0003110, nextf=nextf@entry=0x0, range=range@entry=0, err=0x7fabee7f680c, allidslimit=4000)
    at ldap/servers/slapd/back-ldbm/filterindex.c:144
#22 0x00007fac154da25f in list_candidates (pb=pb@entry=0x7fabc0001df0, be=be@entry=0x5635f19353e0, 
    base=base@entry=0x7fabc00029c0 "cn=groups,cn=accounts,<suffix>", 
    flist=flist@entry=0x7fabc00034b0, ftype=160, err=0x7fabee7f680c, allidslimit=4000) at ldap/servers/slapd/back-ldbm/filterindex.c:817
#23 0x00007fac154d921a in filter_candidates_ext (pb=pb@entry=0x7fabc0001df0, be=be@entry=0x5635f19353e0, 
    base=base@entry=0x7fabc00029c0 "cn=groups,cn=accounts,<suffix>", 
    f=0x7fabc00034b0, nextf=nextf@entry=0x0, range=range@entry=0, err=0x7fabee7f680c, allidslimit=4000)
    at ldap/servers/slapd/back-ldbm/filterindex.c:144
#24 0x00007fac154da82f in filter_candidates (pb=pb@entry=0x7fabc0001df0, be=be@entry=0x5635f19353e0, 
    base=base@entry=0x7fabc00029c0 "cn=groups,cn=accounts,<suffix>", 
    f=<optimized out>, nextf=nextf@entry=0x0, range=range@entry=0, err=0x7fabee7f680c) at ldap/servers/slapd/back-ldbm/filterindex.c:175
#25 0x00007fac1551729b in onelevel_candidates (err=0x7fabee7f680c, lookup_returned_allidsp=0x7fabee7f67fc, managedsait=<optimized out>, 
    filter=<optimized out>, e=0x5635f1d923b0, 
    base=0x7fabc00029c0 "cn=groups,cn=accounts,<suffix>", be=0x5635f19353e0, 
    pb=0x7fabc0001df0) at ldap/servers/slapd/back-ldbm/ldbm_search.c:1143
#26 build_candidate_list (candidates=0x7fabee7f6838, lookup_returned_allidsp=0x7fabee7f67fc, scope=1, 
    base=0x7fabc00029c0 "cn=groups,cn=accounts,<suffix>", e=<optimized out>, 
    be=0x5635f19353e0, pb=0x7fabc0001df0) at ldap/servers/slapd/back-ldbm/ldbm_search.c:1010
#27 ldbm_back_search (pb=0x7fabc0001df0) at ldap/servers/slapd/back-ldbm/ldbm_search.c:657
#28 0x00007fac22918420 in op_shared_search (pb=pb@entry=0x7fabc0001df0, send_result=send_result@entry=1)
    at ldap/servers/slapd/opshared.c:807
#29 0x00007fac2292900e in search_internal_callback_pb (pb=pb@entry=0x7fabc0001df0, callback_data=callback_data@entry=0x7fabee7fb1a0, 
    prc=prc@entry=0x0, psec=psec@entry=0x7fac13540450 <backend_shr_note_entry_sdn_cb>, prec=prec@entry=0x0)
    at ldap/servers/slapd/plugin_internal_op.c:783
#30 0x00007fac22929541 in slapi_search_internal_callback_pb (pb=pb@entry=0x7fabc0001df0, 
    callback_data=callback_data@entry=0x7fabee7fb1a0, prc=prc@entry=0x0, 
    psec=psec@entry=0x7fac13540450 <backend_shr_note_entry_sdn_cb>, prec=prec@entry=0x0) at ldap/servers/slapd/plugin_internal_op.c:564
#31 0x00007fac13541e79 in backend_shr_update_references_cb (
    group=0x7fabd4007ed0 "cn=compat,<suffix>", set=0x7fabd400a970 "cn=groups", 
    flag=<optimized out>, backend_data=0x7fabd400f3e0, cbdata_ptr=0x7fabee7fb290) at back-shr.c:1539
#32 0x00007fac1354e4ff in map_data_foreach_map (state=state@entry=0x5635f1901bc0, domain_name=domain_name@entry=0x0, 
    fn=fn@entry=0x7fac13541520 <backend_shr_update_references_cb>, cbdata=cbdata@entry=0x7fabee7fb290) at map.c:347
#33 0x00007fac1353fe3b in backend_shr_update_references (state=0x5635f1901bc0, pb=pb@entry=0x7fabee7fba30, e=<optimized out>, 
    mods=mods@entry=0x0, modlist=modlist@entry=0x0) at back-shr.c:1825
#34 0x00007fac1354093f in backend_shr_delete_cb (pb=0x7fabee7fba30) at back-shr.c:2707
#35 backend_shr_internal_post_delete_cb (pb=0x7fabee7fba30) at back-shr.c:2735
#36 0x00007fac13540a67 in backend_shr_betxn_post_delete_cb (pb=0x7fabee7fba30) at back-shr.c:2719
#37 0x00007fac22923da9 in plugin_call_func (list=0x5635f19121d0, operation=operation@entry=563, pb=pb@entry=0x7fabee7fba30, 
    call_one=call_one@entry=0) at ldap/servers/slapd/plugin.c:2049
#38 0x00007fac22924014 in plugin_call_list (pb=0x7fabee7fba30, operation=563, list=<optimized out>) at ldap/servers/slapd/plugin.c:1993
#39 plugin_call_plugins (pb=pb@entry=0x7fabee7fba30, whichfunction=whichfunction@entry=563) at ldap/servers/slapd/plugin.c:445
#40 0x00007fac228df105 in dse_delete (pb=0x7fabee7fba30) at ldap/servers/slapd/dse.c:2537
#41 0x00007fac228d4ba0 in op_shared_delete (pb=pb@entry=0x7fabee7fba30) at ldap/servers/slapd/delete.c:333
#42 0x00007fac228d4e83 in do_delete (pb=pb@entry=0x7fabee7fba30) at ldap/servers/slapd/delete.c:97
#43 0x00005635f040e30c in connection_dispatch_operation (pb=0x7fabee7fba30, op=0x5635f1e958e0, conn=0x7fac0432a008)
    at ldap/servers/slapd/connection.c:617
#44 connection_threadmain () at ldap/servers/slapd/connection.c:1759
#45 0x00007fac20ae87df in _pt_root (arg=0x5635f1e4ffe0) at ../../../nspr/pr/src/pthreads/ptthread.c:216
#46 0x00007fac204875ca in start_thread (arg=0x7fabee7fc700) at pthread_create.c:333
#47 0x00007fac201c0f6d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Metadata Update from @tbordaz:
- Issue assigned to someone
- Issue set to the milestone: FreeIPA 4.5 backlog
Metadata
None
None
None
Normal
None
defect
DS
0
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.