ipa-server-4.4.0-14.el7_3.7.x86_64 slapi-nis-0.56.0-4.el7.x86_64
During installation of a freeipa replica, slapi-nis can store empty sdn in a map structure used to update references. The sdn are used as base search for internal searches.
Because of bug https://pagure.io/389-ds-base/issue/49291, it triggers that DS crash. (see backstack below)
This bug is also related to https://pagure.io/slapi-nis/issue/16
The bug is that slapi-nis should ignore empty sdn.
Thread 1 (Thread 0x7fa53bfe7700 (LWP 18900)): #0 0x00007fa579a22711 in send_ldap_result_ext (pb=pb@entry=0x7fa528007550, err=err@entry=53, matched=matched@entry=0x0, text=text@entry=0x7fa579a4c708 "This plugin is not configured to access operation target data", nentries=nentries@entry=0, urls=urls@entry=0x0, ber=ber@entry=0x0) at ldap/servers/slapd/result.c:353 #1 0x00007fa579a22fd1 in send_ldap_result (pb=pb@entry=0x7fa528007550, err=err@entry=53, matched=matched@entry=0x0, text=text@entry=0x7fa579a4c708 "This plugin is not configured to access operation target data", nentries=nentries@entry=0, urls=urls@entry=0x0) at ldap/servers/slapd/result.c:194 #2 0x00007fa579a13260 in slapi_search_internal_callback_pb (pb=pb@entry=0x7fa528007550, callback_data=callback_data@entry=0x7fa53bfe5ea0, prc=prc@entry=0x0, psec=psec@entry=0x7fa56a802df0 <backend_shr_note_entry_sdn_cb>, prec=prec@entry=0x0) at ldap/servers/slapd/plugin_internal_op.c:559 #3 0x00007fa56a804a61 in backend_shr_update_references_cb (group=0x7fa520007d50 "cn=compat,<suffix>", set=0x7fa52000aa50 "cn=groups", flag=<optimized out>, backend_data=0x7fa52392daa0, cbdata_ptr=<optimized out>) at back-shr.c:1539 #4 0x00007fa56a81203f in map_data_foreach_map (state=state@entry=0x7fa57b585a30, domain_name=domain_name@entry=0x0, fn=fn@entry=0x7fa56a803fd0 <backend_shr_update_references_cb>, cbdata=cbdata@entry=0x7fa53bfe5f50) at map.c:347 #5 0x00007fa56a80279b in backend_shr_update_references (state=0x7fa57b585a30, pb=pb@entry=0x7fa53bfe6a90, e=<optimized out>, mods=mods@entry=0x0, modlist=modlist@entry=0x0) at back-shr.c:1825 #6 0x00007fa56a803a17 in backend_shr_add_cb (pb=pb@entry=0x7fa53bfe6a90) at back-shr.c:1953 #7 0x00007fa56a803b41 in backend_shr_add_cb (pb=0x7fa53bfe6a90) at back-shr.c:1883 #8 backend_shr_betxn_post_add_cb (pb=0x7fa53bfe6a90) at back-shr.c:1965 #9 0x00007fa579a0db18 in plugin_call_func (list=0x7fa57b58e910, operation=operation@entry=560, pb=pb@entry=0x7fa53bfe6a90, call_one=call_one@entry=0) at ldap/servers/slapd/plugin.c:2049 #10 0x00007fa579a0dda3 in plugin_call_list (pb=0x7fa53bfe6a90, operation=560, list=<optimized out>) at ldap/servers/slapd/plugin.c:1993 #11 plugin_call_plugins (pb=pb@entry=0x7fa53bfe6a90, whichfunction=whichfunction@entry=560) at ldap/servers/slapd/plugin.c:445 #12 0x00007fa56c7b9b94 in ldbm_back_add (pb=0x7fa53bfe6a90) at ldap/servers/slapd/back-ldbm/ldbm_add.c:1150 #13 0x00007fa5799b1ed8 in op_shared_add (pb=pb@entry=0x7fa53bfe6a90) at ldap/servers/slapd/add.c:699 #14 0x00007fa5799b31a0 in do_add (pb=pb@entry=0x7fa53bfe6a90) at ldap/servers/slapd/add.c:226 #15 0x00007fa579ee59c3 in connection_dispatch_operation (pb=0x7fa53bfe6a90, op=0x7fa57b7521b0, conn=0x7fa57bc05118) at ldap/servers/slapd/connection.c:612 #16 connection_threadmain () at ldap/servers/slapd/connection.c:1759 #17 0x00007fa577bdb9bb in _pt_root (arg=0x7fa57bbcf3d0) at ../../../nspr/pr/src/pthreads/ptthread.c:216 #18 0x00007fa57757bdc5 in start_thread (arg=0x7fa53bfe7700) at pthread_create.c:308 #19 0x00007fa5772aa76d in __lseek_nocancel () at ../sysdeps/unix/syscall-template.S:81 #20 0x0000000000000000 in ?? ()
The crash occurred while only one thread was active
The reason of the crash is https://pagure.io/slapi-nis/issue/16 But the conditions that lead to that crash are the purpose of this ticket
slapi-nis maps two domains
# 2 domains: # compat # sudoers # (gdb) print map_data $1 = {master = 0x0, domains = 0x7fa523d3bb80, n_domains = 2, lock = 0x7fa57b58c2f0} (gdb) print map_data.domains[0] $2 = {name = 0x7fa520007d50 "cn=compat,<suffix>", maps = 0x7fa51339ce20, n_maps = 4} (gdb) print map_data.domains[1] $4 = {name = 0x7fa510f039b0 "ou=sudoers,<suffix>", maps = 0x7fa520006a30, n_maps = 1}
Domain 'compat' has 4 maps
# Domain: compat # map: cn=users # map: cn=ng # map: cn=groups # map: cn=computers # (gdb) print ((struct map) map_data.domains[0]->maps[0]) $9 = {name = 0x7fa520003850 "cn=users", last_changed = 1497378023, secure = 0, entries = 0x7fa51075fda0, n_unique_entries = 10448, n_key_trees = 1, key_trees = 0x7fa520263f00, id_tree = 0x7fa520265a60, backend_data = 0x7fa520001b60, free_backend_data = 0x7fa56a8024c0 <backend_shr_set_config_free_config>} (gdb) print ((struct map) map_data.domains[0]->maps[1]) $10 = {name = 0x7fa520008420 "cn=ng", last_changed = 1497378024, secure = 0, entries = 0x7fa5100828c0, n_unique_entries = 38, n_key_trees = 1, key_trees = 0x7fa523e09020, id_tree = 0x7fa510bd20b0, backend_data = 0x7fa510b94070, free_backend_data = 0x7fa56a8024c0 <backend_shr_set_config_free_config>} (gdb) print ((struct map) map_data.domains[0]->maps[2]) $11 = {name = 0x7fa52000aa50 "cn=groups", last_changed = 1497378037, secure = 0, entries = 0x7fa511decbc0, n_unique_entries = 6823, n_key_trees = 1, key_trees = 0x7fa510f8b130, id_tree = 0x7fa5120fa090, backend_data = 0x7fa52392daa0, free_backend_data = 0x7fa56a8024c0 <backend_shr_set_config_free_config>} (gdb) print ((struct map) map_data.domains[0]->maps[3]) $12 = {name = 0x7fa5104585c0 "cn=computers", last_changed = 1497378037, secure = 0, entries = 0x0, n_unique_entries = 0, n_key_trees = 0, key_trees = 0x0, id_tree = 0x0, backend_data = 0x7fa5122c4730, free_backend_data = 0x7fa56a8024c0 <backend_shr_set_config_free_config>}
The crash occurred while evaluating references of 'member' attribute in 'cn=groups' map
(gdb) print *((struct backend_shr_set_data *)((struct map) map_data.domains[0]->maps[2])->backend_data) $34 = {state = 0x7fa57b585a30, group = 0x7fa521a75df0 "cn=compat,<suffix>", set = 0x7fa5238e28a0 "cn=groups", bases = 0x7fa5238ea0c0, entry_filter = 0x7fa520008220 "(objectclass=posixGroup)", rel_attrs = 0x7fa523eae650, rel_attr_list = 0x7fa528007860 "cn,ipaanchoruuid,gidNumber,memberUid,ipauniqueid,member,uid", rel_attrs_list = 0x7fa523eae650, ref_attrs = 0x0, inref_attrs = 0x0, ref_attr_list = 0x7fa510582480, inref_attr_list = 0x0, skip_uninteresting_updates = 1, restrict_subtrees = 0x0, ignore_subtrees = 0x7fa523917fd0, self = 0x7fa52392daa0} # only one attribute in that map (gdb) print *((struct backend_shr_set_data *)((struct map) map_data.domains[0]->maps[2])->backend_data)->ref_attr_list[0] $35 = {group = 0x7fa523eae720 "cn=compat,<suffix>", set = 0x7fa523eae750 "cn=groups", links = 0x7fa510458fe0, n_links = 2} (gdb) print *((struct backend_shr_set_data *)((struct map) map_data.domains[0]->maps[2])->backend_data)->ref_attr_list[1] Cannot access memory at address 0x0 # attribute is 'member' (gdb) print *((struct backend_shr_set_data *)((struct map) map_data.domains[0]->maps[2])->backend_data)->ref_attr_list[0]->links $36 = {attribute = 0x7fa523eae6e0 "member", filter_str = 0x0, filter = 0x0, base_sdn_list = 0x7fa512b598e0, base_sdn_list2 = 0x7fa512b59910}
The crash occurred because the base sdn list, that should be NULL terminated, contains an empty base
# base suffixes # "cn=groups,cn=accounts,<suffix>" (gdb) print *((struct backend_shr_set_data *)((struct map) map_data.domains[0]->maps[2])->backend_data)-> ref_attr_list[0]->links->base_sdn_list[0] $39 = {flag = 15 '\017', udn = 0x7fa510f8b210 "cn=groups,cn=accounts,<suffix>", dn = 0x7fa523e64b60 "cn=groups,cn=accounts,<suffix>", ndn = 0x7fa523e64a20 "cn=groups,cn=accounts,<suffix>", ndn_len = 37} # "cn=users,cn=accounts,<suffix>" (gdb) print *((struct backend_shr_set_data *)((struct map) map_data.domains[0]->maps[2])->backend_data)-> ref_attr_list[0]->links->base_sdn_list[1] $40 = {flag = 15 '\017', udn = 0x7fa523e64b30 "cn=users,cn=accounts,<suffix>", dn = 0x7fa5102f3200 "cn=users,cn=accounts,<suffix>", ndn = 0x7fa5102f30e0 "cn=users,cn=accounts,<suffix>", ndn_len = 36} # empty sdn (gdb) print *((struct backend_shr_set_data *)((struct map) map_data.domains[0]->maps[2])->backend_data)-> ref_attr_list[0]->links->base_sdn_list[2] $41 = {flag = 9 '\t', udn = 0x0, dn = 0x0, ndn = 0x0, ndn_len = 0}
The base sdn has been added with format_add_sdn_list. This fonction adds the sdn in both lists base_sdn_list (unsorted) and base_sdn_list2 (sorted)
# # unsorted list gdb) print ((struct backend_shr_set_data *)((struct map) map_data.domains[0]->maps[2])->backend_data)-> ref_attr_list[0]->links[0]->base_sdn_list[0] $1 = (struct slapi_dn *) 0x7fa510f8b1e0 (gdb) print ((struct backend_shr_set_data *)((struct map) map_data.domains[0]->maps[2])->backend_data)-> ref_attr_list[0]->links[0]->base_sdn_list[1] $2 = (struct slapi_dn *) 0x7fa523e64c50 (gdb) print ((struct backend_shr_set_data *)((struct map) map_data.domains[0]->maps[2])->backend_data)-> ref_attr_list[0]->links[0]->base_sdn_list[2] $3 = (struct slapi_dn *) 0x7fa512b59a70 # # unsorted list contains the same sdn address (gdb) print ((struct backend_shr_set_data *)((struct map) map_data.domains[0]->maps[2])->backend_data)-> ref_attr_list[0]->links[0]->base_sdn_list2[0] $4 = (struct slapi_dn *) 0x7fa512b59a70 (gdb) print ((struct backend_shr_set_data *)((struct map) map_data.domains[0]->maps[2])->backend_data)-> ref_attr_list[0]->links[0]->base_sdn_list2[1] $5 = (struct slapi_dn *) 0x7fa510f8b1e0 (gdb) print ((struct backend_shr_set_data *)((struct map) map_data.domains[0]->maps[2])->backend_data)-> ref_attr_list[0]->links[0]->base_sdn_list2[2] $6 = (struct slapi_dn *) 0x7fa523e64c50
So there was no corruption of the lists
Looking at all format_add_sdn_list calls, nothing prevents adding an empty sdn. No obvious bug found while investigating those calls.
Nothing in error logs can reveal when empty sdn was added
In conclusion: The crash occurred because at some time format_add_sdn_list was called to add an empty sdn to the 'member' attribute of the 'cn=groups' map. No tips to know when/why it happened
Next steps
We can inverstigate more the conditions at each calls of format_add_sdn_list but it would be very time consuming without guarantie to find the RC
I suggest to harden format_add_sdn_list, so that it does not add empty sdn
A reproducible test case should be to create a group entry, that slapi-nis will map, with a corrupted member attribute that contains an empty 'member' value. Details are attached in the case