389-ds-base

#629 Contention on config lock while sending data
Closed: wontfix Opened by tbordaz.

Closed: wontfix
Reopen Issue

There is a contention on the configuration lock that impacts all the threads sending back data.
This contention looks useless because it is created to access the ioblocktimeout config parameter. Even if the config paramater is about to be change, using the previous value or the next one for a single send of data looks valid.

test case:[[BR]]

- create 1000 entries
- set nsslapd-threadnumber to a large number greater than the number of cores

(for example 60 on a 8*4cores machine)
- use rsearch so that each search returns all 1000 entries (possibly only one attribute is fine), using the same number of thread that nsslapd-threadnumber
- using dbx dump the address of : print global_slapdFrontendConfig->cfg_rwlock
- tune a stap script to take backtrace when there is contention on that address ($uaddr)

results are:[[BR]]

lock 0x1cee410 contended 96 times, 288 avg us
Histogram linear 0x1cee410
value |-------------------------------------------------- count
    0 |                                                    0
   50 |                                                    0
  100 |@@@                                                 6
  150 |@@@@@                                              11
  200 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                 68
  250 |@@                                                  4
  300 |@                                                   2
  350 |                                                    0
  400 |                                                    0
  450 |                                                    0
  500 |                                                    0
 >500 |@@                                                  5
Stack __lll_lock_wait+0x24 [libpthread-2.12.so]
__pthread_rwlock_rdlock+0x74 [libpthread-2.12.so]
config_get_ioblocktimeout+0x20 [libslapd.so.0.0.0]
openldap_write_function+0x5d [ns-slapd]
ber_int_sb_write+0x4f [liblber-2.4.so.2.5.6]
ber_flush2+0xcb [liblber-2.4.so.2.5.6]
flush_ber+0x148 [libslapd.so.0.0.0]
send_ldap_search_entry_ext+0xf74 [libslapd.so.0.0.0]
send_ldap_search_entry+0x1c [libslapd.so.0.0.0]
iterate.clone.2+0x166 [libslapd.so.0.0.0]
send_results_ext.clone.0+0x57 [libslapd.so.0.0.0]
op_shared_search+0xcab [libslapd.so.0.0.0]
do_search+0xa94 [ns-slapd]
connection_threadmain+0xa7a [ns-slapd]
PR_JoinThread+0x143 [libnspr4.so]
start_thread+0xd1 [libpthread-2.12.so]
__clone+0x6d [libc-2.12.so]
        =>79
Stack __lll_lock_wait+0x24 [libpthread-2.12.so]
pthread_rwlock_unlock+0x80 [libpthread-2.12.so]
config_get_ioblocktimeout+0x2b [libslapd.so.0.0.0]
openldap_write_function+0x5d [ns-slapd]
ber_int_sb_write+0x4f [liblber-2.4.so.2.5.6]
ber_flush2+0xcb [liblber-2.4.so.2.5.6]
flush_ber+0x148 [libslapd.so.0.0.0]
send_ldap_search_entry_ext+0xf74 [libslapd.so.0.0.0]
send_ldap_search_entry+0x1c [libslapd.so.0.0.0]
iterate.clone.2+0x166 [libslapd.so.0.0.0]
send_results_ext.clone.0+0x57 [libslapd.so.0.0.0]
op_shared_search+0xcab [libslapd.so.0.0.0]
do_search+0xa94 [ns-slapd]
connection_threadmain+0xa7a [ns-slapd]
PR_JoinThread+0x143 [libnspr4.so]
start_thread+0xd1 [libpthread-2.12.so]
__clone+0x6d [libc-2.12.so]
        =>15

94 out of 96 stacks are showing contention to read ioblocktimeout.
This is useless because it is not that important to use an exact value when this value is changed

Suggested fix:[[BR]]

I think we can safely get rid of the config read lock while reading ioblocktimeout.
Using the previous value is not a big deal when we change it.

Note: This should also apply for other config paramaters

I agree that a rwlock is overkill for getting/setting the ioblocktimeout parameter, but there is a chance that we can read a partially updated value if we don't have any protection. To prevent this, we could use atomic get/set operations.

It looks like the code has logic for using atomic operations, but it's not being used for some reason. From a quick check, it looks like ATOMIC_GETSET_IOBLOCKTIMEOUT should be defined, which would avoid using the rwlock. Your stack traces above show that the rwlock is being used though. If there is some reason we can't use the ATOMIC_GETSET_IOBLOCKTIMEOUT code, we could use the gcc __sync_bool_compare_and_swap() atomic built-in function to write our own atomic get/set functions. There are examples of this in the Slapi_Counter code.

If it is 32 bits or less, sizeof(PRInt32), by default all access and update is atomic, and is guaranteed to have atomic write with the use of PR_AtomicSet. Unless a 64-bit value is needed, we don't have to use sync_bool_compare_and_swap for ioblocktimeout.

If you are using 1.2.11, that explains why there is so much contention - in current master code, atomics are used for ioblocktimeout.

This was fixed in ticket #508. Closing this ticket as a duplicate.

Metadata Update from @rmeggins:
- Issue set to the milestone: N/A

389-ds-base is moving from Pagure to Github. This means that new issues and pull requests
will be accepted only in 389-ds-base's github repository.

This issue has been cloned to Github and is available here:
- https://github.com/389ds/389-ds-base/issues/629

If you want to receive further updates on the issue, please navigate to the github issue
and click on subscribe button.

Thank you for understanding. We apologize for all inconvenience.

Metadata Update from @spichugi:
- Issue close_status updated to: wontfix (was: Duplicate)
Metadata
None
None
None
None
High
Community
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.