389-ds-base

#49625 TSAN warnings generated by suits/basic
Closed: wontfix by spichugi. Opened by vashirov.

Closed: wontfix
Reopen Issue

Issue Description

I ran suits/basic with TSAN-enabled build. There are many warnings reported by TSAN, in the attached logs there are some duplicates. Tests didn't finish, server hangs during the import tests.

Package Version and Platform

389-ds-base-1.4.0.6-20180322git5ba01818f.fc27.x86_64

Steps to reproduce

  1. Build with --enable-tsan
  2. setenforce 0 (without it SELinux prevents TSAN to work)
  3. Edit /usr/lib/systemd/system/dirsrv@.service to contain the following:
Environment=TSAN_OPTIONS=log_path=/var/run/dirsrv/ns-slapd-%i.tsan:print_stacktrace=1:second_deadlock_stack=1:history_size=7:exitcode=0
  1. Run tests like this:
TSAN_OPTIONS="log_path=/var/run/dirsrv/ns-slapd.tsan print_stacktrace=1 second_deadlock_stack=1 history_size=7 exitcode=0" DEBUGGING=True py.test -s -v ds/dirsrvtests/tests/suites/basic/

TSAN_OPTIONS is needed here for setup-ds.pl, since by default TSAN-enabled ns-slapd exits with 1 if any warnings were found and setup-ds.pl fails.
5. Logs can be found at /var/run/dirsrv/
I'm attaching logs as well.

tsan_logs.tar.gz

Metadata Update from @vashirov:
- Custom field component adjusted to None
- Custom field origin adjusted to None
- Custom field reviewstatus adjusted to None
- Custom field type adjusted to None
- Custom field version adjusted to None

All the results are referring to the server starting up, and an online import. Is that just because that's all the server did? Or were other operations/tasks performed?

DS hung during ds/dirsrvtests/tests/suites/basic/basic_test.py::test_basic_import_export
I couldn't get past it. Now I commented it out and rerun again, new logs are attached.
tsan_logs2.tar.gz

And another data point, failed test_nuncstans_stress_small during rpm build (make check):

==================
WARNING: ThreadSanitizer: data race (pid=349)
  Atomic write of size 8 at 0x558e8d1a0040 by thread T2 (mutexes: write M839):
    #0 __tsan_atomic64_fetch_add <null> (libtsan.so.0+0x000000064c50)
    #1 client_response_cb src/nunc-stans/test/test_nuncstans_stress_core.c:364 (lt-test_nuncstans_stress_small+0x00000000234e)
    #2 work_job_execute src/nunc-stans/ns/ns_thrpool.c:291 (libnunc-stans.so.0+0x0000000044a2)
    #3 worker_thread_func src/nunc-stans/ns/ns_thrpool.c:405 (libnunc-stans.so.0+0x00000000489d)
    #4 <null> <null> (libtsan.so.0+0x0000000257eb)
  Previous read of size 8 at 0x558e8d1a0040 by thread T3 (mutexes: write M236574478065304368):
    #0 client_response_cb src/nunc-stans/test/test_nuncstans_stress_core.c:368 (lt-test_nuncstans_stress_small+0x00000000235a)
    #1 work_job_execute src/nunc-stans/ns/ns_thrpool.c:291 (libnunc-stans.so.0+0x0000000044a2)
    #2 worker_thread_func src/nunc-stans/ns/ns_thrpool.c:405 (libnunc-stans.so.0+0x00000000489d)
    #3 <null> <null> (libtsan.so.0+0x0000000257eb)
  Location is global 'client_success_count' of size 8 at 0x558e8d1a0040 (lt-test_nuncstans_stress_small+0x000000205040)
  Mutex M839 (0x7b3c00010860) created at:
    #0 pthread_mutex_init <null> (libtsan.so.0+0x00000002971e)
    #1 new_ns_job src/nunc-stans/ns/ns_thrpool.c:700 (libnunc-stans.so.0+0x000000004a1e)
    #2 alloc_io_context src/nunc-stans/ns/ns_thrpool.c:726 (libnunc-stans.so.0+0x000000004cd9)
    #3 ns_add_io_job src/nunc-stans/ns/ns_thrpool.c:858 (libnunc-stans.so.0+0x000000004cd9)
    #4 client_initiate_connection_cb src/nunc-stans/test/test_nuncstans_stress_core.c:419 (lt-test_nuncstans_stress_small+0x000000002190)
    #5 work_job_execute src/nunc-stans/ns/ns_thrpool.c:291 (libnunc-stans.so.0+0x0000000044a2)
    #6 worker_thread_func src/nunc-stans/ns/ns_thrpool.c:405 (libnunc-stans.so.0+0x00000000489d)
    #7 <null> <null> (libtsan.so.0+0x0000000257eb)
  Mutex M236574478065304368 is already destroyed.
  Thread T2 (tid=370, running) created by main thread at:
    #0 pthread_create <null> (libtsan.so.0+0x000000028e03)
    #1 ns_thrpool_new src/nunc-stans/ns/ns_thrpool.c:1462 (libnunc-stans.so.0+0x000000005c75)
    #2 ns_stress_test src/nunc-stans/test/test_nuncstans_stress_core.c:470 (lt-test_nuncstans_stress_small+0x000000002e7c)
    #3 <null> <null> (libcmocka.so.0+0x000000004ae8)
    #4 __libc_start_main <null> (libc.so.6+0x000000020f29)
  Thread T3 (tid=371, running) created by main thread at:
    #0 pthread_create <null> (libtsan.so.0+0x000000028e03)
    #1 ns_thrpool_new src/nunc-stans/ns/ns_thrpool.c:1462 (libnunc-stans.so.0+0x000000005c75)
    #2 ns_stress_test src/nunc-stans/test/test_nuncstans_stress_core.c:470 (lt-test_nuncstans_stress_small+0x000000002e7c)
    #3 <null> <null> (libcmocka.so.0+0x000000004ae8)
    #4 __libc_start_main <null> (libc.so.6+0x000000020f29)
SUMMARY: ThreadSanitizer: data race (/lib64/libtsan.so.0+0x64c50) in __tsan_atomic64_fetch_add
==================
==================
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=349)
  Cycle in lock order graph: M46 (0x7b3c000000f0) => M45 (0x7b0c000002d8) => M46
  Mutex M45 acquired here while holding mutex M46 in thread T7:
    #0 pthread_mutex_lock <null> (libtsan.so.0+0x00000003b62e)
    #1 sds_tqueue_dequeue src/libsds/sds/queue/tqueue.c:50 (libsds.so.0+0x00000000d6a7)
    #2 sds_lqueue_dequeue src/libsds/sds/queue/lqueue.c:243 (libsds.so.0+0x00000000c3e0)
    #3 get_new_event_requests src/nunc-stans/ns/ns_thrpool.c:554 (libnunc-stans.so.0+0x000000003fab)
    #4 wakeup_cb src/nunc-stans/ns/ns_thrpool.c:671 (libnunc-stans.so.0+0x0000000041fe)
    #5 work_job_execute src/nunc-stans/ns/ns_thrpool.c:291 (libnunc-stans.so.0+0x0000000044a2)
    #6 event_cb src/nunc-stans/ns/ns_thrpool.c:642 (libnunc-stans.so.0+0x0000000046f4)
    #7 event_cb src/nunc-stans/ns/ns_event_fw_event.c:118 (libnunc-stans.so.0+0x0000000065f3)
    #8 event_base_loop <null> (libevent-2.0.so.5+0x000000010f18)
    #9 event_loop_thread_func src/nunc-stans/ns/ns_thrpool.c:581 (libnunc-stans.so.0+0x0000000040c3)
    #10 <null> <null> (libtsan.so.0+0x0000000257eb)
  Mutex M46 previously acquired by the same thread here:
    #0 pthread_mutex_lock <null> (libtsan.so.0+0x00000003b62e)
    #1 work_job_execute src/nunc-stans/ns/ns_thrpool.c:284 (libnunc-stans.so.0+0x00000000444e)
    #2 event_cb src/nunc-stans/ns/ns_thrpool.c:642 (libnunc-stans.so.0+0x0000000046f4)
    #3 event_cb src/nunc-stans/ns/ns_event_fw_event.c:118 (libnunc-stans.so.0+0x0000000065f3)
    #4 event_base_loop <null> (libevent-2.0.so.5+0x000000010f18)
    #5 event_loop_thread_func src/nunc-stans/ns/ns_thrpool.c:581 (libnunc-stans.so.0+0x0000000040c3)
    #6 <null> <null> (libtsan.so.0+0x0000000257eb)
  Mutex M46 acquired here while holding mutex M45 in main thread:
    #0 pthread_mutex_lock <null> (libtsan.so.0+0x00000003b62e)
    #1 internal_ns_job_done src/nunc-stans/ns/ns_thrpool.c:217 (libnunc-stans.so.0+0x0000000035f7)
    #2 job_queue_cleanup src/nunc-stans/ns/ns_thrpool.c:210 (libnunc-stans.so.0+0x000000004987)
    #3 sds_queue_destroy src/libsds/sds/queue/queue.c:118 (libsds.so.0+0x00000000c2e3)
    #4 sds_tqueue_destroy src/libsds/sds/queue/tqueue.c:60 (libsds.so.0+0x00000000d724)
    #5 sds_lqueue_destroy src/libsds/sds/queue/lqueue.c:249 (libsds.so.0+0x00000000c41c)
    #6 ns_thrpool_destroy src/nunc-stans/ns/ns_thrpool.c:1532 (libnunc-stans.so.0+0x000000005649)
    #7 ns_stress_test src/nunc-stans/test/test_nuncstans_stress_core.c:540 (lt-test_nuncstans_stress_small+0x00000000333f)
    #8 <null> <null> (libcmocka.so.0+0x000000004ae8)
    #9 __libc_start_main <null> (libc.so.6+0x000000020f29)
  Mutex M45 previously acquired by the same thread here:
    #0 pthread_mutex_lock <null> (libtsan.so.0+0x00000003b62e)
    #1 sds_tqueue_destroy src/libsds/sds/queue/tqueue.c:59 (libsds.so.0+0x00000000d714)
    #2 sds_lqueue_destroy src/libsds/sds/queue/lqueue.c:249 (libsds.so.0+0x00000000c41c)
    #3 ns_thrpool_destroy src/nunc-stans/ns/ns_thrpool.c:1532 (libnunc-stans.so.0+0x000000005649)
    #4 ns_stress_test src/nunc-stans/test/test_nuncstans_stress_core.c:540 (lt-test_nuncstans_stress_small+0x00000000333f)
    #5 <null> <null> (libcmocka.so.0+0x000000004ae8)
    #6 __libc_start_main <null> (libc.so.6+0x000000020f29)
  Thread T7 (tid=375, finished) created by main thread at:
    #0 pthread_create <null> (libtsan.so.0+0x000000028e03)
    #1 ns_thrpool_new src/nunc-stans/ns/ns_thrpool.c:1466 (libnunc-stans.so.0+0x000000005eb7)
    #2 ns_stress_test src/nunc-stans/test/test_nuncstans_stress_core.c:481 (lt-test_nuncstans_stress_small+0x000000002ec3)
    #3 <null> <null> (libcmocka.so.0+0x000000004ae8)
    #4 __libc_start_main <null> (libc.so.6+0x000000020f29)
SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) (/lib64/libtsan.so.0+0x3b62e) in pthread_mutex_lock
==================
==================
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=349)
  Cycle in lock order graph: M40 (0x7b3c00000000) => M13 (0x7b0c00000068) => M40
  Mutex M13 acquired here while holding mutex M40 in thread T5:
    #0 pthread_mutex_lock <null> (libtsan.so.0+0x00000003b62e)
    #1 sds_tqueue_dequeue src/libsds/sds/queue/tqueue.c:50 (libsds.so.0+0x00000000d6a7)
    #2 sds_lqueue_dequeue src/libsds/sds/queue/lqueue.c:243 (libsds.so.0+0x00000000c3e0)
    #3 get_new_event_requests src/nunc-stans/ns/ns_thrpool.c:554 (libnunc-stans.so.0+0x000000003fab)
    #4 wakeup_cb src/nunc-stans/ns/ns_thrpool.c:671 (libnunc-stans.so.0+0x0000000041fe)
    #5 work_job_execute src/nunc-stans/ns/ns_thrpool.c:291 (libnunc-stans.so.0+0x0000000044a2)
    #6 event_cb src/nunc-stans/ns/ns_thrpool.c:642 (libnunc-stans.so.0+0x0000000046f4)
    #7 event_cb src/nunc-stans/ns/ns_event_fw_event.c:118 (libnunc-stans.so.0+0x0000000065f3)
    #8 event_base_loop <null> (libevent-2.0.so.5+0x000000010f18)
    #9 event_loop_thread_func src/nunc-stans/ns/ns_thrpool.c:581 (libnunc-stans.so.0+0x0000000040c3)
    #10 <null> <null> (libtsan.so.0+0x0000000257eb)
  Mutex M40 previously acquired by the same thread here:
    #0 pthread_mutex_lock <null> (libtsan.so.0+0x00000003b62e)
    #1 work_job_execute src/nunc-stans/ns/ns_thrpool.c:284 (libnunc-stans.so.0+0x00000000444e)
    #2 event_cb src/nunc-stans/ns/ns_thrpool.c:642 (libnunc-stans.so.0+0x0000000046f4)
    #3 event_cb src/nunc-stans/ns/ns_event_fw_event.c:118 (libnunc-stans.so.0+0x0000000065f3)
    #4 event_base_loop <null> (libevent-2.0.so.5+0x000000010f18)
    #5 event_loop_thread_func src/nunc-stans/ns/ns_thrpool.c:581 (libnunc-stans.so.0+0x0000000040c3)
    #6 <null> <null> (libtsan.so.0+0x0000000257eb)
  Mutex M40 acquired here while holding mutex M13 in main thread:
    #0 pthread_mutex_lock <null> (libtsan.so.0+0x00000003b62e)
    #1 internal_ns_job_done src/nunc-stans/ns/ns_thrpool.c:217 (libnunc-stans.so.0+0x0000000035f7)
    #2 job_queue_cleanup src/nunc-stans/ns/ns_thrpool.c:210 (libnunc-stans.so.0+0x000000004987)
    #3 sds_queue_destroy src/libsds/sds/queue/queue.c:118 (libsds.so.0+0x00000000c2e3)
    #4 sds_tqueue_destroy src/libsds/sds/queue/tqueue.c:60 (libsds.so.0+0x00000000d724)
    #5 sds_lqueue_destroy src/libsds/sds/queue/lqueue.c:249 (libsds.so.0+0x00000000c41c)
    #6 ns_thrpool_destroy src/nunc-stans/ns/ns_thrpool.c:1532 (libnunc-stans.so.0+0x000000005649)
    #7 ns_stress_test src/nunc-stans/test/test_nuncstans_stress_core.c:541 (lt-test_nuncstans_stress_small+0x000000003347)
    #8 <null> <null> (libcmocka.so.0+0x000000004ae8)
    #9 __libc_start_main <null> (libc.so.6+0x000000020f29)
  Mutex M13 previously acquired by the same thread here:
    #0 pthread_mutex_lock <null> (libtsan.so.0+0x00000003b62e)
    #1 sds_tqueue_destroy src/libsds/sds/queue/tqueue.c:59 (libsds.so.0+0x00000000d714)
    #2 sds_lqueue_destroy src/libsds/sds/queue/lqueue.c:249 (libsds.so.0+0x00000000c41c)
    #3 ns_thrpool_destroy src/nunc-stans/ns/ns_thrpool.c:1532 (libnunc-stans.so.0+0x000000005649)
    #4 ns_stress_test src/nunc-stans/test/test_nuncstans_stress_core.c:541 (lt-test_nuncstans_stress_small+0x000000003347)
    #5 <null> <null> (libcmocka.so.0+0x000000004ae8)
    #6 __libc_start_main <null> (libc.so.6+0x000000020f29)
  Thread T5 (tid=373, finished) created by main thread at:
    #0 pthread_create <null> (libtsan.so.0+0x000000028e03)
    #1 ns_thrpool_new src/nunc-stans/ns/ns_thrpool.c:1466 (libnunc-stans.so.0+0x000000005eb7)
    #2 ns_stress_test src/nunc-stans/test/test_nuncstans_stress_core.c:470 (lt-test_nuncstans_stress_small+0x000000002e7c)
    #3 <null> <null> (libcmocka.so.0+0x000000004ae8)
    #4 __libc_start_main <null> (libc.so.6+0x000000020f29)
SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) (/lib64/libtsan.so.0+0x3b62e) in pthread_mutex_lock
==================

Metadata Update from @mreynolds:
- Issue set to the milestone: 1.4.0

Metadata Update from @mreynolds:
- Issue set to the milestone: 1.4 backlog (was: 1.4.0)

389-ds-base is moving from Pagure to Github. This means that new issues and pull requests
will be accepted only in 389-ds-base's github repository.

This issue has been cloned to Github and is available here:
- https://github.com/389ds/389-ds-base/issues/2684

If you want to receive further updates on the issue, please navigate to the github issue
and click on subscribe button.

Thank you for understanding. We apologize for all inconvenience.

Metadata Update from @spichugi:
- Issue close_status updated to: wontfix
- Issue status updated to: Closed (was: Open)
Metadata
None
None
None
None
None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.