{{{ (gdb) run Starting program: /home/william/build/nunc-stans/.libs/lt-cmocka_stress_test [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". [==========] Running 1 test(s). [ RUN ] ns_stress_test 7 ns_thrpool_new(): max threads, (80) stacksize (0), event q size (unbounded), work q size (unbounded)
Program received signal SIGSEGV, Segmentation fault. 0x00007ffff79c83aa in lfds710_stack_init_valid_on_current_logical_core (ss=0x6054e0, user_state=0x0) at ../../src/lfds710_stack/lfds710_stack_init.c:13 13 LFDS710_PAL_ASSERT( (lfds710_pal_uint_t) ss->top % LFDS710_PAL_ATOMIC_ISOLATION_IN_BYTES == 0 ); Missing separate debuginfos, use: dnf debuginfo-install libcmocka-1.1.0-3.fc26.x86_64 nss-softokn-freebl-3.26.0-4.fc26.x86_64 (gdb) bt
(gdb) }}}
Without asan, segfault occurs on stress test.
However, the struct appears correct:
{{{ (gdb) print *ss $2 = {top = {0x0, 0x0}, user_state = 0x0, pop_backoff = {lock = 0, backoff_iteration_frequency_counters = {0, 0}, metric = 0, total_operations = 0}, push_backoff = {lock = 0, backoff_iteration_frequency_counters = {0, 0}, metric = 0, total_operations = 0}} (gdb) print ss->top $3 = {0x0, 0x0} }}}
So the fault is from
{{{
LFDS710_PAL_ASSERT( (lfds710_pal_uint_t) ss->top % LFDS710_PAL_ATOMIC_ISOLATION_IN_BYTES == 0 );
}}}
Where the ASSERT fails. So ss->top == 0, which means that ss>top % 0x80 == 0. So this is not correct, something weird here ...
int main(int argc, char **argv) { printf("%d\n", !(0 % 128 == 0)); return 0; }
Which correctly prints 0, and the assert macro is:
So give that if will execute the code if nonzero, something funky is happening here.
Ahhhh I see. I was mislead by gdb. {0x0, 0x0} is because it knows the array is of fixed size. So what's actually happening is that ss->top % LFDS710_PAL_ATOMIC_ISOLATION_IN_BYTES is checking the address of the array, but gdb "helpfully" is just showing the content.
This yields
{{{ (gdb) print &ss->top $3 = (struct lfds710_stack_element * volatile(*)[2]) 0x6054e0 (gdb) print 0x6054e0 % 128 $4 = 96 }}}
This then triggers the assert, because the library won't be able to CAS on an address that isn't aligned a 128byte boundary.
Curiously, this doesn't affect a build that uses ASAN ...
This appears to be an issue between Fedora gcc vs Rhel GCC
Affects - Fedora: gcc-6.2.1-2.fc26.x86_64
Does not affect - Rhel: gcc-4.8.5-4.el7.x86_64
Okay, its not an issue in GCC. It's a flaw in my understanding of the alignment operator. This only affects stack allocated elements. It was by chance that previously on el7 this would allocate on a 128 byte alignment. It just so happens that gcc 6 produces different code, and thus the issue.
So as we are mallocing on the heap, the stack state and queue states for lfds, these must be aligned to a 128 byte boundary, but malloc doesn't guarantee this! How does malloc know the struct wants alignment at runtime. It doesn't.
So we need to use the posix_memalign functions to satisfy this.
This means we'll need to add a similar memalign function to slapi_ch_ to satisfy this.
attachment 0001-Ticket-63-segfault-in-thr_stack_create.patch
commit a11fb3a4848d0a04753abd27b651fa9d0e0bfd15 Compressing objects: 100% (33/33), done. Writing objects: 100% (33/33), 9.68 KiB | 0 bytes/s, done. Total 33 (delta 25), reused 0 (delta 0) To ssh://git.fedorahosted.org/git/nunc-stans.git 9dacc22..59aa673 master -> master