Please indicate the version of 389-ds-base - rpm -q 389-ds-base

Also, please read http://port389.org/wiki/FAQ#Debugging_Crashes about how to set up your system to be able to produce core files and stack traces

attachment
stacktrace.1399469156.zip

hi
389-ds-base-1.2.11.28-1.el5
389-ds-base-debuginfo-1.2.11.28-1.el5

The core is set according to the link, but still no core was found (look at /var/log ,/ anyplace)

Had create a stacktrace for you

BTW, do you know what the next line mean, I saw this on 2 ldap server. Had tried to recreate the replicate agreement, but the warning is still there
"[07/May/2014:15:32:22 +0200] NSMMReplicationPlugin - agmt="cn=ds.dsb.dk" (ds:389): Replica has a different generation ID than the local data."

Replying to [comment:2 van12]:

hi
389-ds-base-1.2.11.28-1.el5
389-ds-base-debuginfo-1.2.11.28-1.el5

The core is set according to the link, but still no core was found (look at /var/log ,/ anyplace)

Perhaps it is different on EL5. I guess you'll just have to do a web search to figure out how to enable setuid programs to dump core on EL5. In order to fix this bug, we are going to have either a core file with a stack trace from the core file, or an easy way to reproduce this bug.

Had create a stacktrace for you

But this is not from a core file?

BTW, do you know what the next line mean, I saw this on 2 ldap server. Had tried to recreate the replicate agreement, but the warning is still there
"[07/May/2014:15:32:22 +0200] NSMMReplicationPlugin - agmt="cn=ds.dsb.dk" (ds:389): Replica has a different generation ID than the local data."

Yes, it means the consumer (Replica) has not been initialized.

Thanks for the answer about the "Replica has a different generation"

I found the way to make the core:
/etc/limits.conf:
* soft core unlimited
/etc/sysctl.conf:
kernel.core_uses_pid = 1
kernel.core_pattern = /tmp/core-%e-%p

/etc/profile
ulimit -c unlimited

I attack hereby the stacktrace thstacktrace.1399578091.zip from the core, the previously was not from the core.

attachment
stacktrace.1399578091.zip

Thanks. This looks like Ticket #47448. Please try 1.2.11.29-2 from epel-testing.

Many thanks for the great support Rich, the 2 new 389-ds-base and 389-ds-libs work good since yesterday.

Have a nice weekend
Tuan

it works...but...For the last 3 weeks, there are at least 10 persons (incl me) who can't login (even the login works fine days/hours ago) to the master ldap server with the new 389-ds-base and 389-ds-libs. Login to other servers which use that ldap server work sometime (90% of time; maybe cache on the clients?))

I don't know what can it be?, by reenter the user ldap password (in GUI), it then work again. By restart of slapd, sshd didn't help.
password is 8 char, were not change recently. Can there be problem with cache on ldap server or the ldap server truncates somehow some characters in the password?

what can it be? it just happen you don't know when and how to replicate this behavious

Jun 5 05:25:02 dsbmgt01 sshd[24108]: pam_ldap: ldap_result Timed out
Jun 5 05:25:02 dsbmgt01 sshd[24108]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=nx03 user=jifu
Jun 5 05:25:04 dsbmgt01 sshd[24108]: Failed password for jifu from 217.16.100.20 port 57343 ssh2
Jun 5 05:25:04 dsbmgt01 sshd[24111]: Connection closed by 217.16.100.20

this morning there is problem with my acocunt again, "id" shows ok, ldapsearch works ok, but I can't login inot ldap server and other clients using that ldap server

I try now 1.2.11.29-1 from http://rpm.pbone.net, until now all OK

Replying to [comment:8 van12]:

it works...but...For the last 3 weeks, there are at least 10 persons (incl me) who can't login (even the login works fine days/hours ago) to the master ldap server with the new 389-ds-base and 389-ds-libs. Login to other servers which use that ldap server work sometime (90% of time; maybe cache on the clients?))

I don't know what can it be?, by reenter the user ldap password (in GUI), it then work again. By restart of slapd, sshd didn't help.
password is 8 char, were not change recently. Can there be problem with cache on ldap server or the ldap server truncates somehow some characters in the password?

what can it be? it just happen you don't know when and how to replicate this behavious

When this occurs, you should attempt to perform an LDAP bind as your user using ldapsearch. This will allow us to see if the issue is related to the server side or the client side, as it will not use any client-side cache.

thanks, here are 2 outputs of the ldapsearch
Notice the one "ldap seach BAD toward dsbmgt01.txt" contains two tests, one when I enter a wrong password. When I enter the right password, the ldapsearch just hang.
The other file is a test toward another ldap server dsbmgt02

[root@dsbmgt01 slapd-NNIT]# rpm -qa |grep -i 389
389-ds-base-1.2.11.29-1.el5
389-ds-base-libs-1.2.11.29-1.el5
389-ds-console-1.2.6-1.el5
389-ds-1.2.1-1.el5
[root@dsbmgt01 slapd-NNIT]# uname -a
Linux dsbmgt01.dsb.dk 2.6.18-348.18.1.el5 #1 SMP Fri Sep 6 12:37:18 EDT 2013 x86_64 x86_64 x86_64 GNU/Linux

[tnng@dsbmgt02 ~]$ rpm -qa |grep -i 389
389-ds-base-libs-1.2.11.15-32.el6_5.x86_64
389-ds-1.2.2-1.el6.noarch
389-ds-base-1.2.11.15-32.el6_5.x86_64
389-ds-console-1.2.6-1.el6.noarch
[tnng@dsbmgt02 ~]$
[tnng@dsbmgt02 ~]$ uname -a
Linux dsbmgt02.dsb.dk 2.6.32-358.23.2.el6.x86_64 #1

attachment
ldap seach BAD toward dsbmgt01.txt

attachment
ldap seacrh GOOD toward dsbmgt02.txt

When you run this BAD test case,

Attachment ldap seach BAD toward dsbmgt01.txt​ added
what do you get in the server's access log and error log?
And what's the status of the server at the moment?
I don't think the server is also hanging, but it might be helpful if we could see the stacktraces:
http://directory.fedoraproject.org/wiki/FAQ#Debugging_Hangs

there is nothing in errors,
I attachec hereby access log and stacktrace

attachment
access-log.txt

attachment
stacktrace.1403091735.txt

I found out it related to the 130 CLOSE_WAIT we have in "netstat -an". The sockets refer (-p in netstat) to different applications which use ldap (ssh, sudo,patrol).

After killing the processes having those CLOSE_WAIT (someone should be upset at me :-) ), slapd works again.

When it is OK, ns-slapd (see below) should be the process which owns the socket (not ssh, sudo, patrol..)

so somehow the ns-slapd didn't close the socket and leave the socket to the parent "ssh, sshd, sudo, patrol ..). It might happen when cronjob restart ns-slapd on this server every morning (for getting data replicate from the global ldap server using ldif file import; because the Firewall is blocked between them)
16 3 * * * /sbin/service dirsrv stop;/usr/lib64/dirsrv/slapd-NNIT/ldif2db -s "dc=nnit" -i /tmp/ldap.ldif;/sbin/service dirsrv start

nb: we have another setup (arlmgtdk01)similar to this one, which doesn't have problem with CLOSE_WAIT
[root@arlmgtdk01 ~]# rpm -qa|grep 389
389-ds-base-libs-1.2.10.2-20.el6_3.x86_64
389-ds-base-1.2.10.2-20.el6_3.x86_64
389-adminutil-1.1.15-1.el6.x86_64
389-ds-1.2.2-1.el6.noarch

dsbmgt01 (the one with CLOSE_WAIT)
[root@dsbmgt01 ~]# rpm -qa |grep 389
389-ds-base-1.2.11.29-2.el5
389-ds-console-1.2.6-1.el5
389-ds-1.2.1-1.el5
389-ds-base-libs-1.2.11.29-2.el5

[root@dsbmgt01 ~]# grep keep /etc/sysctl.conf
net.ipv4.tcp_keepalive_time = 120 (sat from 600 to 120s)

[root@dsbmgt01 ~]# netstat -pan|grep 389|grep CLOSE
tcp 38 0 10.139.40.15:33737 10.139.40.15:389 CLOSE_WAIT 2326/PatrolAgent
tcp 38 0 10.139.40.15:36067 10.139.40.15:389 CLOSE_WAIT 10093/sshd
tcp 38 0 10.139.40.15:36069 10.139.40.15:389 CLOSE_WAIT 10097/sshd
tcp 38 0 10.139.40.15:36070 10.139.40.15:389 CLOSE_WAIT 10098/ssh
tcp 38 0 10.139.40.15:29219 10.139.40.15:389 CLOSE_WAIT 24139/sshd
tcp 38 0 10.139.40.15:29217 10.139.40.15:389 CLOSE_WAIT 24135/sshd
[root@dsbmgt01 ~]#

When it is OK, ns-slapd should be the process which owns the socket
[root@dsbmgt01 ~]# netstat -pan|grep ":389 "
tcp 0 0 0.0.0.0:389 0.0.0.0:* LISTEN 31135/ns-slapd
tcp 0 0 10.139.40.15:389 10.139.40.15:48385 ESTABLISHED 31135/ns-slapd
tcp 0 0 10.139.40.15:389 10.35.10.109:45291 ESTABLISHED 31135/ns-slapd
tcp 0 0 10.139.40.15:389 10.35.10.109:45290 ESTABLISHED 31135/ns-slapd
tcp 0 0 10.139.40.15:389 10.35.10.109:45293 ESTABLISHED 31135/ns-slapd
tcp 0 0 10.139.40.15:389 10.35.10.109:45292 ESTABLISHED 31135/ns-slapd
tcp 0 0 10.139.40.15:389 10.35.10.109:45295 ESTABLISHED 31135/ns-slapd

[root@dsbmgt01 ~]# ps -ef|grep 31135
fds 31135 1 1 Jun18 ? 00:13:31 /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-NNIT -i /var/run/dirsrv/slapd-NNIT.pid -w /var/run/dirsrv/slapd-NNIT.startpid

What do you think?

From the net: explanation
Crist Clark
CLOSE_WAIT means that the local end of the connection has received a FIN from the other >>end, but the OS is waiting for the program at the local end to actually close its >>connection.

The problem is your program running on the local machine is not closing the socket. It >>is not a TCP tuning issue. A connection can (and quite correctly) stay in CLOSE_WAIT >>forever while the program holds the connection open.

Once the local program closes the socket, the OS can send the FIN to the remote end >>which transitions you to LAST_ACK while you wait for the ACK of the FIN. Once that is >>received, the connection is finished and drops from the connection table (if you're end >>is in CLOSE_WAIT you do not end up in the TIME_WAIT state).

One other thing, I have an old session, surprisely the socket towards the ldap server is still there after 2 days. (today is the 20th of June).

The tcp_keepalive_time is 120s, I expect the "389" socket will time out (close) after a short time. What is the reason the socket keep opening after the authentication is finished? Does my "infos/id" not cache on the client? does the client need to get the data from ldap server every second?

[tnng@dsbmgt01 ~]$ ps -ef|grep tnng
tnng 13959 13923 0 Jun18 ? 00:00:00 sshd: tnng@pts/9
root 13923 28609 0 Jun18 ? 00:00:00 sshd: tnng [priv]
tnng 13994 13959 0 Jun18 pts/9 00:00:00 -ksh
tnng 14015 13994 0 Jun18 pts/9 00:00:00 bash

[tnng@dsbmgt01 ~]$ netstat -pan |grep 389|egrep 'ssh|bash'
tcp 0 0 10.139.40.15:2691 10.139.60.2:389 ESTABLISHED 14015/bash
tcp 0 0 10.139.40.15:2683 10.139.60.2:389 ESTABLISHED 13959/sshd

I assume the stacktrace was taken when this ldapsearch in the "ldap seach BAD toward dsbmgt01.txt" ​is hung.

ldapsearch -d 5 -x -h dsbmgt01 -p 389 -D "cn=Tuan Nguyen,cn=unixtek,ou=Infrastructure,dc=nnit" -W -s base

I see this BIND request has no result in the access log.
550 [18/Jun/2014:13:32:41 +0200] conn=111597 fd=65 slot=65 connection from 10.139.40.15 to 10.139.40.15
551 [18/Jun/2014:13:32:41 +0200] conn=111597 op=0 BIND dn="cn=Tuan Nguyen,cn=unixtek,ou=Infrastructure,dc=nnit" method=128 version=3

I don't see anything suspicious in the stacktraces... All the threads are idle. (Next time you capture stacktraces, could you please install 389-ds-base-debuginfo?)
Attachment stacktrace.1403091735.txt​ added
And the issue you originally reported was solved and this ticket is closed. Could you open a new one for this problem if you wish? Otherwise, this ticket won't be processed properly since the it had been closed...

hi Nhosoi

Ok I will open a new one If I see "close_wait" Again as I describe above.

Thanks Noriko

Replying to [comment:17 van12]:

hi Nhosoi

Ok I will open a new one If I see "close_wait" Again as I describe above.

Thanks Noriko

Thank YOU, Tuan!

389-ds-base is moving from Pagure to Github. This means that new issues and pull requests
will be accepted only in 389-ds-base's github repository.

This issue has been cloned to Github and is available here:
- https://github.com/389ds/389-ds-base/issues/1127

If you want to receive further updates on the issue, please navigate to the github issue
and click on subscribe button.

Thank you for understanding. We apologize for all inconvenience.