Continuation of a 389-users thread from last week... "Changes" referred to in the thread below were disabling vattr and a large (128->32) reduction in number of threads.
Attaching last round of stack traces taken from the server during pauses in new connection (port 389) processing.
Date: Fri, 25 Jul 2014 13:04:55 -0600 From: Rich Megginson rmeggins@redhat.com To: 389-users@lists.fedoraproject.org Subject: Re: [389-users] port 389 listener getting hung up on connection locks? User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.6.0
On 07/25/2014 07:59 AM, Thomas Walker wrote:
On Thu, Jul 24, 2014 at 04:14:13PM -0400, Thomas Walker wrote: On Thu, Jul 24, 2014 at 01:50:39PM -0600, Rich Megginson wrote: Did the above make any significant difference with respect to the performance? The problem is definitely load related and transient so it is hard to say immediately. I definitely still see many multisecond bind times, althought they're more of the 3-6 second variety so far (as opposed +to teens in the teens like this morning). Since it comes and goes, I'll need to let it run for awhile to see how things average out. Can definitely confirm that it has made a substantial difference... Max and average response times are about 25% of what they were. Still get into high single digit seconds in some cases, but a significant +improvement to be sure.
On Thu, Jul 24, 2014 at 04:14:13PM -0400, Thomas Walker wrote:
On Thu, Jul 24, 2014 at 01:50:39PM -0600, Rich Megginson wrote: Did the above make any significant difference with respect to the performance? The problem is definitely load related and transient so it is hard to say immediately. I definitely still see many multisecond bind times, althought they're more of the 3-6 second variety so far (as opposed +to teens in the teens like this morning). Since it comes and goes, I'll need to let it run for awhile to see how things average out.
On Thu, Jul 24, 2014 at 01:50:39PM -0600, Rich Megginson wrote:
Did the above make any significant difference with respect to the performance? The problem is definitely load related and transient so it is hard to say immediately. I definitely still see many multisecond bind times, althought they're more of the 3-6 second variety so far (as opposed +to teens in the teens like this morning). Since it comes and goes, I'll need to let it run for awhile to see how things average out.
Did the above make any significant difference with respect to the performance?
The problem is definitely load related and transient so it is hard to say immediately. I definitely still see many multisecond bind times, althought they're more of the 3-6 second variety so far (as opposed +to teens in the teens like this morning). Since it comes and goes, I'll need to let it run for awhile to see how things average out.
Can definitely confirm that it has made a substantial difference... Max and average response times are about 25% of what they were. Still get into high single digit seconds in some cases, but a significant +improvement to be sure.
Ok. I suggest that you open a ticket at https://fedorahosted.org/389/newticket so that we can track this issue, discuss, attach files, etc.
-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users
attachment stacktraces2.tar.gz
Downgrading severity as a couple of weeks with the "new settings" (disable vattr, drastically reduce number of threads and increase number of replicated server instances behind the load balancers to make up for it) have resulted in significant improvements. I'd close it completely but it does sound like a good idea to look at possibly moving the listener out to a separate so that new connections don't wait unnecessarily. But then I'm only somewhat familiar with the 389 internals so feel free to close if the devs don't agree.
Thanks again for the help.
Per triage, push the target milestone to 1.3.6.
Metadata Update from @thwalker3: - Issue set to the milestone: 1.3.6.0
I'm going to resolve this because the new settings resolved the issue for our user. Additionally, we have now made nunc-stans default, so it likely will prevent this. Please create a new issue if this persists.
Metadata Update from @firstyear: - Custom field reviewstatus adjusted to review - Issue close_status updated to: None
Metadata Update from @firstyear: - Issue close_status updated to: worksforme - Issue status updated to: Closed (was: Open)
389-ds-base is moving from Pagure to Github. This means that new issues and pull requests will be accepted only in 389-ds-base's github repository.
This issue has been cloned to Github and is available here: - https://github.com/389ds/389-ds-base/issues/1198
If you want to receive further updates on the issue, please navigate to the github issue and click on subscribe button.
subscribe
Thank you for understanding. We apologize for all inconvenience.
Metadata Update from @spichugi: - Issue close_status updated to: wontfix (was: worksforme)