#49895 Ticket 49029 - improve internal operations logging
Closed by spichugi. Opened by mreynolds.
mreynolds/389-ds-base ticket49029  into  master

Download 49895.patch

Description: This patch uses "local thread storage" to keep track of
the connection and op id numbers for internal operations.
This makes it easier to foloow the access logs and know
what connection is actually doing what.

          It also maintains its own operation id count for each
          internal operation.  This adds complexity once internal ops
          start nesting (plugins calling plugins).  To handle this
          two LTS variables were added to keep track of the nested
          level or count, and one to know once a series of nested
          operations are finally unnested/complete.  This is needed
          to maintain the correct internal operation id counter correct
          and in sequence.

https://pagure.io/389-ds-base/issue/49029

Reviewed by: ?

Not sure to understand, why we need this second increase of count_ptr. Will it jump the internal op counter ?

Not sure if was discussed elsewhere.
I have a slight preference to have both connid/op_id either in '( )' or without. Here we will have connid in parentheses and op_id not

I had another suggestion, was just testing it, will update the ticket soon

I'm fine changing the log format - that easy :-) @lkrispen has some ideas he is working on too

Not sure to understand, why we need this second increase of count_ptr. Will it jump the internal op counter ?

I tried to explain this in the comments, but it is confusing. So once we get nested (plugins calling plugin that all do logging) the internal op id basically gets out of sync and we have to keep it inline so the logging is correct. Once we are out of the nested calls, and back to the first internal op that started the whole nested loop, the op id is behind by 1. So in that particular case we have to increase it a second time for the next internal op - this is actually what caused me so much pain to fix and why I needed the SLAPI_TD_OP_NESTED thread data variable.

I hope that helps explain it, but feel free to ask more questions.

@mreynolds thanks for the explanation. I understand the difficulty with nested operation. I agree with your patch and wait for format agreement.

Just a remark would it help/simplify the nested problem to log the nested level as well. for example

 conn=Internal(3) op=66(1) lvl=2 MOD dn="dc=example,dc=com"
 conn=Internal(3) op=1(1) lvl=3 SRCH base="cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config" scope=0 filter="objectclass=*" attrs="nsslapd-backend nsslapd-distribution-plugin
 conn=Internal(3) op=1(1) lvl=3 RESULT err=0 tag=48 nentries=1 etime=0.0000165786
 conn=Internal(3) op=66(1) lvl=2 RESULT err=0 tag=48 nentries=0 etime=0.0014946418
 conn=Internal(3) op=67(1) lvl=2 SRCH base="cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config" scope=0 filter="objectclass=*" attrs="nsslapd-backend nsslapd-distribution-plugin
 conn=Internal(3) op=67(1) lvl=2 RESULT err=0 tag=48 nentries=1 etime=0.0000165786

conn=Internal(3) op=1(1) lvl=3 RESULT err=0 tag=48 nentries=1 etime=0.0000165786

but you then no longer see that it is part of op=66, the lvl alone is, in my opinion not so helpful.

I think it is already a huge improvement, and we can improve further eg if we will amke logconv deal with it

rebased onto 9bc85d80b360360803f9ac22c0f0e3fad231ea7d

rebased onto c1f217edeb72ebb2640b7fa01a6aa75a8e0bf04b

logconv.pl has been amended, please review final patch. Thanks!

rebased onto 90bf17995505dc1534827e50bda77358c0d5be28

Pull-Request has been merged by mreynolds

389-ds-base is moving from Pagure to Github. This means that new issues and pull requests
will be accepted only in 389-ds-base's github repository.

This pull request has been cloned to Github as issue and is available here:
- https://github.com/389ds/389-ds-base/issues/2954

If you want to continue to work on the PR, please navigate to the github issue,
download the patch from the attachments and file a new pull request.

Thank you for understanding. We apologize for all inconvenience.

Pull-Request has been closed by spichugi

Metadata