#50545 Port remaining legacy tools to new python CLI
Closed: wontfix by mreynolds. Opened by mreynolds.

Issue Description

We still have to port these legacy tools:

  • dbmon.sh
  • dbverify
  • dbgen
  • rsearch(?)
  • validate-syntax.pl
  • fixup-memberuid.pl
  • repl-monitor.pl
    • We have this working on the agreement level, but not like the full report that repl-monitor.pl can do

I think that dbgen is already implemented in lib389, we just need the cli for it (probably in dsconf?). Similar, fixup memberuid should already have a fixup task.

I don't even know what rsearch does ....

Metadata Update from @firstyear:
- Custom field origin adjusted to None
- Custom field reviewstatus adjusted to None

Just a note - for validate-syntax.pl, we have an opened issue - https://pagure.io/389-ds-base/issue/50173 (also, there is an ongoing discussion, so it is probably will go as a part of health check tool).

Oh, okay, I haven't found fixup-memberuid.pl because, by some reason, we don't have it...
The only thing we have isadmin/src/scripts/template-fixup-memberuid.pl...
Other scripts available in both fixup-memberuid.pl and template-fixup-memberuid.pl forms

rsearch(?)

This is a repeated search tool, similar to ldclt. It's not written in perl, but in C.

Oh, okay, I haven't found fixup-memberuid.pl because, by some reason, we don't have it...

It's installed as part of instance specific scripts under /usr/lib64/dirsrv/slapd-instance_name. Hence it's present only as a template. I remember we were making these scripts instance agnostic, but this one is not installed under /usr/sbin/

https://pagure.io/389-ds-base/pull-request/50570

https://pagure.io/389-ds-base/pull-request/50583

https://pagure.io/389-ds-base/pull-request/50614

https://pagure.io/389-ds-base/pull-request/50614

cherry-pick repl-monitor to 1.4.1

f6c7e2cb3..8c896dc89 389-ds-base-1.4.1 -> 389-ds-base-1.4.1

@spichugi, could you please cherry pick 761dd658183106363379326490439e381894615f to 1.4.0? The build fails because of missing DSRC_HOME var in _constants.py (comes from 3798137c4b408debfc088f02da1df000e1790cc5).
Thanks!

@spichugi, could you please cherry pick 761dd65 to 1.4.0? The build fails because of missing DSRC_HOME var in _constants.py (comes from 3798137).
Thanks!

Cherry-picked to 1.4.0 - dbb89923719224d3e808ef1c0e6ec0c4967c1148
Thanks for the reporting!

Metadata Update from @spichugi:
- Issue assigned to spichugi

https://pagure.io/389-ds-base/pull-request/50733

The report has two duplicate Supplier headers

Screenshot_from_2019-12-06_15-08-22.png

Well things are still not right

Looking at the new report, the supplier reports are very different from one another, and its hard to tell what I'm looking at

Screenshot_from_2019-12-07_11-45-50.png

For the local instance (slapd-replica) in this case on port 5555 looks good, but the report on localhost:389 is much different. There is only a status and reason, but there is no status and reason fields for the first/local instance.

Also in this case I am getting "Invalid credentials" for the reason. So we default the bind dn to "cn=directory manager", but I use "cn=dm". Even though I entered cn=dm in the interactive modal, it still used "cn=directory manager" to authenticate:

[07/Dec/2019:11:45:03.662868474 -0500] conn=60 fd=64 slot=64 connection from ::1 to ::1
[07/Dec/2019:11:45:03.662996984 -0500] conn=60 op=0 BIND dn="cn=Directory Manager" method=128 version=3
[07/Dec/2019:11:45:03.663043800 -0500] conn=60 op=0 RESULT err=49 tag=97 nentries=0 etime=0.000137342 - No suffix for bind dn found
[07/Dec/2019:11:45:03.672857072 -0500] conn=60 op=1 UNBIND
[07/Dec/2019:11:45:03.672870996 -0500] conn=60 op=1 fd=64 closed - U1

If manually set the bind dn and password in the credential table, then it works. Then I get a very different report than before, but it matches what I would expect:

Screenshot_from_2019-12-07_11-56-58.png

So when we can not connect to a replica, I think the report should have a more obvious message:

Can not get replication information from Replica

Status: Unavailable
Reason: Invalid credentials

Sorry to nit pick, but this report is very important and I want to get it perfect (Replication Reports for Dummy's). The report should be easy to understand for people who don't understand replication.

Also it would be nice to have the report set up in a way where you can sort all the agreements/replicas by lag time. It would be nice to quickly find what agreements are way behind the others. This is what my old "Lag Report" did. The Lag Time is probably the most important stat from the report, so we should be able to use it effectively to find troubled replicas. I'm not sure if that breaks your existing model, but it is important to be able to sort based on Lag Time. Maybe we need multiple reports?

Maybe just add an additional "simple/short" table at the end of the report with all the agreements so we can sort by lag time?

Repl report UI work complete:

commit 3054205d93f6dddbfb86de040ac55521c08ca13b

02fdfa713..32c1505be 389-ds-base-1.4.2 -> 389-ds-base-1.4.2

903a41f22..dbfa7a135 389-ds-base-1.4.1 -> 389-ds-base-1.4.1

Metadata Update from @mreynolds:
- Issue priority set to: major

Porting dbmon.sh to dsconf....

Metadata Update from @mreynolds:
- Issue assigned to mreynolds (was: spichugi)

Metadata Update from @mreynolds:
- Custom field rhbz adjusted to https://bugzilla.redhat.com/show_bug.cgi?id=1795943

Porting dbmon.sh to dsconf....

https://pagure.io/389-ds-base/pull-request/51001

Commit 2523f487 relates to this ticket

dbmon.sh port

Commit 2523f48 relates to this ticket

03103f19f..53fad04d9 389-ds-base-1.4.2 -> 389-ds-base-1.4.2

dbgen.pl is the last script to port. lib389 has a very simple dbgen class, but it does nothing like what dbgen.pl can do. I also have a ldif gen tool that does most of what dbgen.pl could do and more, so I will look at merging my script into the lib389 dbgen class next...

Commit c868a418 relates to this ticket

Ugh, forgot to remove the "incr" option from the argparse list:

Commit c868a41 relates to this ticket

53fad04d9..877f62868 389-ds-base-1.4.2 -> 389-ds-base-1.4.2

@mreynolds I think I already did dbgen .... ? It may not have a front end though.

@mreynolds I think I already did dbgen .... ? It may not have a front end though.

All it did was create one type of entry, and yes it not included in the CLI. But it should be able to create entries that can also be consumed by ldclt for starters. I'm merging in my personal LDIF tool, it also creates COS definitions and templates, various roles, static groups, and modification LDIFs, etc. It shouldn't take long, I expect to have a PR ready in a few days, then we will finally have ported everything to python!!!

Ahh cool :) nice! That sounds really good :D

Ahh cool :) nice! That sounds really good :D

https://pagure.io/389-ds-base/pull-request/51022

Please review :-)

Metadata Update from @mreynolds:
- Custom field rhbz adjusted to https://bugzilla.redhat.com/show_bug.cgi?id=1795943, https://bugzilla.redhat.com/show_bug.cgi?id=1798394 (was: https://bugzilla.redhat.com/show_bug.cgi?id=1795943)

Ugh, forgot to remove the "incr" option from the argparse list:

Commit c868a41 relates to this ticket

53fad04..877f628 389-ds-base-1.4.2 -> 389-ds-base-1.4.2

dbmon porting issue on 1.4.2

24c501db3..92e6937eb 389-ds-base-1.4.2 -> 389-ds-base-1.4.2

Commit 326be2c9 fixes this issue

Commit 326be2c fixes this issue

a7d807bf0..cdeaed900 389-ds-base-1.4.3 -> 389-ds-base-1.4.3

dbgen port completed, no other legacy tools to port.

Metadata Update from @mreynolds:
- Issue close_status updated to: None (was: Fixed)

Well done @mreynolds and @spichugi for all your work on this - I honestly thought this was an impossible goal in the past, but here we are with a python toolset. Seriously, amazing work from you both :D

Well done @mreynolds and @spichugi for all your work on this - I honestly thought this was an impossible goal in the past, but here we are with a python toolset. Seriously, amazing work from you both :D

Thanks, but the elephant in the room is still logconv.pl :zap: [thunder clap]

Just be quiet and take the compliment mate :) We'll get to logconv.pl ... one day .... >.>

Commit e8f510e9 relates to this ticket

Commit 98a8287e relates to this ticket

389-ds-base is moving from Pagure to Github. This means that new issues and pull requests
will be accepted only in 389-ds-base's github repository.

This issue has been cloned to Github and is available here:
- https://github.com/389ds/389-ds-base/issues/3601

If you want to receive further updates on the issue, please navigate to the github issue
and click on subscribe button.

Thank you for understanding. We apologize for all inconvenience.

Metadata Update from @spichugi:
- Issue close_status updated to: wontfix

Metadata
Related Pull Requests