the attache patch fixes it for me, I am working on a testcase and the will create a PR.
it is just here if anybody wants to get it earlier

@lkrispen nice catch !!

The patch systematically resets the filter_bypass in case we are under the threshold.
I wonder if we can be in intersect and still benefit from filter_bypass.
Is it an acceptable (in term of perf) drop of filter_bypass because of the low number of candidates or because on intersection we should drop filter_bypass.

https://pagure.io/389-ds-base/pull-request/50235

@tbordaz What do you mean systematically resets? Maybe I don't quite understand the conditions or what's occuring when you say this?

I agree @firstyear better to give an example of my concern.
Assume we have a final filter like : '(&(uid=firstyear)(employeenumber=1111))'
The two components being indexed it leads to candidate list smaller than threshold and filter_bypass=yes.
I was wondering if the patch could reset filter_bypass=no.
If it is, the consequence of the fix will be to skip an optimization to evaluate less than threshold (10) entries. IMHO it is acceptable if it prevents an overcomplicated patch.

@tbordaz I thought that filter_bypass init's as "no" and then only in the function we set yes. So the yes should move up and result in a yes? We could easily prove this by grepping filter_bypass and checking all the times it's set in the test.

@tbordaz @firstyear First we do set DONT bypass filter, by default we can assume that all the entries in the id list match the filter, only inspecial cases, allids or no index or below threshold we need to set the flag to NOT bypass filter verification
Next, if we detect that we are under the threshold, we return this ID list and clear all other sub lists, so intersection is done.
If you want to somehow defer the decision if to use the trheshold or to continue wit intersection you would have to make a design proposal.
I think the issue that we can return wrong results for a onelevel search is severe, and we should provide a fix as soon as possible.

@lkrispen , I agree this bug important/urgent. I do not intend to change the optimization to stop intersection when we have less than threshold candidates.
The patch sets DONT bypass filter if we have less than threshold candidates, so my understanding is that we will stop intersection but evaluate the filter. Am I missing something ?

Note that I am fine evaluating the filter with such small number of candidates.

@lkrispen , I agree this bug important/urgent. I do not intend to change the optimization to stop intersection when we have less than threshold candidates.
The patch sets DONT bypass filter if we have less than threshold candidates, so my understanding is that we will stop intersection but evaluate the filter. Am I missing something ?

No, thats what the patch wants to do (and what we did some time ago), and what I think should be done

Note that I am fine evaluating the filter with such small number of candidates.
you have no choice, either remove the threshold or face the filter evaluation

@lkrispen thanks for the explanation, I realize my mistake. Stopping the intersection requires the filter evaluation.
You have my ACK regarding the fix

@tbordaz @firstyear First we do set DONT bypass filter, by default we can assume that all the entries in the id list match the filter, only inspecial cases, allids or no index or below threshold we need to set the flag to NOT bypass filter verification
Next, if we detect that we are under the threshold, we return this ID list and clear all other sub lists, so intersection is done.
If you want to somehow defer the decision if to use the trheshold or to continue wit intersection you would have to make a design proposal.
I think the issue that we can return wrong results for a onelevel search is severe, and we should provide a fix as soon as possible.

Yes, this matched my expectations of how the bypass filter test worked. Perhaps we were using different words, but this is what I expected. The filter starts in "bypass" mode, and then if any element contains "don't bypass" then we keep that and setting it up to the parent filter so that the final candidate set is filter tested.

389-ds-base is moving from Pagure to Github. This means that new issues and pull requests
will be accepted only in 389-ds-base's github repository.

This issue has been cloned to Github and is available here:
- https://github.com/389ds/389-ds-base/issues/3293

If you want to receive further updates on the issue, please navigate to the github issue
and click on subscribe button.

Thank you for understanding. We apologize for all inconvenience.