We now have two systems which want to use taskotron to trigger tasks and report results, but the actual execution happens elsewhere. It's openqa (D748 is the first piece of that) and tunir (this ticket). The problem is that if we use one of our disposable minions for scheduling a task somewhere else (on openqa machine or in beaker), wait until it completes and then report the results, it is effectively blocking one of our minions while doing nothing. That looks like a problem, if such tasks are executed often, or there are many of them.
We should think about possible ways to make this work better. A few ideas: * We could schedule the tasks in an async way - just start it, and then wait until the task sends a fedmsg that it completed, and then run a second part of the task which would gather the results and report them. But that requires that the task itself has access to fedmsg emitting inside fedora infra, which might be problematic. We might need to use our own fedmsg hub for that. * Have a special persistent minion flagged "noop" inside buildbot (or somewhere else). Have the task specify in its formula that it is a "noop" task. When we run such task, we run it on a noop minion which allows running multiple tasks at once, and is either never destroyed (or only if all tasks complete). * We could just start the task and then somehow allow the task to report its results to resultsdb directly (it could still use libtaskotron, but it would not involve a roundtrip back to the minion where the execution started). That would probably involve giving a use-once resultsdb token to the task, so that it can't mess around with resultsdb arbitrarily.
More ideas or their fine-tuning is welcome.
For the short term, I'm fine with blocking the minions. It's significantly easier and if it becomes a problem, we'll add more minions and if it's still a problem at that point, we can be more clever about it.
I'm not saying that it isn't inefficient - I'm just arguing against premature optimization