#8008 Azure Pipeline slicing
Closed: fixed by abbra. Opened by slev.

Currently, unit tests execution time within Azure Pipelines is not balanced:
Base tests take ~13 minutes, while XMLRPC tests ~28 minutes.
The grouping is:

testsToRun:
test_cmdline
test_install
test_ipaclient
test_ipalib
test_ipaplatform
test_ipapython

and

testsToRun:
test_xmlrpc

The same problem exists for Travis, where the grouping is

TESTS_TO_RUN="
test_cmdline 
test_install  
test_ipaclient 
test_ipalib      
test_ipaplatform  
test_ipapython
test_ipaserver 
test_xmlrpc/test_[l-z]*.py"

and

TESTS_TO_RUN="test_xmlrpc/test_[a-k]*.py"

task1 ~ 3 min
task2 ~ 9 min

Some sort of autobalancer is required.

Azure Pipelines supports slicing out of the box:
https://docs.microsoft.com/en-us/azure/devops/pipelines/process/phases?view=azure-devops&tabs=yaml#slicing
https://docs.microsoft.com/ru-ru/azure/devops/pipelines/test/parallel-testing-any-test-runner?view=azure-devops&viewFallbackFrom=vsts

For the record, I highlight the main:

An agent job can be used to run a suite of tests in parallel. For example, you can run a large suite of 1000 tests on a single agent. Or, you can use two agents and run 500 tests on each one in parallel. To leverage slicing, the tasks in the job should be smart enough to understand the slice they belong to.

The step that runs the tests in a job needs to know which test slice should be run. The variables System.JobPositionInPhase and System.TotalJobsInPhase can be used for this purpose:

  • System.TotalJobsInPhase indicates the total number of slices (you can think of this as "totalSlices")
  • System.JobPositionInPhase identifies a particular slice (you can think of this as "sliceNum")

Two problems remain unsolved:
- pytest doesn't understand slicing
- the IPA tests haven't to be splitted just using dividing by the number of jobs
(there are tests which depend on each other within test module)

I've prepared pytest plugin which solves the slicing.


master:

  • 2312b38a678d30b73c08dae19db7f8b74361a957 Simplify ipa-run-tests script
  • a2d4e2a61f46cf6337786b2cb031e3a9f394a708 Make use of Azure Pipeline slicing
  • c7500220c443c6051178211eadd64392ca7bd08a Avoid use of '/tmp' for pip operations

Metadata Update from @abbra:
- Issue set to the milestone: FreeIPA 4.8.1

ipa-4-8:

  • 8aa3ef0630c7cad0ad23bc7d335998cb2e57a3e5 Simplify ipa-run-tests script
  • 5780d6ababdfa8e765edddb411970e25c2e9e9e5 Make use of Azure Pipeline slicing
  • 0ec908bdb05a94f8a2f3d5f7b8fb5a5041cdccc1 Avoid use of '/tmp' for pip operations

Metadata Update from @abbra:
- Issue close_status updated to: fixed
- Issue status updated to: Closed (was: Open)

Metadata