#9207 Failure in AzurePipeline.freeipa (GATING InstallDNSSECFirst_1_to_5)
Closed: fixed by rcritten. Opened by mpolovka.

There seems to be a consistent failure in Azure pipelines run with PR-CI. Multiple re-runs didn't resolve this situation.

The error is

 subprocess.CalledProcessError: Command '['hostnamectl', 'set-hostname', 'master1.ipa.test']' returned non-zero exit status 1

as seen here.
Currently affected PRs: #6368, #6370, #6342


Metadata Update from @slev:
- Issue assigned to slev

systemd-hostnamed with debug loglevel of systemd shows nothing interesting besides tons of remount messages (it seems remounts are very slow or something keeps the resources busy). Note: the remounts are expected and are part of the job of service isolation.
So, systemd-hostnamed does it's job but the start up of this service is very slow and client (hostnamectl) failed with timeout (25sec).

Probably, performance degradation was caused by new azure agent which was updated last Monday:
azure agent 2.204.0 (reply time: 19sec):

1   [2022-07-18 07:29:56] INFO: 1_master_1: run: ['hostnamectl', 'set-hostname', 'master1.ipa.test']
1   [2022-07-18 07:30:15] INFO: 1_master_1: result: 0

azure agent 2.206.1 (reply timeout: 25sec):

1   [2022-07-21 20:20:04] INFO: 1_master_1: run: ['hostnamectl', 'set-hostname', 'master1.ipa.test']
1   [2022-07-21 20:20:30] INFO: 1_master_1: result: 1

It's hard to say what was changed, at least nothing suspicious can be found in the changelog.

This issue can be fixed in two ways:
- raise timeout for hostnamectl
- rearrange jobs

Additionally, there is running clamav on azure agent that is very greedy:

top - 11:15:45 up 1 min,  0 users,  load average: 2.28, 0.82, 0.30
Tasks: 159 total,   2 running, 157 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us,  0.0 sy, 13.3 ni, 86.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :   6943.7 total,   1596.5 free,   1522.8 used,   3824.4 buff/cache
MiB Swap:   4096.0 total,   4096.0 free,      0.0 used.   5108.9 avail Mem 
    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   3621 root      39  19  865532 796648   7972 R  26.7  11.2   0:08.54 clamscan
   1672 vsts      20   0 3538232 124428  57296 S   0.0   1.7   0:04.69 Agent.W+
   1617 vsts      20   0 3439088 113032  52320 S   0.0   1.6   0:01.47 Agent.L+
   3805 root      20   0 1446036  97004  47676 S   0.0   1.4   0:10.92 dockerd
    748 root      20   0 3716468  87548  47084 S   0.0   1.2   0:01.66 provisi+

PR:
https://github.com/freeipa/freeipa/pull/6376

master:

  • 0e8bde317505d57f2160e7ae4d4df01970f4b478 ap: Raise dbus timeout
  • dd094e3889a62a863bf951e879e8dbd98adb3470 ap: Disable azure's security daemon
  • 8542fd3012942814045f9603d6982f5601941ffb ap: Rearrange overloaded jobs
  • e9b232fcf44bbf57deb9eb6de762cbc43d9e7e5d ap: Constrain supported docutils

ipa-4-10:

  • 260d6378ec59d244e5f247f4af81f7ae8c72ac87 ap: Raise dbus timeout
  • acd1d127938aa9feefbbc7ee325963a2e44ef3c3 ap: Disable azure's security daemon
  • 8ff0c1a5ee33946202031a8bc83e855216cd0c95 ap: Rearrange overloaded jobs
  • e5f7356e7e83b605a821ece9f242ac924925f27e ap: Constrain supported docutils

Metadata Update from @rcritten:
- Issue close_status updated to: fixed
- Issue status updated to: Closed (was: Open)

ipa-4-9:

https://pagure.io/freeipa/c/e77b0b08d78d4d5ae7632ef23aebc577848ea507
https://pagure.io/freeipa/c/98c6e96e8db3d5bdc0315094b8a7bf81d196479b
https://pagure.io/freeipa/c/b59baf31bc097821ff7787ecd75affb27ea2a7c3
https://pagure.io/freeipa/c/1ada42e3bce58a729e689377b1a41b6cfa90b508

Metadata