#12514 please don't remove enrolled centos machines from IPA in staging
Closed: Fixed by zlopez. Opened by arrfab.

As CentOS and Fedora are using shared IPA backend for authentication, I'd request that nothing touching enrolled centos machines in IPA would be done (manually or through scripts)
I had to just waste my time this morning investigating why ipsilon (https://id.stg.centos.org) wasn't allowing anyone to auth (and so no openidc for services using our ipsilon instance)

Someone (who ? or a script ?) removed ipsilon.stg.iad2.centos.org from the ipsilon HBAC rule, denying so all auth requests .

Can you identify the root cause and ensure it wouldn't happen again please ?

Thanks a lot


Metadata Update from @zlopez:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: Needs investigation, high-gain, ops

seems related to https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/groups/ipsilon.yml#_92 ...

Metadata Update from @arrfab:
- Issue untagged with: Needs investigation, high-gain, ops
- Issue priority set to: Needs Review (was: Waiting on Assignee)

Metadata Update from @zlopez:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: Needs investigation, high-gain, ops

The change you are referring to happened 4 years ago. So I assume that didn't caused the machine to be removed.

Metadata Update from @kevin:
- Issue assigned to kevin

It's actually https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/groups/ipsilon.yml#_101

It was using the wrong hostname... but that was set in 2021?
b8e6754f97c (Aurélien Bompard 2021-03-22 17:07:45 +0100 101) host: "{{ (env == 'production')|ternary('ipsilon.iad2.centos.org', 'centos-ipa-client02.stg.iad2.fedoraproject.org') }}"

anyhow, I changed it to ipsilon.stg.iad2.centos.org

If you can confirm it's fixed / working?

forgot to give feedback (PTO) but yes, now working again ...
Is there a way to ensure that ansible would not remove hosts not managed by itself ?
Otherwise, just ensure that it's documented somewhere as that means that on my centos ansible infra side I'll never be autonomous to deploy/enroll a machine as then Fedora ansible would remove it :/

Well, I mean we could remove that task from our playbooks and you could depend on yourself to create that ?

Otherwise ansible is going to setup the thing thats defined.

Do you want us to remove that setup from our side?

@kevin : even if you remove that block, I guess that the other one about Fedora host would then remove the existing centos machines again .. so isn't there a parameter for that ipahbacrule to not purge things not declared ? if not then we'll have to still hard-code centos machines there but with dc move and so multiple machines that will need to be enrolled, I'd like that to be documented at least :)

well, my reading / understanding of it is that:

https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/groups/ipsilon.yml#_85 creates the 'ipsilon' hbac rule and

https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/groups/ipsilon.yml#_101
has action: member, so it adds those members/hosts.

So, I guess we could change the first one to just confirm/add the fedora ipsilon servers and then it wouldn't recreate the rule...
but then the rule would have to be... manually created or something?

Would this be high trouble, medium trouble, low trouble or a timeboxed investigation?

What if we do this:

diff --git a/playbooks/groups/ipsilon.yml b/playbooks/groups/ipsilon.yml
index 8c95c4344a..1f4636ece3 100644
--- a/playbooks/groups/ipsilon.yml
+++ b/playbooks/groups/ipsilon.yml
@@ -89,6 +89,14 @@
       hbacsvc:
       - ipsilon
       usercategory: all
+      ipaadmin_password: "{{ ipa_admin_password }}"
+    tags:
+    - ipsilon
+
+  - name: Allow login through ipsilon for the Fedora instance
+    ipahbacrule:
+      name: ipsilon
+      action: member
       host: "{{ (env == 'production')|ternary(groups['ipsilon'], groups['ipsilon_stg']) }}"
       ipaadmin_password: "{{ ipa_admin_password }}"
     tags:

ie, we remove the host call from the one that makes the hbrbac rule, so it should just create it if it doesn't exist.
Then we populate the fedora hosts in a seperate host level task.

just FYI : it happened again and so I guess someone rolled out the unmodified ansible for rdu3, and so it removed access for enrolled rdu3 hosts for centos :/ ....
Can we fix that so that I don't have to manually add back myself enrolled servers, removed by fedora automation ? TIA :)

I think I see the issue here, there is an exception for centos instances, but only for iad2 centos instances.

  - name: Allow login through ipsilon for the CentOS instance
    ipahbacrule:
      name: ipsilon
      action: member
      host: "{{ (env == 'production')|ternary('ipsilon.iad2.centos.org', 'ipsilon.stg.iad2.centos.org') }}"
      ipaadmin_password: "{{ ipa_admin_password }}"
    tags:
    - ipsilon

And I will check if the rdu3 machines are in the ipsilon and ipsilon_stg groups.

I see that even the ipsilon and ipsilon_stg doesn't contain iad2.centos.org machines, because we don't want to deploy them, so that is OK.

Here is the PR that adds the rdu3 machines to hbacrule.

I merged the PR and run the ipsilon playbook and the machine seems to be correctly added to HBAC rule.

Let me close this as fixed. @arrfab feel free to reopen it, if we missed something.

Metadata Update from @zlopez:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

Metadata