Fedora Copr outage announcements

This document is primarily intended for planning outages due to future infrastructure updates. However, in the event of any incidents or accidents such as networking issues, IBM Cloud problems, Fedora Rawhide repository issues, or any other matters that affect users, it’s advisable to refer to this document (possibly jump directly to the “Ongoing State” section).


Schedule an outage even if it needs to occur within the next 5 minutes!

Please familiarize yourself with the Fedora Outage SOP. But in general, follow the steps outlined in this document.

Planned outage

  1. Red Hatters-only: Schedule a calendar event, and invite the Engineering Outage Calendar room.

  2. Prepare an infrastructure ticket using one of these templates

  3. Send email to copr-devel mailing list informing about an upcomming release. We usually copy-paste text of the infrastructure ticket, including a back-reference to the infra ticket. Sending to:

    OPT-IN buildsys@lists.fedoraproject.org
    OPT-IN devel@lists.fedoraproject.org
  4. Adjust the Matrix channel title so it contains a message similar to:

    Planned outage 2022-08-17 20:00 UTC - https://pagure.io/fedora-infrastructure/issue/10854
  5. Create a new “planned” Fedora Status SOP entry.

  1. Create warning banner on Copr homepage:

    copr-frontend warning-banner --outage_time "2022-12-31 13:00-16:00 UTC" --ticket 1234

Ongoing outage

When the outage begins to cause real effects

  1. Change the “planned” Fedora Status SOP entry into an “ongoing” entry.

  2. Announce on Matrix channel — change title like s/Planned outage ../OUTAGE NOW .../ and send some message like WARNING: The scheduled outage just begings!.

Resolved outage

  1. Remove the “Outage” note from the Matrix channel title, and send a message that the outage is over!

  2. Send email to copr-devel mailing list. If there is some important change you can send email to fedora devel mailing list too. Mention the link to the “Highlights from XXXX-XX-XX release” documentation page.

  3. Propose a new “highlights” post for the Fedora Copr Blog, see the example.

  4. Close the Fedora Infra ticket.

  5. Change the “ongoing” Fedora Status SOP entry into a “resolved” one.

  6. Remove the warning banner from frontend page using copr-frontend warning-banner --remove