How to release Copr¶
Go through this page well before you will do the release. Maybe you will want to do some steps in different order, and in any case, it’s good to know what’s ahead.
Keep amending this page if you find something not matching reality or expectations.
The goal is to do as much work pre-release as possible while focusing only on important things and not creating a work overload with tasks, that can be done post-release.
Tag untagged packages that have changes in them¶
Make sure you are on the
main branch and that it is up-to-date:
git checkout main git pull --rebase
tito report --untagged-commits
and walk the directories of packages listed, and tag them. During development, we sometimes put .dev suffix into our packages versions. See what packages has such version:
cat ./.tito/packages/* |grep ".dev"
If a package has .dev suffix, manually increment its version:
tito tag --use-version X.Y
For others, new version can be bumped automatically:
Make sure that the %changelog is nice and meaningful, i.e. remove the frontend:, rpmbuild:, etc. prefixes and filter-out entries which are not interesting for the package end-users (git-log != %changelog). Later on, if properly polished, the %changelogs’ contents may be used for filling the Bodhi update text.
Push all new tags at once:
git push --follow-tags origin
Build all the updated packages into
@copr/copr copr project:
copr build-package @copr/copr --nowait --name python-copr copr build-package @copr/copr --nowait --name copr-frontend ...
Upgrade -dev machines¶
Check that .repo files correctly points to
@copr/copr. And run on batcave01.iad2.fedoraproject.org (if you do not have account there ask Mirek or somebody from fedora-infra):
sudo rbac-playbook -l copr-be-dev.aws.fedoraproject.org \ manual/copr/copr-backend-upgrade.yml sudo rbac-playbook -l copr-be-dev.aws.fedoraproject.org groups/copr-backend.yml sudo rbac-playbook -l copr-keygen-dev.aws.fedoraproject.org \ manual/copr/copr-keygen-upgrade.yml sudo rbac-playbook -l copr-keygen-dev.aws.fedoraproject.org groups/copr-keygen.yml sudo rbac-playbook -l copr-fe-dev.aws.fedoraproject.org \ manual/copr/copr-frontend-upgrade.yml sudo rbac-playbook -l copr-fe-dev.aws.fedoraproject.org groups/copr-frontend.yml sudo rbac-playbook -l copr-dist-git-dev.aws.fedoraproject.org \ manual/copr/copr-dist-git-upgrade.yml sudo rbac-playbook -l copr-dist-git-dev.aws.fedoraproject.org groups/copr-dist-git.yml sudo rbac-playbook -l copr-pulp-dev.aws.fedoraproject.org \ manual/copr/copr-pulp-upgrade.yml sudo rbac-playbook -l copr-pulp-dev.aws.fedoraproject.org groups/copr-pulp.yml
If there is a new version of copr-rpmbuild, follow the Terminate the old instances and Terminate resalloc resources instructions.
Make sure expected versions of Copr packages are installed on the dev instances:
./releng/run-on-all-infra --devel 'rpm -qa | grep copr'
Call for QA¶
Move MODIFIED+ bugzillas to ON_QA.
Ask people to test, verify bugs, and generally help with QA. They will ignore it but you will feel good about giving them a chance.
Run the sanity tests from a Podman container (alternatively this can be run also from Beaker directly).
Build packages for production¶
Make sure that
.tito/releasers.conf has up to date list of branches.
Make sure you are co-maintainer of those packages in Fedora:
copr-backend copr-cli copr-dist-git copr-frontend copr-keygen copr-messaging copr-mocks copr-rpmbuild copr-selinux python-copr python-copr-common
For each package do:
cd <package subdir> # run this for python-copr and copr-cli tito release fedora-git-clients # run this for copr-messaging package tito release fedora-git-messaging # run this for copr-rpmbuild and python3-copr-common package tito release fedora-git-common # run this for other (server) packages (copr-frontend, copr-backend, ...) tito release fedora-git
Koji doesn’t automatically put successfully built packages into the buildroot
for the following builds and therefore you can easily encounter failures of
copr-cli or copr server pacakges because of a missing dependency to
python3-copr-common that you have just built in Koji. To
fix this, you need to create a
Bodhi override for those dependencies.
It takes up to 30 minutes to for the override to be available in the buildroot:
koji wait-repo f34-build --build=python-copr-common-0.13-1.fc34
Tito doesn’t work properly with more than one source, and when releasing
backend, it removes
test-data-copr-backend-2.tar.gz from the DistGit
sources file. Until it gets resolved,
fix this way.
Prepare release notes¶
Go over bugs, which were resolved. Write some nice announce. It is useful to prepare the release notes beforehand because developers usualy don’t remember what they worked on and therefore don’t know what to test once production instances are upgraded. Sharing the prepared notes with team members before doing the actuall release is appreciated.
See previous release notes and try to format them in the same way. Then create a pull request with this release notes against Copr git repository.
Schedule and announce the outage¶
Schedule outage even if it has to happen in the next 5 minutes!
Get faimiliar with the Fedora Outage SOP. In general, please follow these steps:
Prepare the infrastructure ticket similar to this old one.
Send email to copr-devel mailing list informing about an upcomming release. We usually copy-paste text of the infrastructure ticket created in a previous step. Don’t forget to put a link to the ticket at the end of the email. See the example.
op #fedora-buildsys MyIrcNickmessage to
ChanServon libera.chat to get the OP rights, and then adjust the channel title so it starts with message similar to:
Planned outage 2022-08-17 20:00 UTC - https://pagure.io/fedora-infrastructure/issue/10854
Create a new “planned” Fedora Status SOP entry.
Create warning banner on Copr homepage:
copr-frontend warning-banner --outage_time "2022-12-31 13:00-16:00 UTC" --ticket 1234
If all the pre-release preparations were done meticulously and everything was tested properly, the release window shouldn’t take more than ten minutes. That is, if nothing goes terribly sideways…
Let users know¶
Change the “planned” Fedora Status SOP entry into an “ongoing” entry.
#fedora-buildsys, change title like
s/Planned outage ../OUTAGE NOW .../and send some message like
WARNING: The scheduled outage just begings!.
At this moment, every Copr service should be up and running.
Generate Copr project documentation
cd doc ./update_docs.sh
Generate package specific documentation by going to:
And hitting “Build” button for each of those projects.
If schema was modified you should generate new Schema documentation.
Announce the end of the release¶
Remove the “Outage” note from the
Send a message on
fedora-buildsysthat the outage is over!
Send email to copr-devel mailing list. If there is some important change you can send email to fedora devel mailing list too. Mention the link to the “Highlights from XXXX-XX-XX release” documentation page.
Propose a new “highlights” post for the Fedora Copr Blog, see the example.
Close the Fedora Infra ticket.
Change the “ongoing” Fedora Status SOP entry into a “resolved” one.
Remove the warning banner from frontend page using
copr-frontend warning-banner --remove
Release packages to PyPI¶
Make sure you have ~/.pypirc correctly set up and run:
dnf install twine python3 setup.py sdist twine upload dist/<NAME-VERSION>.tar.gz
If you cannot run that, tell somebody with access to run that (msuchy, praiskup, jkadlcik).
This needs to be run for copr-common, python, copr-cli and copr-messaging.
Submit Bodhi updates¶
Create updates in Bodhi for every package built in Koji.
It is useful to do updates in batches, e.g. to group several packages into one
update. You can do this by
fedpkg update, with the following template:
[ copr-backend-1.127-1.fc31, copr-frontend-1.154-1.fc31] type=enhancement notes=copr-frontend - change 1 in frontend - change 2 in frontend copr-backend - change 1 in backend - change 2 in backend
It is often good idea to put new (filtered)
%changelogs entries there.
Check if the MODIFIED bugs (that are not ON_QA) are fixed in released Copr or not, move them ON_QA.
Change status of all ON_DEV, ON_QA, VERIFIED, and RELEASE_PENDING bugs to CLOSED/CURRENTRELEASE with comment like ‘New Copr has been released.’
Fix this document to make it easy for the release nanny of the next release to use it.