Koji 1.36.0 Release notes¶
All changes can be found in the roadmap. Most important changes are listed here.
This is a fairly substantial release. Notable changes include:
Migrating from Koji 1.35.x¶
For details on migrating see Migrating to Koji 1.36
Security Fixes¶
None
System Changes¶
Support rpm v6 format
This adds initial support for v6 format rpms in Koji. The main issue was Koji’s past reliance on the sigmd5 field, which does not exist in v6. Koji will now store and report multiple digest values from the rpm header, not just sigmd5.
In light of these changes the Content Generator Metadata guidelines have been updated.
One notable omission – the oz/imagefactory feature (i.e. createImage) is not covered here (it will fail if an image includes v6 rpms).
Show more info on waitrepo task failures
This causes the client and waitrepo tasks to show a little more detail when repo requests fail.
Client specified signature keys
More complex signature headers cannot be described by a single key. RPM now supports cases like:
rpms signed with multiple keys
rpms signed with newer types of keys
rpms that include ima signing
Koji now takes the approach of treating the sigkey field as a descriptive label for the
signature that can be specified at import or changed later.
This gives us the ability to handle more complex signing without radically altering the
api or file layout.
Policy enhancements
This extends the existing policy handling code adding support for:
negated tests – use
!before a test to negate itbreak N action – break out of subrules without matching
stop action – stop the policy check
flag action – set a named flag (and continue) when a rule matches
flagged test – test for a named flag
multi match rules – option for policy hooks to check for multiple results
These changes are backwards compatible; no policies should require update. Admins may find that using these new options can help simplify complex logic in their existing policies.
The new support for multi-match is forward looking, and is not currently utilized in current policy checks.
Deprecate pathinfo.tmpdir
The pathinfo.tmpdir method in the koji library has never been used by Koji and is now
deprecated.
Hub Changes¶
Disable url sessions by default
The DisableURLSessions setting on the hub now defaults to True.
Koji clients before 1.31 will not be able to authenticate
If an admin wants to support auth for clients before 1.31, they can turn this back on.
Allow querying headers from signed copy
This adds a sigkey option to getRPMHeaders. If given, the corresponding signed copy will be queried instead of the primary copy. This allows querying signature headers that have changed in the signed copy (i.e. the signatures)
Adjust repo lag logic
This change adjusts the repo lag logic and obsoletes the RepoLagWindow setting.
The lag window was intended to avoid redundant regenerations for duplicate requests within a short span of time. Unfortunately, there are still a number of cases where nearby requests fall into separate windows. Depending on settings and repo regen time, this can even result in overlapping newRepo tasks for the same tag.
Recursive task policy data
When creating a new task, the data used for the policy tests includes values pulled from the task
parameters.
This is now handled recursively. That is, the parameters of any parent task also apply.
In particular, this allows making channel policy rules for buildArch tasks based on the target
value of the parent build task.
Capture more events in scheduler logs
The scheduler now includes more detail in the scheduler logs.
This data will appear in the output of koji scheduler-logs
Enable parent tag as name, not ID only
Previously the setInheritanceData call required specifying parent tags by id.
Now they can be specified by name as well.
Additional data for cg volume policy
This makes it possible to consider cg and draft status in the volume policy.
Channel defaults
This change allows policy to distinguish between a user channel override and a system provided
default that is different from default (e.g. for maven and image builds).
There is a new action for the channel policy named default which indicates that the default
channel should be used, regardless of what that default is.
Currently, this change must be enabled by setting DefaultChannelCompat = False.
This default will change to True in a future release.
Add listBuilds query params for promotion data
This allows querying builds by promoter or promotion time
Less log noise for empty or invalid requests
Avoid noisy log output for invalid requests
Adjust upload offset and overwrite logic
This fix addresses a corner case that could cause failed uploads.
Accept both comma-separated and space-separated arches for external repos
This makes calls like addExternalRepoToTag and editTagExternalRepo a bit friendlier.
Match longest extension first for archivetype
This addresses a longstanding bug in get_archive_type. In cases of layered extensions, the code would match a shorter extension before a longer one. E.g. “.gz” would match before “.tar.gz”. We have avoided such overlapping archivetypes for this reason, however we can’t control what instances do.
Report tag_external_repos.arches in query_history
This fixes an oversight in query_history. The arches field of this table was omitted.
Allow setting ttl in protonmsg plugin
A new configuration option for the protonmsg plugin
Partial draft support for cg_import
This removes some obstacles for importing draft builds via the CG interface.
Generators can now set draft: true in the metadata to import draft builds, but this is
currently only allowed for rpm builds.
Admins can control such imports in the cg_import policy by using the is_draft test in a
match rule.
Additionally, the new AllowDraftComponents hub option allows content generator imports to
reference draft builds as components (e.g. a buildroot or image component for an imported build).
We recommend that admins keep this option off (the default) unless they are sure that they need it.
Client Changes¶
Enable anonymous retries by default
The default value for the anon_retry setting is now True.
This will cause the client to retry hub calls even when not authenticated.
Add override option to assign-task
This adds an --override option to the assign-task command.
This option prevents the scheduler from reassigning the task.
Print build in import-cg
Print basic build info after import
Clean up cli for comps management
Adds missing commands for comps management and some new documentation for using these commands.
See also: Managing Comps Data
Enable specific number of builds for dist-repo tasks
The dist-repo command now accepts a --latest-n N argument to use the N latest builds per
package instead of just the latest.
Support –wait-repo/–wait-build in wrapper-rpm command
Added --wait-repo/--wait-build options similar to those for the build command.
Download fallback to unsigned
Previously, if you passed --key to download-build and signed packages weren’t available,
Koji would either skip the unsigned package or error out.
This adds a modified behavior controlled by the new --fallback-unsigned arg.
If this is passed with --key, unsigned copies will be downloaded for packages for which no
signed copy can be found.
Web Changes¶
Port kojiweb to jinja2
The Koji web interface now uses Jinja2 templates instead of Cheetah templates.
Compatibility was a major focus in this port. The web output remains predominantly the same, and url compatibility is maintained.
Note: Cheetah3 is still required on builders for wrapper rpm templates
Add repo requests web ui
This makes repo requests more visible in the web interface.
new page for listing repo requests
new page for request details
new details on the repoinfo page
link from taginfo page to related requests
link from taskinfo page if it is for a request
Add weblink to NVR build for draft builds
This change provides a convenient link from the draft build page when it has been superceded by another build.
Add cg and draft filters for builds page
This change adds more filter options to the builds page in the web ui.
Show source files on srpm info page
When viewing a source rpm in the web ui, the source files are now listed.
Builder Changes¶
Ensure macros and env values passed to mock are strings
If an admin sets a macro value to a non-string value (which is easy to do by accident if the value is a number), the builder will now convert such values to a string.
Improve reliability of kojid service start
If chrony is enabled, wait for it to start before starting kojid.
Return BuildError when kickstart file does not exist
Clarify an error condition
Handle new mock option - bootstrap_image_ready
Admins can set mock.bootstrap_image_ready for a tag and this option will be added to the mock
config.
See also: https://rpm-software-management.github.io/mock/Feature-container-for-bootstrap
Devel and testing¶
New dev tool: fakepolicy
This adds a new script under devtools that facilitates testing policy code against a remote instance. The script runs some heavily mocked hub code that calls out to a remote instance for actual data access.
The reason for this tool’s existence is that policy code is very hard to test without actual data, which is usually locked up inside a Koji instance. This script allows you to see what a given policy would do, without actually deploying it.
Add kojid –single mode
This is an option for debugging kojid. Using this option makes it feasible to use the python debugger on a task handler.
Unit test fixes
This release includes numerous fixes that could cause unit tests to fail in certain environments.
Bug fixes¶
Work around nfs glitch in ensuredir
In a multi-hub + nfs scenario, we’ve seen race situations where mkdir fails with EEXIST, but an immediate lstat does not find the directory. This change works around the unexpected kernel behavior.
Fixes for get_next_build
These fixes address cases where the automatic release incrementing mechanisms for image builds could fail.
Handle disabled hosts in scheduler
This fixes an issue with the how the scheduler handles tasks assigned to disabled builders.
Catch nvra duplicates when promoting
When promoting a build, we check for existing NVR conflicts and give a nice error. However, we did not do the same for the included NVRAs, which are also required to be unique. There are corner cases where a draft can have nvra overlap without nvr overlap. This is still an error case, but the error is now more informative.
Always prefer task arch in find_arch()
This addresses an issue where arch-independent tasks like rebuildSRPM and buildSRPMFromSCM could choose the wrong buildroot architecture.
Fix callnum lookup on hub
Since 1.31, Koji clients have used http headers to pass their session info. However, the hub has been misreading the callnum since that time.
Fix draft symlink handling
When a draft build is promoted, Koji maintains a symlink at the original location. These fixes address cases where the hub did not create that link.
As part of this fix, the system now requires that each volume directory contain a
toplink symlink that points to the main volume.
We recommend using an absolute symlink. E.g.
$ file /mnt/koji/vol/vol3/toplink
/mnt/koji/vol/vol3/toplink: symbolic link to /mnt/koji
When adding a new volume, admins should create this symlink when they create the volume directory. If not, the hub will automatically create it using the configured topdir path and log a warning.
For existing volumes, admins must create this symlink for each preexisting volume during their migration to 1.36. Failure to do so can result in build promotion failures.
Fix listUsers type check
Support passing a list for userType
Ensure repos dir
Create the repos dir if it does not exist.
Write-signed-rpm: fix nvr fallback
This fixes a bug in the write-signed-rpm command where it would fail on valid NVR values.
Use label for waitrepo subtasks
This change prevents restarted tasks from spawning a redundant waitrepo subtask
Prefer getBuildLogs for downloading logs
Fixes a common failure case for log downloads
Windows build fixes
Various small fixes to the windows build daemon.
Handle x86_64_v2 arch for dist-repos
This fixes a KeyError when trying to make a distrepo for the arch “x86_64_v2”.
Other changes¶
Update Cheetah3 to CT3 PyPI repository
Note: While Koji now uses Jinja2 in the web ui, Cheetah3 is still required for wrapper rpm templates.
This simply updates our requirements to point to the new upstream.
koji-gc: Add ccache to koji-gc options
Allows specifying a Kerberos credential cache for koji-gc, similar to kojira and kojid.
Indexes for standard_buildroot
Adds database indexes for query performance
Schemas: wsl file suffix
The “new” WSL images need a WSL suffix. Currently this is managed on a per-instance basis but it’s a bit of work to sync across them.
Add archive types for erofs and squashfs
Add archive types produced by Kiwi
Drop custom threadlocal implementation
Koji previously used its own ThreadLocal class.
Now it just uses threading.local.
Raise weight cap
This adjusts the cap on how much kojid can adjust the weight of a buildArch task.
Documentation updates
Various documentation changes