fedora-infrastructure

#12812 dist-git lookaside cache files served with wrong HTTP headers
Closed: Migrated to Fedora Forge by ryanlerch. Opened by decathorpe.

Closed: Migrated to Fedora Forge
Reopen Issue

It looks like .tar.gz / .crate files that are served by the lookaside cache have the following http headers set:

content-type: application/x-tar
content-encoding: x-gzip

This causes some clients (those without workarounds) to uncompress .tar.gz files on-the-fly while downloading, which results in plain tarfiles being stored on disk (but with .tar.gz file extensions).

It looks like fedpkg sources already works despite that, similar to how spectool -g also handles this case (due to a workaround that's present in its download handling code that suppresses on-the-fly decompression of gzipped files).

An indicator for when this breaks is checksum mismatches, like the one that happened in https://koji.fedoraproject.org/koji/taskinfo?taskID=137346319

Also see https://discussion.fedoraproject.org/t/verifying-the-authenticity-of-files-uploaded-to-the-lookaside-cache/134196 for a discussion thread where the "clients uncompress stuff on-the-fly even if you really don't want that to happen" was also relevant.

In general, it looks like the lookaside cache should not guess the file's MIME type or encoding, but always set content-type: application/octet-stream (which is the default for unknown / binary data): https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/MIME_types/Common_types and send responses with no content-encoding header present at all (since the data is supposed to be used as-is, without decoding first)?

The headers that are sent currently (content-type: application/x-tar, content-encoding: x-gzip) basically tell clients to "decompress this with gzip to get the .tar file you ordered", which is wrong and explains why clients that don't specifically override this behaviour get un-gzipped plain "tar" files (with ".tar.gz" file endings) or un-gzipped ".crate" files (which are just .tar.gz in disguise).

Additionally, for an as of yet unknown reason, anubis-enabled src.fedoraproject.org seems to have caused fedpkg sources from fedpkg-minimal to break with this - or a very similar - failure mode, causing checksum mismatches during koji builds. I don't see a difference in the HTTP headers that would explain that difference in behaviour though. :(

To note: I have disabled anubis for src.fedoraproject.org for now, but we should figure out a workaround so we can re-enable it.

It should be relatively easy to work around this issue in clients like fedpkg-minimal - spectool does two things for this case:

  • set Accept-Encoding: identity for GET requests
  • set decode_content=False on the stream

It appears that both are needed to cover all cases (i.e. don't break downloading plain-text files and don't decompress gzipped files sent by confused servers).

Metadata Update from @zlopez:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: high-gain, medium-trouble, ops

Yeah, although with that we don't fix it for external users? or do we... does 'fedpkg' (non minimal) do the right thing?

It appears that non-minimal fedpkg sources downloads the files correctly, but I'm not sure what's the difference - fedpkg uses pycurl and fedpkg minimal uses the curl CLI, and it looks like they use the same settings (i.e. don't appear to set any headers at all, other than Pragma:). So, I'm quite confused. :)

so... can you duplcate this with src.stg.fedoraproject.org?

(we may need to upload/sync some lookaside files there)

if so, I can reenable anubis there and I can try some fixes on the server end...

 
ben@bean:~/fedora/rust-sig/rust-jod-thread$ fedpkg -d -v sources
Creating repo object from /home/ben/fedora/rust-sig/rust-jod-thread
Initiating a koji session to https://koji.fedoraproject.org/kojihub
Downloading jod-thread-1.0.0.crate from https://src.fedoraproject.org/repo/pkgs
Full url: https://src.fedoraproject.org/repo/pkgs/rpms/rust-jod-thread/jod-thread-1.0.0.crate/sha512/14d0851a0a7d8d805a81313e6ec60a778267acd83f600d259dffbf63fe3f7ebd6a8d98d3ed49a1cb271ff024fac2c35acc1b287d5fb91f4bbe52bb3df3f2b4b1/jod-thread-1.0.0.crate
######################################################################## 100.0%
$ curl -L https://src.fedoraproject.org/repo/pkgs/rpms/rust-jod-thread/jod-thread-1.0.0.crate/sha512/14d0851a0a7d8d805a81313e6ec60a778267acd83f600d259dffbf63fe3f7ebd6a8d98d3ed49a1cb271ff024fac2c35acc1b287d5fb91f4bbe52bb3df3f2b4b1/jod-thread-1.0.0.crate | tar -tz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  6407  100  6407    0     0  18165      0 --:--:-- --:--:-- --:--:-- 18150
jod-thread-1.0.0/.cargo_vcs_info.json
jod-thread-1.0.0/.gitignore
jod-thread-1.0.0/Cargo.lock
jod-thread-1.0.0/Cargo.toml
jod-thread-1.0.0/Cargo.toml.orig
jod-thread-1.0.0/LICENSE-APACHE
jod-thread-1.0.0/LICENSE-MIT
jod-thread-1.0.0/README.md
jod-thread-1.0.0/src/lib.rs
$ curl -L https://src.stg.fedoraproject.org/repo/pkgs/rpms/rust-jod-thread/jod-thread-1.0.0.crate/sha512/14d0851a0a7d8d805a81313e6ec60a778267acd83f600d259dffbf63fe3f7ebd6a8d98d3ed49a1cb271ff024fac2c35acc1b287d5fb91f4bbe52bb3df3f2b4b1/jod-thread-1.0.0.crate
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL was not found on this server.</p>
</body></html>

Is that what you meant to try?

I can't seem to find any packages that have uploaded sources to the stg lookaside cache. Do you have an example?

The intent was that stg would have a read only copy of the prod lookaside.

It did, but there was a messed up link to it. ;)

Can you try again now? 

$ curl -I https://src.stg.fedoraproject.org/repo/pkgs/rpms/rust-jod-thread/jod-thread-1.0.0.crate/sha512/14d0851a0a7d8d805a81313e6ec60a778267acd83f600d259dffbf63fe3f7ebd6a8d98d3ed49a1cb271ff024fac2c35acc1b287d5fb91f4bbe52bb3df3f2b4b1/jod-thread-1.0.0.crate
(...)
content-type: application/x-tar
content-encoding: x-gzip
(...)

Which are the same headers as for the non-stg variant.

Running wget for both URLs (with and without .stg domain) gives me plain uncompressed .tar archives:

$ file jod-thread-1.0.0.crate 
jod-thread-1.0.0.crate: POSIX tar archive (GNU)

... which is exactly what the HTTP headers sent by the server tell wget to do.

Coincidentally, curl ing the same URLs gives me:

$ file jod-thread-1.0.0.crate
jod-thread-1.0.0.crate: gzip compressed data, was "jod-thread-1.0.0.crate", max compression, original size modulo 2^32 25088

So behaviour of what's saved to disk seems to depend on the tool that's used. FWIW I don't think "curl" is doing the correct thing here. :)

So I am a bit confused here.

stg is behaving the same was a prod? Or it's not?

The reason I wanted to check that is so that I could try and change things in staging to find a 'fix' on the server side and then we could deploy it on prod.

I just noticed that I had disabled anubis in staging src too. I am turning it back on so we can see it it has the behavior...

So I am a bit confused here.

stg is behaving the same was a prod? Or it's not?

As far as I can tell, they're giving HTTP responses with identical headers, yes - but if you disabled anubis on stg.src.fp.o too that is kind of expected :)

I'll try again later.

anubis is now enabled on src.stg again.

I tried again, looks like curl is still giving me (mostly) the same HTTP headers for both:

$ curl -I https://src.fedoraproject.org/repo/pkgs/rpms/rust-jod-thread/jod-thread-1.0.0.crate/sha512/14d0851a0a7d8d805a81313e6ec60a778267acd83f600d259dffbf63fe3f7ebd6a8d98d3ed49a1cb271ff024fac2c35acc1b287d5fb91f4bbe52bb3df3f2b4b1/jod-thread-1.0.0.crate | sort 
accept-ranges: bytes
apptime: D=64700
content-encoding: x-gzip
content-length: 6407
content-type: application/x-tar
date: Fri, 10 Oct 2025 18:21:42 GMT
etag: "1907-63f20ec0c5b00"
HTTP/2 200 
last-modified: Fri, 19 Sep 2025 05:41:32 GMT
referrer-policy: same-origin
server: Apache
strict-transport-security: max-age=31536000; includeSubDomains; preload
x-content-type-options: nosniff
x-fedora-appserver: pkgs01.rdu3.fedoraproject.org
x-fedora-proxyserver: proxy10.rdu3.fedoraproject.org
x-fedora-requestid: aOlOthUO8dOW_ab7KBDV2QAI0xQ
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block

$ curl -I https://src.stg.fedoraproject.org/repo/pkgs/rpms/rust-jod-thread/jod-thread-1.0.0.crate/sha512/14d0851a0a7d8d805a81313e6ec60a778267acd83f600d259dffbf63fe3f7ebd6a8d98d3ed49a1cb271ff024fac2c35acc1b287d5fb91f4bbe52bb3df3f2b4b1/jod-thread-1.0.0.crate | sort 
accept-ranges: bytes
apptime: D=73227
content-encoding: x-gzip
content-length: 6407
content-type: application/x-tar
date: Fri, 10 Oct 2025 18:21:46 GMT
etag: "1907-63f20ec0c5b00"
HTTP/2 200 
last-modified: Fri, 19 Sep 2025 05:41:32 GMT
referrer-policy: same-origin
server: Apache
strict-transport-security: max-age=31536000; includeSubDomains; preload
x-content-type-options: nosniff
x-fedora-appserver: pkgs01.stg.rdu3.fedoraproject.org
x-fedora-proxyserver: proxy02.stg.rdu3.fedoraproject.org
x-fedora-requestid: aOlOug3vXHWevdWTx-OcoQAAENE
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block

@music can you try reproducing the issue again? because I can't seem to see any differences here.

(The headers are still wrong IMO - they should tell clients to use the data as-is, not that it needs to be gzip compressed on receipt - but they're not different between stg and prod as far as I can tell.)

Looks like someone else hit this in #12842

I have switched prod anubis off on src for now.

Rough and ditry pr... I applied it in staging already, and it changes the content-type... but it still ungzips it.

https://pagure.io/fedora-infra/ansible/pull-request/2904

ok. I had to put it back... things were just too bad without anubis. ;(

That said, the internal proxies that are used by builders don't have it enabled, so I think real builds should work fine.

Rough and ditry pr... I applied it in staging already, and it changes the content-type... but it still ungzips it.

https://pagure.io/fedora-infra/ansible/pull-request/2904

Hmm, I wouldn’t expect changing just the Content-Type to affect the Content-Encoding (unless it happened to interact with a pre-existing MIME-based rule for content encoding), so at least that result is explicable.

I tried fedpkg sources for rust-jod-thread, and it’s failing again:

$ fedpkg sources
Downloading jod-thread-1.0.0.crate from https://src.fedoraproject.org/repo/pkgs
Could not execute sources: jod-thread-1.0.0.crate failed checksum

I tried fedpkg scratch-build and that failed, too:

DEBUG util.py:461:  jod-thread-1.0.0.crate: FAILED
DEBUG util.py:459:  sha512sum: WARNING: 1 computed checksum did NOT match

https://koji.fedoraproject.org/koji/taskinfo?taskID=138160053

That said, the internal proxies that are used by builders don't have it enabled, so I think real builds should work fine.

I think a (dist-git based, not --srpm) scratch build should be the same as a real build, right? If so, then I expect real builds to fail as well.

I haven’t taken the time to dig through the Ansible configurations and understand how they actually fit together, but is the lookaside cache governed by the mod_deflate configuration in https://pagure.io/fedora-infra/ansible/blob/main/f/roles/fedora-web/main/files/deflate.conf ? Might something like this work?

<Directory /srv/cache/lookaside>
    SetOutputFilter NONE
    […]
</Directory>

I’m not sure if you would need to mess around with the Vary header as well or not, or if something like

<Directory /srv/cache/lookaside>
    SetEnv no-gzip
    SetEnv dont-vary
    […]
</Directory>

would be processed in the right order with respect to the mod_deflate configuration to have the desired effect.

I think the "content-encoding: x-gzip" part is coming from the proxies. But while it's not useful for lookaside data, clients should deal with that.

I think the "content-encoding: x-gzip" part is coming from the proxies. But while it's not useful for lookaside data, clients should deal with that.

The clients do deal with it, by removing the declared content-encoding to obtain what the server is claiming should be the actual payload. The problem is that the content-encoding header is lying. For it to be accurate, the HTTP response would have to be gzipped (again, on top of the gzip encoding that happens to be the outermost layer in the .crate file format).

OOF this is now hitting me too ...

Is there anything we can do, at least as a temporary workaround? Maybe patch fedpkg-minimal (or whatever downloads the sources in the buildSRPMfromSCM task in koji) to avoid decompressing stuff on-the-fly?

It appears that adding -H "Accept-Encoding: identity" to the curl CLI call in fedpkg-minimal works around this issue. I'll file a PR with that ...

https://src.fedoraproject.org/rpms/fedpkg-minimal/pull-request/2

I think thats fine, but it's going to take a while to get that to where it can help. I was hoping we could do something on the server side to get things working faster. ;(

Sigh, I found that dns was wrong and internal stuff was using proxy01/10 instead of proxy101/110.
I have changed it over now, please retry your scratch build from scm and I hope it will work.

Ha, it's always DNS :laughing: I can confirm that a build that failed earlier this evening due to a checksum issue now passed. :balloon:

sccache also started building for me

Okay, I'm going to close this as it looks like it was all DNS :(

Metadata Update from @james:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

Okay, I'm going to close this as it looks like it was all DNS :(

I wouldn’t consider this fixed. This is no longer breaking koji builds from dist-git, because the DNS issue causing koji to use the external proxies was fixed. However, things are still utterly broken for anything that accesses the lookaside cache externally, including COPR builds from dist-git (e.g. https://copr.fedorainfracloud.org/coprs/music/tungstenite/build/9704757/) and things like:

rust-jod-thread$ fedpkg sources
Downloading jod-thread-1.0.0.crate from https://src.fedoraproject.org/repo/pkgs
Could not execute sources: jod-thread-1.0.0.crate failed checksum

Trying to find, patch, and redeploy every piece of code that accesses the lookaside patch to set Accept-Encoding: identity à la https://src.fedoraproject.org/rpms/fedpkg-minimal/pull-request/2 is a plausible workaround, but the root problem is that the server and/or proxy are confused and are lying about what they are serving by tacking on a Content-Encoding: gzip header but not actually adding another layer of gzip compression.

Yeah. We should still try and fix the server end if we can, or failing that at least fix fedpkg/fedpkg-minimal.

Metadata Update from @kevin:
- Issue status updated to: Open (was: Closed)

Okay, I'm going to close this as it looks like it was all DNS :(

I wouldn’t consider this fixed. This is no longer breaking koji builds from dist-git, because the DNS issue causing koji to use the external proxies was fixed. However, things are still utterly broken for anything that accesses the lookaside cache externally, including COPR builds from dist-git (e.g. https://copr.fedorainfracloud.org/coprs/music/tungstenite/build/9704757/) and things like:

```
rust-jod-thread$ fedpkg sources
Downloading jod-thread-1.0.0.crate from https://src.fedoraproject.org/repo/pkgs

Could not execute sources: jod-thread-1.0.0.crate failed checksum
```

Ah, so fedpkg sources really is affected too? That explains why I couldn't find a reason why it shouldn't be :(

Does anybody want to submit a PR to fedpkg equivalent to the one I sent for fedpkg-minimal, or should I do that?

ok, I updated anubis today... can you see if it changed anything? There's a bunch of changes, but it's hard to tell if any would affect this issue.

 
ben@bean:~/fedora/rust-sig/rust-jod-thread$ fedpkg sources
Downloading jod-thread-1.0.0.crate from https://src.fedoraproject.org/repo/pkgs
Could not execute sources: jod-thread-1.0.0.crate failed checksum

Looks the same to me.

Filed a PR for rpkg too: https://pagure.io/rpkg/pull-request/758 , PTAL.

I just encountered this problem in a real Koji build again.

https://koji.fedoraproject.org/koji/taskinfo?taskID=139122958

I just encountered this problem in a real Koji build again.

Just to clarify the impact, the fact that this is happening in koji builds from dist-git again means that a large number of Rust packages will fail to build. No external workaround exists, and much Rust packaging work will be impossible until some kind of fix is implemented.

ok, I dug into this and I think I see what the problem is.

I did make changes to varnish to cache src, and I think it's doing the same thing anubis did...

I'll look at a commit to fix this.

ok, please try now. I tested and it seems to be working in that testing.

ok, please try now. I tested and it seems to be working in that testing.

Yes, it looks like it’s working again!

https://koji.fedoraproject.org/koji/taskinfo?taskID=139139537

Thank you for investigating.

Sorry for the trouble... ;( I knew varnish would break something there, but it's really helping the scraper load...

Sorry for the trouble... ;( I knew varnish would break something there, but it's really helping the scraper load...

We certainly need all the help we can get! Thanks for looking at it. I’m glad it wasn’t too difficult to get things back to the status quo.

This issue has been migrated to Fedora Forge:
https://forge.fedoraproject.org/infra/tickets/issues/12812

Please continue any further discussion there.

Metadata Update from @ryanlerch:
- Issue close_status updated to: Migrated to Fedora Forge
- Issue status updated to: Closed (was: Open)
Metadata
None

high-gain medium-trouble ops

None
None
Normal
Related Pull Requests
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.