kernel-tests

#50 Submit no longer working in F39 + lmbench warnings
Closed: Fixed by tharadash. Opened by tharadash.

Closed: Fixed
Reopen Issue

Hi there
On my main computer running F39 as well as running kerneltest-6.8.1.iso on it and on another notebook computer, I'm unable to submit the results with attached errors.
As far as I remember, the last kernel-test I successfully submitted using my main computer was one or two weeks ago.

After encountering that yesterday, I deleted the kernel-tests folder an re-pulled it from git.
After that, I received a number of lmbench warnings (system language is set to german - sorry!).

Let me know what specific package version information you need (or any other information for that matter!).

Submit errors:
`[snip]
Test suite complete PASS

Your log file is being submitted...
Traceback (most recent call last):
File "/home//kernel-tests/./fedora_submit.py", line 43, in
submitclient.login(
File "/usr/lib/python3.12/site-packages/fedora/client/openidbaseclient.py", line 303, in login
response = openid_login(
^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/fedora/client/openidproxyclient.py", line 138, in openid_login
raise AuthError(output['message'])
fedora.client.AuthError: Invalid request
The following information is not submitted with your log;
it is for informational purposes only.
[snip]`

lmbench warnings:
Test suite called with performance lib_timing.c: In Funktion »touch«: lib_timing.c:1626:17: Warnung: »int« ist Standardtyp in Deklaration von »psize« [-Wimplicit-int] 1626 | static psize; | ^~~~~ lib_sched.c: In Funktion »handle_scheduler«: lib_sched.c:94:17: Warnung: »return« ohne Wert in nicht-void zurückgebender Funktion 94 | return; | ^~~~~~ lib_sched.c:46:1: Anmerkung: hier deklariert 46 | handle_scheduler(int childno, int benchproc, int nbenchprocs) | ^~~~~~~~~~~~~~~~ lib_sched.c: In Funktion »sched_pin«: lib_sched.c:207:26: Warnung: Implizite Deklaration der Funktion »sched_getaffinity« [-Wimplicit-function-declaration] 207 | retval = sched_getaffinity(0, sz * sizeof(unsigned long), cpumask); | ^~~~~~~~~~~~~~~~~ lib_sched.c:231:18: Warnung: Implizite Deklaration der Funktion »sched_setaffinity« [-Wimplicit-function-declaration] 231 | retval = sched_setaffinity(0, sz * sizeof(unsigned long), mask); | ^~~~~~~~~~~~~~~~~ bw_file_rd.c: In Funktion »main«: bw_file_rd.c:149:44: Warnung: Implizite Deklaration der Funktion »lmbench_usage« [-Wimplicit-function-declaration] 149 | if (parallel <= 0) lmbench_usage(ac, av, usage); | ^~~~~~~~~~~~~ bw_mem.c: In Funktion »main«: bw_mem.c:76:44: Warnung: Implizite Deklaration der Funktion »lmbench_usage« [-Wimplicit-function-declaration] 76 | if (parallel <= 0) lmbench_usage(ac, av, usage); | ^~~~~~~~~~~~~ bw_mmap_rd.c: In Funktion »main«: bw_mmap_rd.c:64:44: Warnung: Implizite Deklaration der Funktion »lmbench_usage« [-Wimplicit-function-declaration] 64 | if (parallel <= 0) lmbench_usage(ac, av, usage); | ^~~~~~~~~~~~~ bw_pipe.c: In Funktion »main«: bw_pipe.c:157:44: Warnung: Implizite Deklaration der Funktion »lmbench_usage« [-Wimplicit-function-declaration] 157 | if (parallel <= 0) lmbench_usage(ac, av, usage); | ^~~~~~~~~~~~~ bw_tcp.c: In Funktion »main«: bw_tcp.c:74:44: Warnung: Implizite Deklaration der Funktion »lmbench_usage« [-Wimplicit-function-declaration] 74 | if (parallel <= 0) lmbench_usage(ac, av, usage); | ^~~~~~~~~~~~~ bw_unix.c: In Funktion »main«: bw_unix.c:153:44: Warnung: Implizite Deklaration der Funktion »lmbench_usage« [-Wimplicit-function-declaration] 153 | if (parallel <= 0) lmbench_usage(argc, argv, usage); | ^~~~~~~~~~~~~ lat_select.c: In Funktion »main«: lat_select.c:53:44: Warnung: Implizite Deklaration der Funktion »lmbench_usage« [-Wimplicit-function-declaration] 53 | if (parallel <= 0) lmbench_usage(ac, av, usage); | ^~~~~~~~~~~~~ lat_select.c: In Funktion »doit«: lat_select.c:167:16: Warnung: »int« ist Standardtyp in Deklaration von »count« [-Wimplicit-int] 167 | static count = 0; | ^~~~~ lat_pipe.c: In Funktion »main«: lat_pipe.c:41:44: Warnung: Implizite Deklaration der Funktion »lmbench_usage« [-Wimplicit-function-declaration] 41 | if (parallel <= 0) lmbench_usage(ac, av, usage); | ^~~~~~~~~~~~~ lat_rpc.c: In Funktion »main«: lat_rpc.c:136:33: Warnung: Implizite Deklaration der Funktion »lmbench_usage« [-Wimplicit-function-declaration] 136 | lmbench_usage(ac, av, usage); | ^~~~~~~~~~~~~ lat_syscall.c: In Funktion »main«: lat_syscall.c:116:44: Warnung: Implizite Deklaration der Funktion »lmbench_usage« [-Wimplicit-function-declaration] 116 | if (parallel <= 0) lmbench_usage(ac, av, usage); | ^~~~~~~~~~~~~ lat_tcp.c: In Funktion »main«: lat_tcp.c:64:33: Warnung: Implizite Deklaration der Funktion »lmbench_usage« [-Wimplicit-function-declaration] 64 | lmbench_usage(ac, av, usage); | ^~~~~~~~~~~~~ lat_udp.c: In Funktion »main«: lat_udp.c:78:33: Warnung: Implizite Deklaration der Funktion »lmbench_usage« [-Wimplicit-function-declaration] 78 | lmbench_usage(ac, av, usage); | ^~~~~~~~~~~~~ lat_mmap.c: In Funktion »main«: lat_mmap.c:58:33: Warnung: Implizite Deklaration der Funktion »lmbench_usage« [-Wimplicit-function-declaration] 58 | lmbench_usage(ac, av, usage); | ^~~~~~~~~~~~~ mhz.c: In Funktion »main«: mhz.c:492:25: Warnung: Implizite Deklaration der Funktion »lmbench_usage« [-Wimplicit-function-declaration] 492 | lmbench_usage(ac, av, usage); | ^~~~~~~~~~~~~ lat_proc.c: In Funktion »main«: lat_proc.c:60:44: Warnung: Implizite Deklaration der Funktion »lmbench_usage« [-Wimplicit-function-declaration] 60 | if (parallel <= 0) lmbench_usage(ac, av, usage); | ^~~~~~~~~~~~~ lat_pagefault.c: In Funktion »main«: lat_pagefault.c:54:44: Warnung: Implizite Deklaration der Funktion »lmbench_usage« [-Wimplicit-function-declaration] 54 | if (parallel <= 0) lmbench_usage(ac, av, usage); | ^~~~~~~~~~~~~ lat_connect.c: In Funktion »main«: lat_connect.c:63:25: Warnung: Implizite Deklaration der Funktion »lmbench_usage« [-Wimplicit-function-declaration] 63 | lmbench_usage(ac, av, usage); | ^~~~~~~~~~~~~ lat_fs.c: In Funktion »main«: lat_fs.c:55:44: Warnung: Implizite Deklaration der Funktion »lmbench_usage« [-Wimplicit-function-declaration] 55 | if (parallel <= 0) lmbench_usage(ac, av, usage); | ^~~~~~~~~~~~~ /usr/bin/ld: /tmp/cc81cP6N.o: in functionsetup_names':
lat_fs.c:(.text+0x833): warning: the use of tempnam' is dangerous, better usemkstemp'
lat_sig.c: In Funktion »main«:
lat_sig.c:182:44: Warnung: Implizite Deklaration der Funktion »lmbench_usage« [-Wimplicit-function-declaration]
182 | if (parallel <= 0) lmbench_usage(ac, av, usage);
| ^~~~~~~~~~~~~
lat_mem_rd.c: In Funktion »main«:
lat_mem_rd.c:47:44: Warnung: Implizite Deklaration der Funktion »lmbench_usage« [-Wimplicit-function-declaration]
47 | if (parallel <= 0) lmbench_usage(ac, av, usage);
| ^~~~~~~~~~~~~
lat_ctx.c: In Funktion »main«:
lat_ctx.c:74:44: Warnung: Implizite Deklaration der Funktion »lmbench_usage« [-Wimplicit-function-declaration]
74 | if (parallel <= 0) lmbench_usage(ac, av, usage);
| ^~~~~~~~~~~~~
lat_sem.c: In Funktion »main«:
lat_sem.c:42:44: Warnung: Implizite Deklaration der Funktion »lmbench_usage« [-Wimplicit-function-declaration]
42 | if (parallel <= 0) lmbench_usage(ac, av, usage);
| ^~~~~~~~~~~~~
lat_unix.c: In Funktion »main«:
lat_unix.c:46:44: Warnung: Implizite Deklaration der Funktion »lmbench_usage« [-Wimplicit-function-declaration]
46 | if (parallel <= 0) lmbench_usage(ac, av, usage);
| ^~~~~~~~~~~~~
lat_fifo.c: In Funktion »main«:
lat_fifo.c:45:44: Warnung: Implizite Deklaration der Funktion »lmbench_usage« [-Wimplicit-function-declaration]
45 | if (parallel <= 0) lmbench_usage(ac, av, usage);
| ^~~~~~~~~~~~~
lat_http.c: In Funktion »main«:
lat_http.c:84:25: Warnung: Implizite Deklaration der Funktion »lmbench_usage« [-Wimplicit-function-declaration]
84 | lmbench_usage(ac, av, usage);
| ^~~~~~~~~~~~~
lat_fcntl.c: In Funktion »main«:
lat_fcntl.c:203:44: Warnung: Implizite Deklaration der Funktion »lmbench_usage« [-Wimplicit-function-declaration]
203 | if (parallel <= 0) lmbench_usage(ac, av, usage);
| ^~~~~~~~~~~~~
lat_unix_connect.c: In Funktion »main«:
lat_unix_connect.c:63:44: Warnung: Implizite Deklaration der Funktion »lmbench_usage« [-Wimplicit-function-declaration]
63 | if (parallel <= 0) lmbench_usage(ac, av, usage);
| ^~~~~~~~~~~~~
lat_ops.c: In Funktion »main«:
lat_ops.c:393:44: Warnung: Implizite Deklaration der Funktion »lmbench_usage« [-Wimplicit-function-declaration]
393 | if (parallel <= 0) lmbench_usage(ac, av, usage);
| ^~~~~~~~~~~~~
line.c: In Funktion »main«:
line.c:54:25: Warnung: Implizite Deklaration der Funktion »lmbench_usage« [-Wimplicit-function-declaration]
54 | lmbench_usage(ac, av, usage);
| ^~~~~~~~~~~~~
tlb.c: In Funktion »main«:
tlb.c:69:25: Warnung: Implizite Deklaration der Funktion »lmbench_usage« [-Wimplicit-function-declaration]
69 | lmbench_usage(ac, av, usage);
| ^~~~~~~~~~~~~
par_mem.c: In Funktion »main«:
par_mem.c:64:25: Warnung: Implizite Deklaration der Funktion »lmbench_usage« [-Wimplicit-function-declaration]
64 | lmbench_usage(ac, av, usage);
| ^~~~~~~~~~~~~
par_ops.c: In Funktion »main«:
par_ops.c:413:25: Warnung: Implizite Deklaration der Funktion »lmbench_usage« [-Wimplicit-function-declaration]
413 | lmbench_usage(ac, av, usage);
| ^~~~~~~~~~~~~
stream.c: In Funktion »main«:
stream.c:70:33: Warnung: Implizite Deklaration der Funktion »lmbench_usage« [-Wimplicit-function-declaration]
70 | lmbench_usage(ac, av, usage);
| ^~~~~~~~~~~~~
cache.c: In Funktion »main«:
cache.c:123:25: Warnung: Implizite Deklaration der Funktion »lmbench_usage« [-Wimplicit-function-declaration]
123 | lmbench_usage(ac, av, usage);
| ^~~~~~~~~~~~~
lat_dram_page.c: In Funktion »main«:
lat_dram_page.c:66:25: Warnung: Implizite Deklaration der Funktion »lmbench_usage« [-Wimplicit-function-declaration]
66 | lmbench_usage(ac, av, usage);
| ^~~~~~~~~~~~~
lat_pmake.c: In Funktion »main«:
lat_pmake.c:47:44: Warnung: Implizite Deklaration der Funktion »lmbench_usage« [-Wimplicit-function-declaration]
47 | if (parallel <= 0) lmbench_usage(ac, av, usage);
| ^~~~~~~~~~~~~
lat_rand.c: In Funktion »main«:
lat_rand.c:39:44: Warnung: Implizite Deklaration der Funktion »lmbench_usage« [-Wimplicit-function-declaration]
39 | if (parallel <= 0) lmbench_usage(ac, av, usage);
| ^~~~~~~~~~~~~
lat_usleep.c: In Funktion »main«:
lat_usleep.c:204:17: Warnung: Implizite Deklaration der Funktion »lmbench_usage« [-Wimplicit-function-declaration]
204 | lmbench_usage(ac, av, usage);
| ^~~~~~~~~~~~~
lat_cmd.c: In Funktion »main«:
lat_cmd.c:40:44: Warnung: Implizite Deklaration der Funktion »lmbench_usage« [-Wimplicit-function-declaration]
40 | if (parallel <= 0) lmbench_usage(ac, av, usage);
| ^~~~~~~~~~~~~
[snip]`

i have the same issue with submit. No problem with lmbench.

The fix by jforbes for an anonymous upload works however an authenticated upload still fails.

Perhaps the URL's in fedora_submit.py need updating ?

It does need significant rewriting. Currently you can log in to the website and upload there, but the fedora_submit.py script needs to be rewritten to work with modern auth. Unfortunately the web back end rewrite just went live last week, and I am working limited hours for a bit recovering from a surgery. It is certainly on my to-do list.

Thank you jforbes, best wishes for your recovery.

It does need significant rewriting. Currently you can log in to the website and upload there, but the fedora_submit.py script needs to be rewritten to work with modern auth. Unfortunately the web back end rewrite just went live last week, and I am working limited hours for a bit recovering from a surgery. It is certainly on my to-do list.

Thanks for your quick reply @jforbes !
I wish you a speedy recovery!

gilbert-fernandes commented

Hi jForbes. Hope you recover well and you're safe.
I like to run tests after each kernel update so I ended up here after getting that openid login error.

I'm gonna try to remove the login/password and try the anonymous upload of result.

also got the same lmbench regression but in Swedish while testing Kernel 6.9.2 on Fedora 40 today: https://pastebin.com/b0kSwrEd

hope @jforbes have a good recovery and that this can be fixed

So I looked into fixing this, but I think I hit a bit of a wall. I tried to use https://github.com/puiterwijk/python-openidc-client , the client library we tend to use for this kind of thing (puiterwijk wrote it long ago). This is the code I wound up with:

from __future__ import print_function
from openidc_client import OpenIDCClient
from openidc_client.requestsauth import OpenIDCClientAuther
import requests
import sys
log = sys.argv[1:]
if len(log) != 1:
    print('fedora_submit.py <logfile>')
    sys.exit(2)
log = log[0]
client = OpenIDCClient(
    app_identifier="kerneltest",
    id_provider=f"https://id.fedoraproject.org/openidc/",
    id_provider_mapping={"Token": "Token", "Authorization": "Authorization"},
    client_id="kerneltest",
    client_secret="notsecret",
    useragent="kerneltest",
)
req = requests.post(
    "https://kerneltest.fedoraproject.org/upload/anonymous",
    auth=OpenIDCClientAuther(
        client,
        [
            "openid",
            "email",
            "profile",
            "https://id.fedoraproject.org/scope/groups",
            "https://id.fedoraproject.org/scope/agreements",
            "https://id.fedoraproject.org/scope/fas-attributes",
        ]
    ),
    files={ 'test_result': ('logfile', open(log, 'rb'), 'text/x-log'),},
)
print(req.message)

However, when you go to the authorization URL that you get when running that, you get a 400 error "400 - Bad Request - Invalid redirect_uri".

I think this is because the kerneltest webapp config in ansible infra declares valid redirect URIs - https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/kerneltest/templates/client_secrets.json#_15 - and the localhost redirect_uri that python-openidc-client wants to use is not in the list. What python-openidc-client is trying to do is to 'capture' the authorization code in a local web server it runs; see here for the code.

I don't think it's possible to resolve this without maybe adding something to the kerneltest web app config to allow python-openidc-client's magic to work, or alternatively trying to use an entirely different openidc client (but the three other python ones I found are all abandoned)...

hmm, so I got a bit further...

We actually have several special clients configured in Ipsilon which allow the kinds of redirect URIs python-openidc-client uses, e.g. a client called wikitcms intended for python-wikitcms to use, a client called mbs-authorizer intended for the module build service to use, and so on.

I guess we need to add a new one of these (kerneltest-authorizer?) for this purpose. However, I don't think there's actually anything that prevents you using any of those clients to log in to any Ipsilon-backed service, at least as far as I can tell, so I tried tweaking the code to use the wikitcms client instead:

client = OpenIDCClient(
    app_identifier="kerneltest",
    id_provider=f"https://id.fedoraproject.org/openidc/",
    id_provider_mapping={"Token": "Token", "Authorization": "Authorization"},
    client_id="wikitcms",
    client_secret="notsecret",
    useragent="wikitcms",
)

and...well, I think it works? I actually get a 400 when I try and upload a log with that, but it doesn't appear to be an auth issue - instead, it doesn't like my log file: {"error": "Invalid input file"}. I don't know why not, but I guess that's a separate problem.

Still, it makes things look weird (the auth flow says it's wikitcms that wants to log in), so we should create the new custom client, I guess. This was just a PoC.

Filed https://pagure.io/fedora-infrastructure/issue/11952 . If I'm right, once that's done, we can fix the client easily enough, I'll keep the commit ready here to submit as a PR.
Edited by adamwill

I can't figure out why I'm getting the "Invalid input file" error, though :/ Need better logging on the server end to figure it out, really. The details of what actually went wrong when parsing the file get swallowed and not returned to the submitter or logged anywhere, AFAICS.

OK, figured that error out and fixed it. #50 should fix this.
gilbert-fernandes commented

Checked the new version of the repository.
Turned back on the submit=authenticated + username + password
Started the run-test and I did not see any error related to the upload part.

The log seems to show one of the items failed. It is probably because I am using secure boot and if the kernel module is not signed, its load will fail. Here is the log :

gf@aesir:~/kernel-test/logs$ cat kernel-test-1719475255.log.txt
Date: Thu Jun 27 10:00:55 AM CEST 2024
Test set: default
Kernel: 6.9.5-200.fc40.x86_64
Release: Fedora release 40 (Forty)
Result: FAIL
Failed Tests: ./default/cachedrop
Warned Tests: None
============================================================
Starting test ./default/cachedrop
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 0.111251 s, 943 MB/s
TestError: Can't free dentries and inodes
1558852 1558176
Could not run tests
Starting test ./default/insert_leap_second
Setting time to speed up testing
Running for 1 iterations. Press ctrl-c to stop

Setting time to Fri Jun 28 01:59:50 2024
Scheduling leap second for Fri Jun 28 02:00:00 2024
Fri Jun 28 01:59:57 2024 + 567 us (37) TIME_INS
Fri Jun 28 01:59:57 2024 + 500741 us (37) TIME_INS
Fri Jun 28 01:59:58 2024 + 963 us (37) TIME_INS
Fri Jun 28 01:59:58 2024 + 501542 us (37) TIME_INS
Fri Jun 28 01:59:59 2024 + 2138 us (37) TIME_INS
Fri Jun 28 01:59:59 2024 + 502572 us (37) TIME_INS
Fri Jun 28 01:59:59 2024 + 2715 us (38) TIME_OOP
Fri Jun 28 01:59:59 2024 + 502964 us (38) TIME_OOP
Fri Jun 28 02:00:00 2024 + 3112 us (38) TIME_WAIT
Fri Jun 28 02:00:00 2024 + 503303 us (38) TIME_WAIT
Fri Jun 28 02:00:01 2024 + 3876 us (38) TIME_WAIT
Fri Jun 28 02:00:01 2024 + 504635 us (38) TIME_WAIT
Fri Jun 28 02:00:02 2024 + 4857 us (38) TIME_WAIT
Leap complete

Starting test ./default/libhugetlbfs
Starting test ./default/memfd
memfd: CREATE
memfd: BASIC
memfd: SEAL-WRITE
memfd: SEAL-SHRINK
memfd: SEAL-GROW
memfd: SEAL-RESIZE
memfd: SHARE-DUP
memfd: SHARE-MMAP
memfd: SHARE-OPEN
memfd: SHARE-FORK
memfd: SHARE-DUP (shared file-table)
memfd: SHARE-MMAP (shared file-table)
memfd: SHARE-OPEN (shared file-table)
memfd: SHARE-FORK (shared file-table)
memfd: DONE
Starting test ./default/modsign
Successfully loaded signed module
insmod: ERROR: could not insert module ./minix.ko: Key was rejected by service
Unsigned module load failed in enforcing mode
Starting test ./default/mq-memory-corruption
Test PASSED
Test PASSED
Test PASSED
Test PASSED
Test PASSED
Test PASSED
Test PASSED
Test PASSED
Test PASSED
Test PASSED
Starting test ./default/paxtest
Starting test ./default/posix_timers
Testing posix timers. False negative may happen on CPU execution
based timers if other threads run on the CPU...
Check itimer virtual... [OK]
Check itimer prof... [OK]
Check itimer real... [OK]
Check timer_create() per thread... [OK]
Check timer_create() per process... [OK]
Starting test ./default/selinux-dac-controls

Starting test ./default/stack-randomness
Starting test ./default/sysfs-perms
Starting test ./default/timer-overhead

Glad to hear it worked, thanks.

Turned back on the submit=authenticated + username + password

You don't need username or password in the config file any more, and it'd be safer to remove them. The script no longer uses them, as it is no longer allowed to authenticate directly; the new way is more sophisticated and secure. All you need is submit=authenticated and it will ask you to log in through a web browser the first time you submit, then store a token code it can use to authenticate for future submissions. When the token expires it will ask you to log in through the web browser again.
gilbert-fernandes commented

Thanks. I have removed the login and password and only kept the authenticated part.

As far as I can tell, submits are working fine with the changes in #51 by @adamwill - thank you very much!

Metadata Update from @tharadash:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)
Metadata
None
None
None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.