A kernel self-testing update

Benefits for LWN subscribers

The primary benefit from subscribing to LWN is helping to keep us publishing, but, beyond that, subscribers get immediate access to all site content and access to a number of extra site features. Please sign up today!

The kernel has contained a set of tests for a long time, at least since 2005, Khan said. It was only three years ago, though, that a formal effort to create a self-test "subsystem" was started. The idea was to add a lot more tests and create a regression-test suite for kernel developers. Various organizations, such as Linaro, run it; it is also part of the KernelCI.org and the 0-day test service.

Running the tests is a simple matter:

    make -silent kselftest

The -silent option reduces the clutter in the output. Better results will be had if the tests are run as root, but one should beware of the more disruptive tests, which can force a reboot.

It is possible to add a TARGETS= option to restrict the tests to one or more subsystems; in this case, it only runs the non-disruptive tests. Recent work has enabled the O= option to use a separate build directory, but it still uses the .config file from the source directory. This detail is deemed to be worth fixing, but it is of relatively low importance because the tests don't make much use of the kernel configuration. Ted Ts'o observed, though, that the build architecture is defined in .config, as are options that configure various subsystems out of the kernel entirely.

Another option for testing a specific subsystem is to run:

    make -C tools/testing/<subsystem> run_tests

The output from this command will be a summary of the results; the details can be found in a file named after the specific test in /tmp. Ben Hutchings requested that fixed names in /tmp not be used (presumably to avoid creating yet another vector for symbolic-link attacks), so this behavior may eventually change. There was some discussion on the details of the formatting of the results that didn't lead to any significant conclusions.

Khan concluded by saying that, starting with the 4.12 kernel, test results are reported in the TAP13 format. This is a simple text format that makes it easy to spot differences between test runs; it also supports machine parsing of the results.

Arnd Bergmann said that he has been running the latest self tests on (older) stable kernels. That work has resulted in patches fixing problems with the tests; most maintainers like those patches, but the networking subsystem, in particular, will not accept patches aimed at making current tests run on older kernels. Bergmann said that there is a strong desire to make adding new tests as easy as possible; requiring that those tests behave properly on older kernels raises the barrier.

It is also not clear what should happen with tests for features that are not present on older kernels; there are differences of opinion over whether they should report failure or that the test has been skipped. Ts'o suggested that "skip" is the proper result, and that it would be good to know why some developers object to it. Perhaps, he said, the real problem is that some features are failing to present themselves properly. In that case, a "skip" result could mask a deeper problem. Mathieu Desnoyers suggested that the default should be to signal failure for skipped tests, with an option to simply pass over them instead.

At the end of the session, Matthew Wilcox raised a question regarding the tests for the radix-tree implementation. Perhaps, he said, those tests should be run at every kernel build, with the build as a whole failing if the tests do not pass. If the radix-tree tests don't pass, he said, booting the resulting kernel is not advisable. This idea ran into opposition, though. Beyond slowing down the build, their effectiveness is reduced by the fact that the build environment is often different from the run environment. ARM kernels tend to be built on x86 systems, for example. The discussion, overall, suggested that running self-tests during ordinary kernel builds will be a hard sell.

[Your editor would like to thank the Linux Foundation, LWN's travel sponsor, for supporting his travel to the event.]