GitLab CI for the kernel

Ready to give LWN a try?

With a subscription to LWN, you can stay current with what is happening in the Linux and free-software community and take advantage of subscriber-only site features. We are pleased to offer you a free trial subscription, no credit card required, so that you can see for yourself. Please, join us!

Working on the Linux kernel has always been unlike working on many other software projects. One particularly noticeable difference is the decentralized nature of the kernel's testing infrastructure. Projects such as syzkaller, KernelCI, or the kernel self tests test the kernel in different ways. On February 28, Helen Koike posted a patch set that would add continuous integration (CI) scripts for the whole kernel. The response was generally positive, but several people suggested changes.

Koike's patch set adds a new top-level ci directory that contains YAML configuration files for GitLab's continuous-integration feature, as well as shell scripts to tie those to the existing kernel tests. It reuses some of her existing work from the kernel's graphics subsystem tests, which also use GitLab as a Continuous Integration platform. The patch set currently includes code to run checkpatch and Smatch against proposed patches, and attempt to build the kernel on a few different architectures, but Koike plans to expand the coverage if this initial work is accepted. The patch set also includes a top-level .gitlab-ci file that instructs GitLab to run the tests by default.

The patch set

Several people responded positively to the proposal. Tim Bird thought that the change was useful, saying: "I don't currently use gitlab, but I might install it just to test this out." Maxime Ripard also thought the work could be useful, but questioned how well it could support different use cases. Ripard pointed out that different subsystems and maintainers probably have different uses for continuous integration. "I don't see how the current architecture could [accommodate] for that."

GitLab can be configured to look elsewhere inside the repository for the configuration file, Nikolai Kondrashov pointed out. He suggested that having a top-level configuration might not be best. "This way all the different subtrees can have completely different setup, but some could still use Helen's work and employ the 'scenarios' she implemented."

Linus Torvalds agreed that it would be better not to include the top-level configuration: "I'm not at all interested in having something that people will then either fight about, or - more likely - ignore, at the top level because there isn't some global agreement about what the rules are." Torvalds suggested that it might be better to have the CI project be separate from the kernel.

However, there are benefits to keeping the CI scripts in the same repository as the code itself, according to Kondrashov. This allows developers to ensure that the tests and the code remain in sync. He suggested that "we reframe this contribution as a sort of template, or a reference for people to start their setup with". He also raised the possibility of having the new code live somewhere other than the ci directory.

The suggestion that the code could be a library of pre-made pieces for maintainers to pick and choose from was "a lot more palatable" to Torvalds. He suggested that the tools/ci folder would be the logical place since it is "kind of alongside our tools/testing subdirectory." Koike was fine with that approach, saying that it would make her work as the maintainer of the kernel's graphics subsystem (DRM) tests easier and still support extending test coverage to other subsystems. Her colleague Nicolas Dufresne later recapped the discussion, stating that the top-level configuration file would be removed.

Guenter Roeck thought that there were at least some basic requirements which all kernel developers could agree upon.

Sure, argue against checkpatch as much as you like, but the code should at least _build_, and it should not be necessary for random people to report build failures to the submitters.

Randy Dunlap and Geert Uytterhoeven agreed. Ripard pointed out that running CI like this takes funding, and that the supporters of existing testing efforts like the DRM-CI infrastructure might not want to pay for tests of unrelated parts of the kernel.

I don't really expect, say, the clock framework, to validate that all DRM kunit tests pass for each commit they merge, even though one of them could totally break some of the DRM tests.

A follow-up message clarified that Ripard is in favor of automated tests, but doesn't think it's reasonable to expect people "to pay for builders running things they fundamentally don't care about."

KernelCI

Guillaume Tucker had a different question: "Where does this fit on the KernelCI roadmap?" KernelCI is a project that works to provide a distributed test-automation system for kernel development. It was started in 2012 by Arnd Bergmann, Olof Johansson, and Kevin Hilman to detect build failures for Arm builds of the kernel. In 2019, it was adopted by the Linux Foundation with plans to expand the coverage and infrastructure to cover the entire kernel community's needs.

Kondrashov replied that the work was "an important part of KernelCI the project," although not currently part of the KernelCI service. The project does have existing build infrastructure, which Kondrashov would like to be able to reuse by sending GitLab jobs to the same pipeline.

The existing KernelCI work and the new GitLab-based tooling were meant to serve different purposes, Dufresne suggested. The KernelCI testing is largely "integration" testing that incorporates multiple changes from across the kernel at once. Koike's proposed GitLab testing would run on every push to a repository, effectively testing changes in isolation. That will "help catch submission issues earlier, and reduce [the] kernelCI regression rate", according to Dufresne.

Tucker was not satisfied with that response, saying that the new code was not restricted to that use case, and that it provides "a platform able to cope with the diversity of workflows across the kernel subsystems". Tucker's message also pointed out that the code contained a lot of mentions of KernelCI in variable names and documentation, which serves to confuse whether this work is meant to be part of the project or not. If it is part of the project, Tucker questioned why it was diverging from the project's previous plans to provide a comprehensive platform with new features.

Gustavo Padovan replied that what Tucker was missing was that the community is not really working with the KernelCI project. "If one asks people around, the lack of community engagement with KernelCI is evident." The new work is the result of a renewed effort to provide high quality tests following a change in the project's leadership, according to Padovan. He hopes that the increased involvement will translate not only into better tests, but also the feedback and funding required to bring KernelCI to the level that previous plans had envisioned.

Tucker emphasized that he was in favor of the new CI work, and that he had just wanted clarity on how it was related to the KernelCI project. Despite the overall positive reaction, Koike has not yet sent a new version of the patch set. It seems clear that many kernel developers would like to see more automatic test coverage, however. The BPF developers, for example, have a talk about continuous integration scheduled for the upcoming Linux Filesystem Memory Management and BPF conference. So chances are good that an updated version of this work will make its way in sooner or later.