Large language models for patch review
We're bad at marketingThere have been many discussions in the free-software community about the role of large language models (LLMs) in software development. For the most part, though, those conversations have focused on whether projects should be accepting code output by those models, and under what conditions. But there are other ways in which these systems might participate in the development process. Chris Mason recently started a discussion on the Kernel Summit discussion list about how these models can be used to review patches, rather than create them.We can admit it, marketing is not our strong suit. Our strength is writing the kind of articles that developers, administrators, and free-software supporters depend on to know what is going on in the Linux world. Please subscribe today to help us keep doing that, and so we don’t have to get good at marketing.
Mason's focus was on how LLMs might reduce the load on kernel maintainers
by catching errors before they hit the mailing lists, and by helping
contributors increase the quality of their submissions. To that end, he
has put together a set
of prompts that will produce reviews in a format that maintainers are
used to: "The reviews are meant to look like emails on lkml, and even
when wildly wrong they definitely succeed there
". He included a long
list of sample reviews, some of which hit the mark and others of which did
not.
The prompts are interesting in their own right; they can be seen as constituting the sort of comprehensive patch-review documentation that nobody ever quite got around to writing for humans to use. Perhaps that reflects a higher level of confidence that the LLM will actually read all of this material. These prompts add up to thousands of lines of material, starting with core guidance like:
Struct changes → verify all users use the new struct correctlyPublic API changes → verify documentation updates [...]
Tone Requirements:
- Conversational: Target kernel experts, not beginners
- Factual: No drama, just technical observations
- Questions: Frame as questions about the code, not accusations
Most of the prompts consist of guidance specific to subsystems like locking
("You're not smart enough to understand smp_mb(), smp_rmb(), or
smp_wmb() bugs yet
") and networking
("Socket can outlive its file descriptor
"). All told, it resembles
the sort of rule collection one saw in the expert systems that were
going to take over the world in the 1980s. As noted in the
README file, "the false positive rate is pretty high right now, at
~50%
", so there is still some room for improvement.
In the ensuing discussion, nobody seemed to think that using LLMs in this
way was a bad idea. Sasha Levin called it "a really great
subject to discuss
", and said that, in the previous discussions on LLM use by kernel
developers, the concerns that were raised about LLMs drowned out out any
attempt to find the places where they could be useful. Paul McKenney remarked
that using this technology to review code written by others "seems much
safer than using it to generate actual code
". Krzysztof Kozlowski noted
that Qualcomm has created a similar system and made it available.
There were some concerns raised about the proprietary nature of these systems; Konstantin Ryabitsev was just one of a few who drew parallels with the BitKeeper experience that (briefly) brought kernel development to a halt just over 20 years ago. Laurent Pinchart stated clearly that there are limits to how much proprietary tools can be used or required:
Forcing contributors to pay for access to proprietary tools is not acceptable. Forcing contributors to even run proprietary tools is not acceptable.
He also expressed concerns that the companies behind LLMs would make them available to developers for free to encourage adoption — until the community is well locked in, at which point access could quickly become expensive. Mason, though, was unworried about lock-in, saying that the prompts are sufficiently generic to be adaptable to any system. James Bottomley suggested that LLMs would not be proprietary forever, but Pinchart argued against relying on proprietary software in the hope that there will eventually be free alternatives.
There was some disagreement over who an LLM-based review tool should be created for. Mason's target was maintainers, but Andrew Lunn argued that the plan should be for developers to run these tools themselves before posting code for review. That, he said, would further reduce the workload on maintainers, who would only need to run LLM review to verify the the submitter had already done so.
Pinchart, along with others, pointed
out that getting developers to use the tools (such as
checkpatch.pl) that exist now is difficult; he wondered how
submitters could be encouraged to run any new tools. Tim Bird suggested
annotating patches with a list of the tools that have been run on them so
that maintainers could see that history. Bottomley, instead, said
that these tools should be run automatically on patches sent to the mailing
lists, much like the checks that the 0day
robot runs on posted patches now. Bird, though, said
that running the tools should be expected of submitters. "It then
becomes a cost for the contributor instead of the upstream community, which
is going to scale better.
"
Mason was clear in his belief that LLM-generated reviews should happen in public as part of the submission process:
I think it's also important to remember that AI is sometimes wildly wrong. Having the reviews show up on a list where more established developers can call bullshit definitely helps protect against wasting people's time.
Linus Torvalds, in his
one contribution to the discussion, agreed. He was about the only one
to express concerns about the technology, saying "I think we've all seen
the garbage end of AI, and how it can generate more work rather than
less
". Mason agreed
that Torvalds's concerns were relevant, based on his own experience:
My first prompts told AI to assume the patches had bugs, and it would consistently just invent bugs. That's not the end of the world, but the explanations are always convincing enough that you'd waste a bunch of time tracking it down.
Torvalds mentioned the scraper problem as
well. His concerns notwithstanding, he believes that this technology will
prove helpful, but he feels that its initial adoption has to be aimed at
making life easier for maintainers. "So I think that only once any AI
tools are actively helping maintainers in a day-to-day workflow should
people even *look* at having non-maintainers use them
".
The conversation wound down shortly after that. One clear conclusion,
though, is that these tools seem destined to play an increasing role in the
kernel-development process. At some point, we will likely start seeing
machine-generated reviews showing up on the mailing lists; then, perhaps,
the real value of LLM-based patch-review tools will start to become clear.
It will be interesting to see how the inevitable related discussion at the
2025
Maintainer Summit in December plays out.
| Index entries for this article | |
|---|---|
| Kernel | Development tools/Large language models |