Temporary problem [LWN.net]

Temporary problem

Posted Jun 4, 2026 10:39 UTC (Thu) by malmedal (subscriber, #56172)
In reply to: Temporary problem by paulj
Parent article: BPF in the agentic era

In principle we know how to reduce the energy used per computation. I believe we can relatively easily increase efficiency by a factor of at least ten probably even a hundred.

Problem is that most ways of increasing efficiency also decreases max clock speed. The brain runs at around 20 to 200Hz, we should be able to eventually reach similar efficiencies if we ran our silicon at that speed, obviously this means we'd need at lot of silicon.

In the near term we are very constrained on chip supply so I expect the focus is on better cooling so we can run the chips hotter and even less efficient.

to post comments

Temporary problem

Posted Jun 4, 2026 11:55 UTC (Thu) by paulj (subscriber, #341) [Link] (1 responses)

Which ways are you thinking of?

If we look at the underlying architecture of biological neural nets, and work on the assumption there is something to learn there on energy efficient NNs, then that suggests we need to move to asynchronous architectures of very large numbers of simple units that combine both communication with 'nearby' units and integrating those signals (memory and compute). (Where current architecture is using large numbers of simple computational units, with hierarchies of /separate/ memory, and then emulating the communication side with compute over that separate memory - which can take multiple steps).

So SNNs possibly, running on some kind of much, much, much more massive SpiNNaker-type tiled system. But, SpiNNaker hasn't yet provided results that have been... revolutionary (let's say). ?

Or you're thinking of something else?

Temporary problem

Posted Jun 4, 2026 16:28 UTC (Thu) by malmedal (subscriber, #56172) [Link]

That yes, but also things like process changes.

Since the availability of chips currently is the bottleneck, people run them as hard as possible, if the production increases to the point where the limit is the available power there are many things that can be done to get more compute per joule of energy. Currently it appears to be mostly academic research in older processes, e.g:

https://www.researchgate.net/publication/358974979_An_Ult...

but I believe if it becomes commercially interesting then industry can quickly get efficiency gains of 10 to 100 times. Same goes for network interconnect.

There are also pie in the sky things like reversible computing, https://www.sciencenews.org/article/computer-chip-reuses-...
Theoretically feasible, wouldn't hold my breath while waiting though.

Temporary problem

Posted Jun 4, 2026 13:36 UTC (Thu) by dskoll (subscriber, #1630) [Link] (6 responses)

A big difference between the brain and silicon chips is that the brain can make connections in three dimensions, while silicon chips are effectively 2D. Even with stacked chips, the connectivity in the third dimension is much less than in the other two. So the brain can have a much more dense and complex set of interconnections than a silicon chip and I think it's one reason it can operate so well with less energy and a lower "clock speed".

Temporary problem

Posted Jun 4, 2026 13:47 UTC (Thu) by daroc (editor, #160859) [Link] (5 responses)

This is a reason that chip fabs are working on ways to make chips with 3D stacked layers of transistors. They haven't cracked it yet, but it's one of those research areas that someone is always poking at because a discovery would be revolutionary.

Temporary problem

Posted Jun 4, 2026 17:03 UTC (Thu) by malmedal (subscriber, #56172) [Link]

I believe the issue is mostly heat, and a little bit of interconnect. For instance 3D Flash is currently around 300 layers, but to try that in normal logic would make the inner layers melt.

Hmm, or it might posibly work if you ran such a chip on 200Hz like the human brain does.

Temporary problem

Posted Jun 5, 2026 9:32 UTC (Fri) by paulj (subscriber, #341) [Link] (3 responses)

Stacked dies are already common practice for high-speed DRAM on GPUs and high-speed cache on some CPUs. A.k.a. HBM.

This still doesn't solve the inefficiency of having compute logic in one place, and memory in another place, and having to constantly transfer information back and forth between the two to a) carry out the /emulation/ of the network b) so as to enable the computation of this emulated network. In the brain, compute and storage are part of the same unit and the network is intrinsic to the physical structure of the 'computer'. In silicon NNs, we are spending power to /emulate/ a network, on top of a physical structure that doesn't resemble the desired network in any meaningful way.

Temporary problem

Posted Jun 5, 2026 11:32 UTC (Fri) by malmedal (subscriber, #56172) [Link] (2 responses)

Several companies for instance Cerebras.ai does what you suggest.

It is clear that this is faster, it is not clear if it is more power-efficient or not.

Simplifying a lot but basically to produce a new token you have a big matrix and multiply it with a 1d vector. If you make the vector a two wide matrix you can produce a token each for two unrelated sessions for just a tiny bit extra work. You can keep widening the matrix quite a bit before it becomes too slow.

Several providers give you a fast mode e.g. 3 times faster for three times the price which I assume means they give you fewer neighbors in the matrix multiply.

Eventually I expect custom chips will replace the current GPUs, but right now the basic architecture is still in flux, previously RNNs were the hot stuff for AI, currently it's mostly transformers but there is promising research into replacing those with diffusers. So for the time being we want something flexible.

Temporary problem

Posted Jun 5, 2026 12:52 UTC (Fri) by paulj (subscriber, #341) [Link] (1 responses)

Yes, I mentioned Cerberas in one of my comments antecedent to these. Cerebrus puts SRAM on the tile, but this is still separate from the compute logic. Obviously so from the die shots. Have a look at DOI:10.1109/MM.2023.3256384. It's going a bit in that direction, but it's still emulating the NN network architecture with a traditional Von Neumann machine, with separate memory and compute and a load-compute-store architecture (however parallel, and emulating the network by modelling it as a matrix).

Temporary problem

Posted Jun 5, 2026 14:30 UTC (Fri) by malmedal (subscriber, #56172) [Link]

Yeah. I'm sure full custom will the end-point, but currently we don't know what the optimal architecture is. For instance the brain does have a number of long-range connections, which appears to be important, but while these can be grown dynamically in the human brain we can't yet do that in silicon, so for the time being I believe it's necessary to have something where the connections can be reprogrammed at will.