Temporary problem [LWN.net]

Temporary problem

Posted Jun 4, 2026 13:47 UTC (Thu) by daroc (editor, #160859)
In reply to: Temporary problem by dskoll
Parent article: BPF in the agentic era

This is a reason that chip fabs are working on ways to make chips with 3D stacked layers of transistors. They haven't cracked it yet, but it's one of those research areas that someone is always poking at because a discovery would be revolutionary.

Temporary problem

Posted Jun 5, 2026 9:32 UTC (Fri) by paulj (subscriber, #341) [Link] (3 responses)

Stacked dies are already common practice for high-speed DRAM on GPUs and high-speed cache on some CPUs. A.k.a. HBM.

This still doesn't solve the inefficiency of having compute logic in one place, and memory in another place, and having to constantly transfer information back and forth between the two to a) carry out the /emulation/ of the network b) so as to enable the computation of this emulated network. In the brain, compute and storage are part of the same unit and the network is intrinsic to the physical structure of the 'computer'. In silicon NNs, we are spending power to /emulate/ a network, on top of a physical structure that doesn't resemble the desired network in any meaningful way.

Temporary problem

Posted Jun 5, 2026 11:32 UTC (Fri) by malmedal (subscriber, #56172) [Link] (2 responses)

Several companies for instance Cerebras.ai does what you suggest.

It is clear that this is faster, it is not clear if it is more power-efficient or not.

Simplifying a lot but basically to produce a new token you have a big matrix and multiply it with a 1d vector. If you make the vector a two wide matrix you can produce a token each for two unrelated sessions for just a tiny bit extra work. You can keep widening the matrix quite a bit before it becomes too slow.

Several providers give you a fast mode e.g. 3 times faster for three times the price which I assume means they give you fewer neighbors in the matrix multiply.

Eventually I expect custom chips will replace the current GPUs, but right now the basic architecture is still in flux, previously RNNs were the hot stuff for AI, currently it's mostly transformers but there is promising research into replacing those with diffusers. So for the time being we want something flexible.

Temporary problem

Posted Jun 5, 2026 12:52 UTC (Fri) by paulj (subscriber, #341) [Link] (1 responses)

Yes, I mentioned Cerberas in one of my comments antecedent to these. Cerebrus puts SRAM on the tile, but this is still separate from the compute logic. Obviously so from the die shots. Have a look at DOI:10.1109/MM.2023.3256384. It's going a bit in that direction, but it's still emulating the NN network architecture with a traditional Von Neumann machine, with separate memory and compute and a load-compute-store architecture (however parallel, and emulating the network by modelling it as a matrix).

Temporary problem

Posted Jun 5, 2026 14:30 UTC (Fri) by malmedal (subscriber, #56172) [Link]

Yeah. I'm sure full custom will the end-point, but currently we don't know what the optimal architecture is. For instance the brain does have a number of long-range connections, which appears to be important, but while these can be grown dynamically in the human brain we can't yet do that in silicon, so for the time being I believe it's necessary to have something where the connections can be reprogrammed at will.