CXL 2: Pooling, sharing, and I/O-memory resources

I think a lot of people are confused about how CXL memory is going to work. Trying to munge it in memory hotplug is just going to be hopelessly broken. The problem is that the receiving system does not actually own that memory. The memory appliance does. Suppose that you have a CXL memory device with gobs of memory. The compute machines that are attached will request memory and give it back. But what happens if $VENDOR driver pukes over kernel memory and causes a kernel panic. The exciting data that gets written back to the appliance doesn't matter so much as what the device does in response to the fact that the machine that is bleeding out has some of its memory. A sane spec will have a mechanism for the appliance to say, "You are giving up this memory NOW!"

So if you want the kernel to manage this memory there will have to be two separate concepts, the memory the device owns and cannot be taken away, and memory that it does not own and subject to being ripped away. I'm half of the mind that a better model for this is to expose remote CXL devices in /dev/ and then mmap() that like a normal 'file'.