Netdev 2018 day 2 [LWN.net]

Netdev 2018 day 2

Posted Aug 29, 2018 12:34 UTC (Wed) by zdzichu (subscriber, #17118)
In reply to: Netdev 2018 day 2 by Cyberax
Parent article: Netdev 2018 day 2

Because default_qdisc setting applies to all network interfaces.

to post comments

Netdev 2018 day 2

Posted Aug 29, 2018 17:31 UTC (Wed) by mtaht (guest, #11087) [Link] (10 responses)

I don't know the right answer for CAN. Multiple qdiscs specify their actual base qdisc, or noqueue, or mq, and so on. I like to think that FQ_codel + ecn would help multiplexing a variety of useful signals over can deterministically.

But the idea of can bus folk thinking queues are infinite and non-loss ever scares the bejesus out of me. Plenty of other reasons (like running out of memory, or corruption due to a wild memory write, or a cosmic ray event) exist for a packet to be lost.

I won't sleep better if you can point me at what's going on over in that world, but please do so....

Netdev 2018 day 2

Posted Aug 29, 2018 18:21 UTC (Wed) by mtaht (guest, #11087) [Link] (7 responses)

meant to say "device driver".

my general hope is that the CAN folk takeaway is not that fq_codel is dangerous, but short buffer and packet loss is. Merely switching back to a fifo masks the problem. So I'd hope - after getting burned by fq_codel (what speed does can run at?) they'd apply rigor to their protocol designs to make them tolerant of loss in general. (and figure out how much steady-state buffering is needed) An example would be an accelerator sensor should always send absolute positions not deltas, at a high sample rate and the overall capacity of the bus left with sufficient headroom with all the other devices on it to never get even close to capacity.

Another takeaway from the systemd bug I cite above is that Linux's network behavior is still far from optimal on short paths. TSQ doesn't scale past a dozen flows. We hit a wall on decreasing cwnd in tcp that can possibly be met with better pacing. Or not pulling data out of the socket buffer until it's needed...

On short paths on that test, there was 760 times more buffering than that required to fill the gigE path, even with codel dropping madly. I ran the same test with a target 1ms, got it down to half that, same throughput same rtt (and quite a few RTOs)....

Netdev 2018 day 2

Posted Aug 29, 2018 18:44 UTC (Wed) by mtaht (guest, #11087) [Link] (2 responses)

so I went looking...

this can device is noqueue. I don't know what that means, device has a lot of internal buffering?

http://docs.automotivelinux.org/docs/apis_services/en/dev...

this on the other hand might be a problem

https://github.com/lxc/lxc/issues/2474

but this suggests it isn't though a correct fq_codel target setting would be well above 50ms in this case

https://github.com/GBert/openwrt-misc/tree/master/mcp2515...

Netdev 2018 day 2

Posted Aug 29, 2018 18:45 UTC (Wed) by mtaht (guest, #11087) [Link] (1 responses)

on the other other hand the mtu on a can bus is 16???

Netdev 2018 day 2

Posted Aug 29, 2018 19:39 UTC (Wed) by mtaht (guest, #11087) [Link]

I knew nothing about CAN before 3 hours ago. In particular I retract what I was saying about ecn (that was an IP specific bus). It looks like the FQ part in fq_codel, would "work" if there was a dissector for it. As the MTU is tiny the codelly bit would "work" also at the 5ms default - but (after reading up on it) both FQ and codel seem very dangerous to deploy in a CAN environment as anything other than a test of the robustness of the system. FQ (DRR) could deliver messages de-interleaved to various devices, and the HOL blocking in the media access force drops in codel.

Thanks for pointing this out. For the first time in years I'm led to conclude strict fifo is better in a given system. Not that I don't see CAN has problems, mind you, but they aren't fq_codels domain to solve.

I'll have to write this up somewhere, maybe discuss on the can list.

Netdev 2018 day 2

Posted Aug 29, 2018 20:03 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link] (3 responses)

CAN devices typically are not designed to tolerate dropped packets. It's also not easy to do complicated stuff when your whole payload is 8 bytes.

Typically people just over-provision CAN networks so there's almost no contention.

Netdev 2018 day 2

Posted Aug 29, 2018 20:29 UTC (Wed) by mtaht (guest, #11087) [Link] (2 responses)

I do my best work when frightened! I now think the best option for can devices is noqueue, and I can imagine quite a few that relied on pfifo_fast as the default suddenly breaking with any other qdisc.

Netdev 2018 day 2

Posted Aug 29, 2018 20:57 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link] (1 responses)

Noqueue might be a bit too aggressive, you don't want to drop a packet if the bus happens to be busy at the moment of transmission, pfifo_fast sounds about right.

Netdev 2018 day 2

Posted Aug 29, 2018 21:28 UTC (Wed) by mtaht (guest, #11087) [Link]

Well... my assumption would be the device driver is buffering up one or two packets... based on that whopping 3 hours of experience and googling. :)

default pfifo_fast would be buffering up 1000, which strikes me as a bit much. I did see (I think) something that set txqueuelen to 10.

I'm (sigh) going to have to go look at every "packet" interface in linux now...

Netdev 2018 day 2

Posted Aug 30, 2018 7:02 UTC (Thu) by zdzichu (subscriber, #17118) [Link] (1 responses)

It looks like you done enough research already, but for the matter of completeness:

The bug report about the default qdisc vs CAN: https://github.com/systemd/systemd/issues/9194

It has originated from this thread: https://marc.info/?l=linux-can&m=152785826628439&w=2

Basicaly, it is impossible to have a good default, fighting bufferbloat and fitting all Linux usages at the same time.

Netdev 2018 day 2

Posted Aug 30, 2018 15:08 UTC (Thu) by mtaht (guest, #11087) [Link]

Nothing is impossible! But sometimes takes a long time!

https://bugs.openwrt.org/index.php?do=details&task_id...