Netdev 2018 day 2 [LWN.net]

Netdev 2018 day 2

Posted Aug 28, 2018 20:35 UTC (Tue) by mtaht (guest, #11087)
Parent article: Netdev 2018 day 2

"The desired qdisc is one which optimizes the completion time, not fairness, because with fairness packets of two transactions are interleaved, while packets of each transaction could be batched and sent in round robin. This optimizes the completion by allowing one transaction to finish in half of the time. "

This is debatable, although I get where he is coming from from a tcp perspective in the datacenter. If your 100Gige transaction is 10 packets in 6 nanoseconds, by ghu, burst up.

10 packets in a row for 13ms (10mbit), not so much.

The other major reason for my continued advocacy of "pure FQ" is for the wide range of traffic that has "no round trip completion" concept except "get me through fast with minimal delay and jitter" - voip, gaming, dns, videoconferencing, in particular.

Also, as multiple people have finally noticed ( https://github.com/systemd/systemd/issues/9725 ) , sch_fq is grossly unfair to anything other than locally sourced tcp traffic, with the large 15k initial quantum, and a quantum of 3028, to avoid del-acks. It should have been called sch_tcp_pacing or something like that. Real packet average sizes are 300 bytes.

Despite me saying that I do support DRR over SFQ and sometimes larger quantums - as it helps bunch up packets *a little bit* (easing the load on the host stack) and can be configured up or down as needed. In my pathetically slow portion of the universe we often run fq_codel with 300 byte quantums, and sch_cake autoconfigures and does gso-splitting by default, leading to much fairer results at a gigE and below.

"No existing qdisc can do this. But, the carousel/timing-wheel scheduler can accomplish this easily."

I like the timer wheel idea a lot regardless, and, with hardware support, seems needed and useful at speeds over 100mbit. If it works in software... oh, wow, we get to toss out 7 years of bufferbloat-fighting with tbf + fq_codel in favor of something new and shiny. I can't wait to see whatever eric comes up with. :)

... but I kind of figure I'll have to dig up some dusty research into virtualclock FQ systems do make it work where I want.

to post comments

Netdev 2018 day 2

Posted Aug 29, 2018 6:24 UTC (Wed) by zdzichu (subscriber, #17118) [Link] (15 responses)

There's also and issue pointing that fq_codel wreaks havoc on CAN networks (the stuff inside automobiles). Apparently dropping packets in such networks can be dangerous, who would expect that?
This is sadly an illustration that we can not have nice defaults in this world. Everything is broken :(

I, for one, still haven't found an answer what would be the best setting for my home server. It serves some webpages out to the internet, and various media inside my home network (bulk transfer), interacts with IoT (lowest latency required), hosts few virtual machines and ALSO is a router to the internet and and wifi access-point. CAKE looks like a thing that will finally behave correctly in such scenario.

(Also an interesting note: although I'm a bit of network guy, you post was very close to technobabble, and required increased mental power to process. I wonder if network issues are complicated by their nature).

Netdev 2018 day 2

Posted Aug 29, 2018 9:16 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link] (12 responses)

Why would you use codel with CAN?!?!?!?!

It's like adding ketchup to cotton candy.

Netdev 2018 day 2

Posted Aug 29, 2018 12:34 UTC (Wed) by zdzichu (subscriber, #17118) [Link] (11 responses)

Because default_qdisc setting applies to all network interfaces.

Netdev 2018 day 2

Posted Aug 29, 2018 17:31 UTC (Wed) by mtaht (guest, #11087) [Link] (10 responses)

I don't know the right answer for CAN. Multiple qdiscs specify their actual base qdisc, or noqueue, or mq, and so on. I like to think that FQ_codel + ecn would help multiplexing a variety of useful signals over can deterministically.

But the idea of can bus folk thinking queues are infinite and non-loss ever scares the bejesus out of me. Plenty of other reasons (like running out of memory, or corruption due to a wild memory write, or a cosmic ray event) exist for a packet to be lost.

I won't sleep better if you can point me at what's going on over in that world, but please do so....

Netdev 2018 day 2

Posted Aug 29, 2018 18:21 UTC (Wed) by mtaht (guest, #11087) [Link] (7 responses)

meant to say "device driver".

my general hope is that the CAN folk takeaway is not that fq_codel is dangerous, but short buffer and packet loss is. Merely switching back to a fifo masks the problem. So I'd hope - after getting burned by fq_codel (what speed does can run at?) they'd apply rigor to their protocol designs to make them tolerant of loss in general. (and figure out how much steady-state buffering is needed) An example would be an accelerator sensor should always send absolute positions not deltas, at a high sample rate and the overall capacity of the bus left with sufficient headroom with all the other devices on it to never get even close to capacity.

Another takeaway from the systemd bug I cite above is that Linux's network behavior is still far from optimal on short paths. TSQ doesn't scale past a dozen flows. We hit a wall on decreasing cwnd in tcp that can possibly be met with better pacing. Or not pulling data out of the socket buffer until it's needed...

On short paths on that test, there was 760 times more buffering than that required to fill the gigE path, even with codel dropping madly. I ran the same test with a target 1ms, got it down to half that, same throughput same rtt (and quite a few RTOs)....

Netdev 2018 day 2

Posted Aug 29, 2018 18:44 UTC (Wed) by mtaht (guest, #11087) [Link] (2 responses)

so I went looking...

this can device is noqueue. I don't know what that means, device has a lot of internal buffering?

http://docs.automotivelinux.org/docs/apis_services/en/dev...

this on the other hand might be a problem

https://github.com/lxc/lxc/issues/2474

but this suggests it isn't though a correct fq_codel target setting would be well above 50ms in this case

https://github.com/GBert/openwrt-misc/tree/master/mcp2515...

Netdev 2018 day 2

Posted Aug 29, 2018 18:45 UTC (Wed) by mtaht (guest, #11087) [Link] (1 responses)

on the other other hand the mtu on a can bus is 16???

Netdev 2018 day 2

Posted Aug 29, 2018 19:39 UTC (Wed) by mtaht (guest, #11087) [Link]

I knew nothing about CAN before 3 hours ago. In particular I retract what I was saying about ecn (that was an IP specific bus). It looks like the FQ part in fq_codel, would "work" if there was a dissector for it. As the MTU is tiny the codelly bit would "work" also at the 5ms default - but (after reading up on it) both FQ and codel seem very dangerous to deploy in a CAN environment as anything other than a test of the robustness of the system. FQ (DRR) could deliver messages de-interleaved to various devices, and the HOL blocking in the media access force drops in codel.

Thanks for pointing this out. For the first time in years I'm led to conclude strict fifo is better in a given system. Not that I don't see CAN has problems, mind you, but they aren't fq_codels domain to solve.

I'll have to write this up somewhere, maybe discuss on the can list.

Netdev 2018 day 2

Posted Aug 29, 2018 20:03 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link] (3 responses)

CAN devices typically are not designed to tolerate dropped packets. It's also not easy to do complicated stuff when your whole payload is 8 bytes.

Typically people just over-provision CAN networks so there's almost no contention.

Netdev 2018 day 2

Posted Aug 29, 2018 20:29 UTC (Wed) by mtaht (guest, #11087) [Link] (2 responses)

I do my best work when frightened! I now think the best option for can devices is noqueue, and I can imagine quite a few that relied on pfifo_fast as the default suddenly breaking with any other qdisc.

Netdev 2018 day 2

Posted Aug 29, 2018 20:57 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link] (1 responses)

Noqueue might be a bit too aggressive, you don't want to drop a packet if the bus happens to be busy at the moment of transmission, pfifo_fast sounds about right.

Netdev 2018 day 2

Posted Aug 29, 2018 21:28 UTC (Wed) by mtaht (guest, #11087) [Link]

Well... my assumption would be the device driver is buffering up one or two packets... based on that whopping 3 hours of experience and googling. :)

default pfifo_fast would be buffering up 1000, which strikes me as a bit much. I did see (I think) something that set txqueuelen to 10.

I'm (sigh) going to have to go look at every "packet" interface in linux now...

Netdev 2018 day 2

Posted Aug 30, 2018 7:02 UTC (Thu) by zdzichu (subscriber, #17118) [Link] (1 responses)

It looks like you done enough research already, but for the matter of completeness:

The bug report about the default qdisc vs CAN: https://github.com/systemd/systemd/issues/9194

It has originated from this thread: https://marc.info/?l=linux-can&m=152785826628439&w=2

Basicaly, it is impossible to have a good default, fighting bufferbloat and fitting all Linux usages at the same time.

Netdev 2018 day 2

Posted Aug 30, 2018 15:08 UTC (Thu) by mtaht (guest, #11087) [Link]

Nothing is impossible! But sometimes takes a long time!

https://bugs.openwrt.org/index.php?do=details&task_id...

Netdev 2018 day 2

Posted Aug 29, 2018 17:21 UTC (Wed) by mtaht (guest, #11087) [Link]

fq_codel may make more noticible packet loss in a can network (cite?), but loss happens in any network system (fifo or fq), and it scares me that more folk are not aware of it, but think buffers are infinite. If your car crashes one turn in 100 with fq_codel 5ms delay, and one in 10000 with a 1000 packet fifo... it's still gonna crash.

One use for ecn is certainly in networks that cannot tolerate loss, I know of one application in a flying spacecraft, (thankfully mostly merely multiplexing sensors across the bus which has a predictable outcome) and I would certainly approve of ecn + *rigorous* congestion awareness in every app in a can (or nmea 20000 network. Certainly I'd support setting an appropriate target delay to supply adaquate buffering.

I'm sorry for the technobabble, I've been at this gig for too long. I did a lot of introductory talks in the beginning....

Netdev 2018 day 2

Posted Aug 29, 2018 17:23 UTC (Wed) by mtaht (guest, #11087) [Link]

It's kind of my hope that the per host fq in your scenario helps.

It's still turtles all the way down.