The embedded long-term support initiative [LWN.net]

The embedded long-term support initiative

Posted Oct 29, 2011 11:02 UTC (Sat) by cortana (subscriber, #24596)
In reply to: The embedded long-term support initiative by kragilkragil2
Parent article: The embedded long-term support initiative

A perverse problem is that customers *hate* updating devices. Everyone I know who owns a Playstation 3 complains when Sony push a mandatory update down their throat. They just want to play a game, or watch a Blu-ray, not have to wait 30 minutes for an update to be downloaded and installed, together with the presentation of a new multi-page EULA displayed in a tiny unreadable font. And Sony probably have the best, most streamlined update process of any embedded device vendor! People like me are stuck with an old, known buggy version of Android because their mobile phone model was discontinued a year ago and their phone company don't have any incentive to push out a software update.

IMO, updates have to be re-thought completely. The ideal system will update automatically when the device is not in use; will perform updates silently, in the background, without imposing any interruption upon the end-user (and yet, the updates have to be 'in effect' immediately; e.g., quitting and restarting the updated movie playing application in the device without dropping a frame or interrupting the audio); will never cause any regressions; will not be spoofable by a nefarious third party that wants to take over my TV in order to send spam (or display fake news stories); and will not be deferrable, or even disable-able, without disconnecting the system from the network entirely. A tall order!

to post comments

The embedded long-term support initiative

Posted Oct 29, 2011 11:40 UTC (Sat) by kragilkragil2 (guest, #76172) [Link]

They don't hate updating their Chrome browsers, because they hardly ever notice. It is all about the way you do it.

And I have to strongly disagree with Sony being a good example.
Their updates are way too big, dumb and really are mostly made of fail:
http://arstechnica.com/gaming/news/2011/09/resistance-3-h...
http://arstechnica.com/gaming/news/2010/08/dear-sony-ther...
They should provide delta updates, that download in the background and are fast to install, all of which they don't do.

MS updates Xbox games way faster and most of their OS/firmware updates for Xbox are tiny and fast, only twice a year there is a big update.

The embedded long-term support initiative

Posted Oct 29, 2011 16:19 UTC (Sat) by fuhchee (guest, #40059) [Link] (3 responses)

"Everyone I know who owns a Playstation 3 complains when Sony push a mandatory update down their throat."

That's partly because Sony is known to regularly abuse their customers this way.

The embedded long-term support initiative

Posted Oct 29, 2011 16:23 UTC (Sat) by cortana (subscriber, #24596) [Link] (2 responses)

Ugh. My point is that at least Sony bother to push out updates--unlike HTC or Samsung, who don't bother (not singling them out--I just happen to own devices made by both that I know are based on Linux, and have security vulnerabilities, and that don't have any updates available). And yet, the average user may _prefer_ the vulnerable device because it's less hassle to use--no pesky updates interrupting them!

The embedded long-term support initiative

Posted Nov 2, 2011 15:05 UTC (Wed) by jmm82 (guest, #59425) [Link] (1 responses)

People do not like the PS3 updates because they often lock down the system more and take away liberties they once had.

People do not like updating the firmware on their router because everyone has updated a router that worked perfectly fine only to either a. brick the device or b. have some feature that used to work suddenly intermittently fail.

Updating a kernel on a stable system is risky business. The people here should know that more than anyone, yet if security is a priority then it must be done.

The embedded long-term support initiative

Posted Nov 2, 2011 16:10 UTC (Wed) by dlang (guest, #313) [Link]

If the updates add additional features and seldom fail then customers generally like them

It's when the updates either don't add any new features, or worse, remove existing features (even if they add other new features), then users complain

This isn't limited to firmware/kernel updates.

The KDE and GNOME '.0' updates are perfect examples of updates that remove some things that existing users notice and therefor people are unhappy with them, even though the developers add a lot of new features as well.

The embedded long-term support initiative

Posted Oct 30, 2011 7:33 UTC (Sun) by jcm (subscriber, #18262) [Link] (19 responses)

I hate updating devices but for another reason: realistic pragmatism.

While it's fun to apply random software updates when you're hacking on some gadget, I personally like my technology to work when I need it to. Updates that fix one liner security bugs are useful to have and I am glad they are made available. But updates that introduce a whole new kernel version or make other non-critical updates (perhaps even adding new features) do me a disservice as a consumer. When these changes are made, they introduce the risk that the update breaks something I am relying on for production use. Most non-hackers also realize this reality of life. They know that computers are not perfect, updates often seem to introduce problems, and things are working "fine" today so they don't need "fixing". The real solution is to provide rock solid security and other critical fixes only so that consumers will feel safe in applying updates in the future.

"Oh but Jon, you're just...". Yea. I am. For the same reason I own three Android phones and deploy all Google supplied OTA updates to one test phone before allowing my production phone to update, or stage other updates before allowing them near production machines. The last OTA update to my Android phone running stock Nexus S software was supposed to contain a fix to the previously issued OTA update that I had not yet received and which would have broken tethering. Alas, this OTA update with the "fix" also breaks tethering on my staging one. By having an interim staging testbed I am able to avoid a crticial feature breaking for me because I didn't allow it near my phone. At least the phone gave me an option of deferring the update. Had it taken the "we know better" approach of updating automatically without any choice, I would have no useful way to connect on the go at this point. So, automatic updates without a choice to skip them are bad, updates that provide other than critical fixes are unwanted, and long term support should focus on what I actually want: just the bare minimal set of fixes until I choose to go to a newer product or revision of the software for that product.

Jon.

The embedded long-term support initiative

Posted Oct 30, 2011 11:34 UTC (Sun) by nix (subscriber, #2304) [Link] (18 responses)

Quite so. There's another thing: reversion. On your local machine you can revert failed upgrades or upgrades you just don't like easily, but mobile devices rarely give you that freedom. Also, because of the absence of anything like a boot menu, if the update is completely nonfunctional you have a brick.

So they are less like PC kernel upgrades than like BIOS upgrades. Quick, who here keeps their BIOS religiously up to date? I think I've upgraded mine once, when I had a serious bug I had to fix, and never again, because every upgrade carries with it the possibility of bricking your machine (and is there a recovery path other than pulling the chip and putting in a new one? Do you *have* a spare? I doubt it.)

Now thanks to ACPI and SMM, BIOSes can have security holes in them -- in fact given their general code quality I suspect they are a mass of security holes packed edge-to-edge with no space between. But *even so* I don't upgrade unless I must, and upgrading fills me with trepidation. Embedded and mobile devices are just like that, except that if the upgrade is automatic you don't even get a chance to say no (or in the case of games consoles 'hey, I pay for my bandwidth, you bastard!')

The embedded long-term support initiative

Posted Oct 30, 2011 15:37 UTC (Sun) by kragilkragil2 (guest, #76172) [Link] (14 responses)

IMO the suckage of current update mechanisms is no good argument for not having a sane automatic one in the future.

Chrome and ChromeOS seems to do it mostly right, so it not impossible.

Automatic delta updates on two OS partitions are a sane solution. Of course they should be better tested than what Google provides OTA atm, but in case you miss tethering you have the old OS partition without the updates to go to.

The embedded long-term support initiative

Posted Oct 30, 2011 21:09 UTC (Sun) by nix (subscriber, #2304) [Link] (13 responses)

Indeed not. I wasn't trying to argue against proper update mechanisms, with reversion and non-insane bandwidth requirements and maybe even *gasp* a changelog you can see as the download happens.

I was just pointing out just how far from that the current state of the art is.

The embedded long-term support initiative

Posted Oct 30, 2011 22:04 UTC (Sun) by jcm (subscriber, #18262) [Link] (1 responses)

The problem with partitions and current rollback is that it doesn't actually work. What actually happens is that your data sitting on a separate partition from the system is automatically converted to whatever newer incompatible format has been shoved into the update with no thought to rollback.

No. I don't trust updates to working production systems. I already have to duplicate everything I use to work around broken design. The fault is with changing what works. If it works, don't break it until the world moves to less hackish approaches to software platform consistency.

Jon.

The embedded long-term support initiative

Posted Oct 31, 2011 19:06 UTC (Mon) by dlang (guest, #313) [Link]

Tivo does this right. the data is on a separate partition, and they do not change it to the 'flavor of the day' format in the update.

10 years worth of updates and still running

Bwahahah

Posted Oct 31, 2011 7:35 UTC (Mon) by khim (subscriber, #9252) [Link] (10 responses)

I wasn't trying to argue against proper update mechanisms, with reversion and non-insane bandwidth requirements and maybe even *gasp* a changelog you can see as the download happens.

It's not even funny. When geeks start talking about "proper update mechanism" they invariably raise insane requirements which are totally pointless and useless.

Most people encounter "almost perfect" update mechanism (with almost 100% updated devices!) every day and don't even know that. Where are these devices? How come noone talks about them? Well... why should they? They "just work"(tm) - and that's enough. And people cheerfully keep then up-to-date without any complains!

Heresy? Fantasy? Nope. I'm talking about your cable TV set-top box. These are quite complex devices novadays (with built-in browser, the ability to write and replay TV programs, etc, etc). They are regularly updated (when TV company decides to sell you new capabilities) - and people rarely complain.

Note: no reversion mechanism, no "sane bandwidth requirements" and no changelogs. Just two things are required:
1. It must be invisible (i.e.: update is pushed not when it urgently needed but slow and steady - this way bandwidth requirements are hidden).
2. "It must work"(tm): if update goew haywire then it's not something user should fix somehow, this occasion should be treated the same way power brick explosion is treated (and it should happen rarely - like power brick explosions).

ChromeOS updates are decent imitations but sadly they are not as robust for now...

Bwahahah

Posted Oct 31, 2011 15:50 UTC (Mon) by aginnes (subscriber, #81011) [Link] (3 responses)

Some set-top boxes have an architecture where they keep two copies of the system software so they can revert to the previous version if it all goes wrong.

It's a lot easier for the set-top box software to be tested before it is downloaded, plus they own the broadcast bandwidth, so they can use as much as they want. The cable (or satellite, or IPTV) company is operating in a highly constrained environment, they control what devices are connected to their network, they know exactly what hardware is out there. So they can test the software on every different type of hardware that they have deployed before it goes out. This may take them a couple of months. Of course they have a big incentive to make sure their updates don't break the box as every call to their customer support call centre costs money, and if they brick the box, a truck roll costs an arm and a leg!

So updating set-top boxes in a closed environment is a significantly different (and easier) problem to a general "update Linux on any embedded device".

Bwahahah

Posted Oct 31, 2011 16:21 UTC (Mon) by martinfick (subscriber, #4455) [Link] (1 responses)

They also do not have any real user data on them, which as jcm points out is usually the real problem with updates.

Of course they do!

Posted Oct 31, 2011 20:15 UTC (Mon) by khim (subscriber, #9252) [Link]

Even older boxes had customization capabilities (for example you had the ability to select few "favorite" channels). Newer ones often include DVR capabilities and video rent capabilities so personal information is most definitely there.

What you probably meant is "they contain no unclassified personal data in there"... but is a good thing to have on your embedded device? We are entering era of parental computing - and set-top boxes show that people are quite willing to accept it if it means hassle-free gadgets.

Sure...

Posted Oct 31, 2011 16:58 UTC (Mon) by khim (subscriber, #9252) [Link]

So updating set-top boxes in a closed environment is a significantly different (and easier) problem to a general "update Linux on any embedded device".

Sure. But my point still stands:
1. It's possible to safely upgrade software in a device.
2. People tolerate updates just fine as long as they "just work".
3. Reversions, bandwidth requirements and changelogs don't affect acceptance at all.
3a. As you've noted people rarely care about bandwidth, but they do care about money spent on bandwidth. This is different - and solveable - problem.

Some set-top boxes have an architecture where they keep two copies of the system software so they can revert to the previous version if it all goes wrong.

Sure, but this is the same approach ChromeOS is using. This is mereluy detail of upgrade mechanism implementation which makes it more robust. I've never seen a set top box which allowed you to arbitrarily select one of two version of software to boot. Sometimes it can be accomplished using some combinations of knobs, but it's usually part of "recovery procedure", not something end-user is supposed to do.

When (and if) btrfs will be mature enough snapshots can do the same with smaller overhead.

Bwahahah

Posted Oct 31, 2011 17:43 UTC (Mon) by nix (subscriber, #2304) [Link] (2 responses)

Yeah, that would be enough... if you trusted things of the complexity of modern software upgrades to 'just work'. All too often, they don't. All too often, even BIOS upgrades don't, and BIOSes are not very complex compared to most software.

BIOS is certainly less complex then set-top box :-)

Posted Oct 31, 2011 20:50 UTC (Mon) by khim (subscriber, #9252) [Link] (1 responses)

Sure, BIOS updates sometimes fail - but that just means they were not designed to be constantly updated. Aforementioned set-top boxes are significantly more complex then BIOSes - yet they are routinely upgraded with no ill effects.

Actually another example fits as well: PS3 upgrades may be annoying because they routinely remove features, but situation when they break something they are not designed to break are exceedingly rare.

It's not hard or impossible to develop thing which can be safely auto-updated, that just means that you must severely restrict it. Complexity has nothing to do with the ability to reliably upgrade something. Diversity is the killer.

BIOS is certainly less complex then set-top box :-)

Posted Nov 1, 2011 14:22 UTC (Tue) by nix (subscriber, #2304) [Link]

True. I suppose the problem with BIOS updates is mostly the diversity of the hardware the BIOS is used with, and the near-impossibility of thoroughly testing it, not with all conditions found during use, but with all conditions found *at boot* or even during the running of an update: hence the relative likelihood of boot-time failures bricking your system after a BIOS upgrade.

I do wonder, though, how much of the set-top boxes' firmware is actually upgraded by an update. If the components necessary to boot and do an update don't change, then that might reduce the likelihood of failure -- though the fact that we rarely see them fail in any way suggests that this is not the problem.

So... continuous upgrading works for stuff (even complex stuff) that runs in simple or consistent environments or that is not itself necessary to run the upgrade process. That's probably enough for embedded stuff, since their environments are normally nailed down by the vendor and all variations known. Just look out for embedded systems that run as part of larger systems, and whose upgrading is controlled by the upstream vendor: those may not have been tested in the environment they're being upgraded in. (This, too, is probably rare: the integrator would probably want to control the upgrade stream in such a case, since it's them on the line if things go wrong.)

Bwahahah

Posted Oct 31, 2011 17:47 UTC (Mon) by raven667 (subscriber, #5198) [Link] (1 responses)

You've just described how the bandwidth requirements are managed and these devices certainly do allow updates to be reverted so I'm not sure you made the point you wished to make. Further more cable boxes are often owned by the cable company, not the subscriber, and they certainly do test and read changelogs before updating their own hardware. Are you suggesting that end-user hardware should only be rented and not owned?

Who owns the system, who is responsible for doing the maintenance? That's the person who wants access to manage revisions and read changelogs although certainly they decide to just apply changes as they become available. Having a robust mechanism makes that easier.

Perhaps your providers are different from mine...

Posted Oct 31, 2011 21:11 UTC (Mon) by khim (subscriber, #9252) [Link]

These devices certainly do allow updates to be reverted.

Technically - yes, they can. But then they next picture you'll see on TV when you'll connect them to network is "please wait while system update is installed". IOW: revert capability is only there to assist in upgrade process, it's not designed to be used by end-user (except when directed by tech support to "properly" upgrade device).

Further more cable boxes are often owned by the cable company, not the subscriber, and they certainly do test and read changelogs before updating their own hardware.

Does not make any difference in all cases I've encountered: yes, you can buy the box - but this only affects you monthly payments, if you buy it you can do whatever you want with it till you connect it to cable network. But if you do want to connect it to cable network - you automatically agree to install network's firmware.

Who owns the system, who is responsible for doing the maintenance?

These are two different question - and they naturally have different answers. That's true for plumbing, electric wiring, cars and so on... so why should it be the same for computers and gadgets?

That's the person who wants access to manage revisions and read changelogs although certainly they decide to just apply changes as they become available. Having a robust mechanism makes that easier.

Right, but how to help the guy who actually does maintenance is separate issue from how to push updates to end-user. Even there changelogs are not all that helpful: someone who's responsible for pushing updates to end-user does not need to know what changes are there, s/he only needs to know what can go wrong and how to fix the problems.

Bwahahah

Posted Nov 1, 2011 1:49 UTC (Tue) by foom (subscriber, #14868) [Link]

Sorry, but your example doesn't work for me. The number of people I know who have had a software update to their leased DVR set-top-box erase all their stored shows is quite high.

Even though that's the case, people still don't tend to complain...very much...because it's just recorded TV programs, after all, not something *important*.

The embedded long-term support initiative

Posted Oct 30, 2011 16:59 UTC (Sun) by andreasb (guest, #80258) [Link] (1 responses)

BIOS flashing and recovery isn't as bad as it was 10 years ago. Just as an example, the Asus P5Q board I have here goes into an automatic recovery mode when it finds a bad BIOS checksum where it will attempt to find a BIOS image on CD, floppy or USB stick and flash it. Plus it has a menu driven BIOS flashing program in the main BIOS for regular upgrades.

Thankfully no comparison to the olden times when you had to boot DOS from floppy to do an upgrade and had to replace the flash chip (or at least reprogram it outside the system) when it got corrupted

The embedded long-term support initiative

Posted Oct 30, 2011 21:12 UTC (Sun) by nix (subscriber, #2304) [Link]

Unfortunately, those olden times are still here for pretty much everyone but Asus (who remain excellent). e.g. the Tyan mobo in my server has a DOS flashing program and can only boot from CD, hard drive, or floppy. The machine doesn't have a floppy drive, so in order to flash the BIOS I have to find a way to get DOS onto a bootable CD, then find a way to interact with a headless machine in order to run the flashing program and tell it to get going... and if it goes wrong there is of course no reversion mechanism at all.

But why would you want an error recovery mechanism on great big servers as good as the one on J Random Nobody's desktop? :(

The embedded long-term support initiative

Posted Oct 30, 2011 17:38 UTC (Sun) by raven667 (subscriber, #5198) [Link]

At least for servers I generally try to track the dell firmware. They don't seem to update unless there is a real reason so it's not so much breakage due to churn as it is avoiding already-fixed problems. There's not much worse than encountering a hardware crash bug that might have been fixed in a newer firmware revision.