The embedded long-term support initiative [LWN.net]

The embedded long-term support initiative

Posted Oct 29, 2011 10:21 UTC (Sat) by kragilkragil2 (guest, #76172)
Parent article: The embedded long-term support initiative

Nice start and definately an improvement. But shouldn't the long term goal be way longer support?

I don't want my TV,NAS,Router etc to be only secure for 2 years. Maybe just because not a lot of bad things happened so far doesn't mean they aren't to in the future. A world stuffed with vulnerable Linux devices that soon posess a lot of computatinal power and storage are a big target for spammers, crackers and spies. Especially with XP dying and Windows getting more secure.

In my perfect world embedded devices would standardize more so that some kind of automated update system would work and each device would pull updates automatically (with enough checks and backups so that very rarely something breaks)

It is not easy, but definately possible. But I guess consumers would need awareness to vote with their money units to really make it happen ;-(
I would love to see awareness about community supported devices grow. There are a few communities that provide easy updates. Most of them are way too hard for your average guy, that would need to change but OEMs could help there too. If they don't do more they are at risk that someone like Apple might come in and just improves on the current state of things a bit, but offers better support with new features for years. All the OEMs need to understand that they are selling products that are mostly defined by software now.

to post comments

The embedded long-term support initiative

Posted Oct 29, 2011 11:02 UTC (Sat) by cortana (subscriber, #24596) [Link] (25 responses)

A perverse problem is that customers *hate* updating devices. Everyone I know who owns a Playstation 3 complains when Sony push a mandatory update down their throat. They just want to play a game, or watch a Blu-ray, not have to wait 30 minutes for an update to be downloaded and installed, together with the presentation of a new multi-page EULA displayed in a tiny unreadable font. And Sony probably have the best, most streamlined update process of any embedded device vendor! People like me are stuck with an old, known buggy version of Android because their mobile phone model was discontinued a year ago and their phone company don't have any incentive to push out a software update.

IMO, updates have to be re-thought completely. The ideal system will update automatically when the device is not in use; will perform updates silently, in the background, without imposing any interruption upon the end-user (and yet, the updates have to be 'in effect' immediately; e.g., quitting and restarting the updated movie playing application in the device without dropping a frame or interrupting the audio); will never cause any regressions; will not be spoofable by a nefarious third party that wants to take over my TV in order to send spam (or display fake news stories); and will not be deferrable, or even disable-able, without disconnecting the system from the network entirely. A tall order!

The embedded long-term support initiative

Posted Oct 29, 2011 11:40 UTC (Sat) by kragilkragil2 (guest, #76172) [Link]

They don't hate updating their Chrome browsers, because they hardly ever notice. It is all about the way you do it.

And I have to strongly disagree with Sony being a good example.
Their updates are way too big, dumb and really are mostly made of fail:
http://arstechnica.com/gaming/news/2011/09/resistance-3-h...
http://arstechnica.com/gaming/news/2010/08/dear-sony-ther...
They should provide delta updates, that download in the background and are fast to install, all of which they don't do.

MS updates Xbox games way faster and most of their OS/firmware updates for Xbox are tiny and fast, only twice a year there is a big update.

The embedded long-term support initiative

Posted Oct 29, 2011 16:19 UTC (Sat) by fuhchee (guest, #40059) [Link] (3 responses)

"Everyone I know who owns a Playstation 3 complains when Sony push a mandatory update down their throat."

That's partly because Sony is known to regularly abuse their customers this way.

The embedded long-term support initiative

Posted Oct 29, 2011 16:23 UTC (Sat) by cortana (subscriber, #24596) [Link] (2 responses)

Ugh. My point is that at least Sony bother to push out updates--unlike HTC or Samsung, who don't bother (not singling them out--I just happen to own devices made by both that I know are based on Linux, and have security vulnerabilities, and that don't have any updates available). And yet, the average user may _prefer_ the vulnerable device because it's less hassle to use--no pesky updates interrupting them!

The embedded long-term support initiative

Posted Nov 2, 2011 15:05 UTC (Wed) by jmm82 (guest, #59425) [Link] (1 responses)

People do not like the PS3 updates because they often lock down the system more and take away liberties they once had.

People do not like updating the firmware on their router because everyone has updated a router that worked perfectly fine only to either a. brick the device or b. have some feature that used to work suddenly intermittently fail.

Updating a kernel on a stable system is risky business. The people here should know that more than anyone, yet if security is a priority then it must be done.

The embedded long-term support initiative

Posted Nov 2, 2011 16:10 UTC (Wed) by dlang (guest, #313) [Link]

If the updates add additional features and seldom fail then customers generally like them

It's when the updates either don't add any new features, or worse, remove existing features (even if they add other new features), then users complain

This isn't limited to firmware/kernel updates.

The KDE and GNOME '.0' updates are perfect examples of updates that remove some things that existing users notice and therefor people are unhappy with them, even though the developers add a lot of new features as well.

The embedded long-term support initiative

Posted Oct 30, 2011 7:33 UTC (Sun) by jcm (subscriber, #18262) [Link] (19 responses)

I hate updating devices but for another reason: realistic pragmatism.

While it's fun to apply random software updates when you're hacking on some gadget, I personally like my technology to work when I need it to. Updates that fix one liner security bugs are useful to have and I am glad they are made available. But updates that introduce a whole new kernel version or make other non-critical updates (perhaps even adding new features) do me a disservice as a consumer. When these changes are made, they introduce the risk that the update breaks something I am relying on for production use. Most non-hackers also realize this reality of life. They know that computers are not perfect, updates often seem to introduce problems, and things are working "fine" today so they don't need "fixing". The real solution is to provide rock solid security and other critical fixes only so that consumers will feel safe in applying updates in the future.

"Oh but Jon, you're just...". Yea. I am. For the same reason I own three Android phones and deploy all Google supplied OTA updates to one test phone before allowing my production phone to update, or stage other updates before allowing them near production machines. The last OTA update to my Android phone running stock Nexus S software was supposed to contain a fix to the previously issued OTA update that I had not yet received and which would have broken tethering. Alas, this OTA update with the "fix" also breaks tethering on my staging one. By having an interim staging testbed I am able to avoid a crticial feature breaking for me because I didn't allow it near my phone. At least the phone gave me an option of deferring the update. Had it taken the "we know better" approach of updating automatically without any choice, I would have no useful way to connect on the go at this point. So, automatic updates without a choice to skip them are bad, updates that provide other than critical fixes are unwanted, and long term support should focus on what I actually want: just the bare minimal set of fixes until I choose to go to a newer product or revision of the software for that product.

Jon.

The embedded long-term support initiative

Posted Oct 30, 2011 11:34 UTC (Sun) by nix (subscriber, #2304) [Link] (18 responses)

Quite so. There's another thing: reversion. On your local machine you can revert failed upgrades or upgrades you just don't like easily, but mobile devices rarely give you that freedom. Also, because of the absence of anything like a boot menu, if the update is completely nonfunctional you have a brick.

So they are less like PC kernel upgrades than like BIOS upgrades. Quick, who here keeps their BIOS religiously up to date? I think I've upgraded mine once, when I had a serious bug I had to fix, and never again, because every upgrade carries with it the possibility of bricking your machine (and is there a recovery path other than pulling the chip and putting in a new one? Do you *have* a spare? I doubt it.)

Now thanks to ACPI and SMM, BIOSes can have security holes in them -- in fact given their general code quality I suspect they are a mass of security holes packed edge-to-edge with no space between. But *even so* I don't upgrade unless I must, and upgrading fills me with trepidation. Embedded and mobile devices are just like that, except that if the upgrade is automatic you don't even get a chance to say no (or in the case of games consoles 'hey, I pay for my bandwidth, you bastard!')

The embedded long-term support initiative

Posted Oct 30, 2011 15:37 UTC (Sun) by kragilkragil2 (guest, #76172) [Link] (14 responses)

IMO the suckage of current update mechanisms is no good argument for not having a sane automatic one in the future.

Chrome and ChromeOS seems to do it mostly right, so it not impossible.

Automatic delta updates on two OS partitions are a sane solution. Of course they should be better tested than what Google provides OTA atm, but in case you miss tethering you have the old OS partition without the updates to go to.

The embedded long-term support initiative

Posted Oct 30, 2011 21:09 UTC (Sun) by nix (subscriber, #2304) [Link] (13 responses)

Indeed not. I wasn't trying to argue against proper update mechanisms, with reversion and non-insane bandwidth requirements and maybe even *gasp* a changelog you can see as the download happens.

I was just pointing out just how far from that the current state of the art is.

The embedded long-term support initiative

Posted Oct 30, 2011 22:04 UTC (Sun) by jcm (subscriber, #18262) [Link] (1 responses)

The problem with partitions and current rollback is that it doesn't actually work. What actually happens is that your data sitting on a separate partition from the system is automatically converted to whatever newer incompatible format has been shoved into the update with no thought to rollback.

No. I don't trust updates to working production systems. I already have to duplicate everything I use to work around broken design. The fault is with changing what works. If it works, don't break it until the world moves to less hackish approaches to software platform consistency.

Jon.

The embedded long-term support initiative

Posted Oct 31, 2011 19:06 UTC (Mon) by dlang (guest, #313) [Link]

Tivo does this right. the data is on a separate partition, and they do not change it to the 'flavor of the day' format in the update.

10 years worth of updates and still running

Bwahahah

Posted Oct 31, 2011 7:35 UTC (Mon) by khim (subscriber, #9252) [Link] (10 responses)

I wasn't trying to argue against proper update mechanisms, with reversion and non-insane bandwidth requirements and maybe even *gasp* a changelog you can see as the download happens.

It's not even funny. When geeks start talking about "proper update mechanism" they invariably raise insane requirements which are totally pointless and useless.

Most people encounter "almost perfect" update mechanism (with almost 100% updated devices!) every day and don't even know that. Where are these devices? How come noone talks about them? Well... why should they? They "just work"(tm) - and that's enough. And people cheerfully keep then up-to-date without any complains!

Heresy? Fantasy? Nope. I'm talking about your cable TV set-top box. These are quite complex devices novadays (with built-in browser, the ability to write and replay TV programs, etc, etc). They are regularly updated (when TV company decides to sell you new capabilities) - and people rarely complain.

Note: no reversion mechanism, no "sane bandwidth requirements" and no changelogs. Just two things are required:
1. It must be invisible (i.e.: update is pushed not when it urgently needed but slow and steady - this way bandwidth requirements are hidden).
2. "It must work"(tm): if update goew haywire then it's not something user should fix somehow, this occasion should be treated the same way power brick explosion is treated (and it should happen rarely - like power brick explosions).

ChromeOS updates are decent imitations but sadly they are not as robust for now...

Bwahahah

Posted Oct 31, 2011 15:50 UTC (Mon) by aginnes (subscriber, #81011) [Link] (3 responses)

Some set-top boxes have an architecture where they keep two copies of the system software so they can revert to the previous version if it all goes wrong.

It's a lot easier for the set-top box software to be tested before it is downloaded, plus they own the broadcast bandwidth, so they can use as much as they want. The cable (or satellite, or IPTV) company is operating in a highly constrained environment, they control what devices are connected to their network, they know exactly what hardware is out there. So they can test the software on every different type of hardware that they have deployed before it goes out. This may take them a couple of months. Of course they have a big incentive to make sure their updates don't break the box as every call to their customer support call centre costs money, and if they brick the box, a truck roll costs an arm and a leg!

So updating set-top boxes in a closed environment is a significantly different (and easier) problem to a general "update Linux on any embedded device".

Bwahahah

Posted Oct 31, 2011 16:21 UTC (Mon) by martinfick (subscriber, #4455) [Link] (1 responses)

They also do not have any real user data on them, which as jcm points out is usually the real problem with updates.

Of course they do!

Posted Oct 31, 2011 20:15 UTC (Mon) by khim (subscriber, #9252) [Link]

Even older boxes had customization capabilities (for example you had the ability to select few "favorite" channels). Newer ones often include DVR capabilities and video rent capabilities so personal information is most definitely there.

What you probably meant is "they contain no unclassified personal data in there"... but is a good thing to have on your embedded device? We are entering era of parental computing - and set-top boxes show that people are quite willing to accept it if it means hassle-free gadgets.

Sure...

Posted Oct 31, 2011 16:58 UTC (Mon) by khim (subscriber, #9252) [Link]

So updating set-top boxes in a closed environment is a significantly different (and easier) problem to a general "update Linux on any embedded device".

Sure. But my point still stands:
1. It's possible to safely upgrade software in a device.
2. People tolerate updates just fine as long as they "just work".
3. Reversions, bandwidth requirements and changelogs don't affect acceptance at all.
3a. As you've noted people rarely care about bandwidth, but they do care about money spent on bandwidth. This is different - and solveable - problem.

Some set-top boxes have an architecture where they keep two copies of the system software so they can revert to the previous version if it all goes wrong.

Sure, but this is the same approach ChromeOS is using. This is mereluy detail of upgrade mechanism implementation which makes it more robust. I've never seen a set top box which allowed you to arbitrarily select one of two version of software to boot. Sometimes it can be accomplished using some combinations of knobs, but it's usually part of "recovery procedure", not something end-user is supposed to do.

When (and if) btrfs will be mature enough snapshots can do the same with smaller overhead.

Bwahahah

Posted Oct 31, 2011 17:43 UTC (Mon) by nix (subscriber, #2304) [Link] (2 responses)

Yeah, that would be enough... if you trusted things of the complexity of modern software upgrades to 'just work'. All too often, they don't. All too often, even BIOS upgrades don't, and BIOSes are not very complex compared to most software.

BIOS is certainly less complex then set-top box :-)

Posted Oct 31, 2011 20:50 UTC (Mon) by khim (subscriber, #9252) [Link] (1 responses)

Sure, BIOS updates sometimes fail - but that just means they were not designed to be constantly updated. Aforementioned set-top boxes are significantly more complex then BIOSes - yet they are routinely upgraded with no ill effects.

Actually another example fits as well: PS3 upgrades may be annoying because they routinely remove features, but situation when they break something they are not designed to break are exceedingly rare.

It's not hard or impossible to develop thing which can be safely auto-updated, that just means that you must severely restrict it. Complexity has nothing to do with the ability to reliably upgrade something. Diversity is the killer.

BIOS is certainly less complex then set-top box :-)

Posted Nov 1, 2011 14:22 UTC (Tue) by nix (subscriber, #2304) [Link]

True. I suppose the problem with BIOS updates is mostly the diversity of the hardware the BIOS is used with, and the near-impossibility of thoroughly testing it, not with all conditions found during use, but with all conditions found *at boot* or even during the running of an update: hence the relative likelihood of boot-time failures bricking your system after a BIOS upgrade.

I do wonder, though, how much of the set-top boxes' firmware is actually upgraded by an update. If the components necessary to boot and do an update don't change, then that might reduce the likelihood of failure -- though the fact that we rarely see them fail in any way suggests that this is not the problem.

So... continuous upgrading works for stuff (even complex stuff) that runs in simple or consistent environments or that is not itself necessary to run the upgrade process. That's probably enough for embedded stuff, since their environments are normally nailed down by the vendor and all variations known. Just look out for embedded systems that run as part of larger systems, and whose upgrading is controlled by the upstream vendor: those may not have been tested in the environment they're being upgraded in. (This, too, is probably rare: the integrator would probably want to control the upgrade stream in such a case, since it's them on the line if things go wrong.)

Bwahahah

Posted Oct 31, 2011 17:47 UTC (Mon) by raven667 (subscriber, #5198) [Link] (1 responses)

You've just described how the bandwidth requirements are managed and these devices certainly do allow updates to be reverted so I'm not sure you made the point you wished to make. Further more cable boxes are often owned by the cable company, not the subscriber, and they certainly do test and read changelogs before updating their own hardware. Are you suggesting that end-user hardware should only be rented and not owned?

Who owns the system, who is responsible for doing the maintenance? That's the person who wants access to manage revisions and read changelogs although certainly they decide to just apply changes as they become available. Having a robust mechanism makes that easier.

Perhaps your providers are different from mine...

Posted Oct 31, 2011 21:11 UTC (Mon) by khim (subscriber, #9252) [Link]

These devices certainly do allow updates to be reverted.

Technically - yes, they can. But then they next picture you'll see on TV when you'll connect them to network is "please wait while system update is installed". IOW: revert capability is only there to assist in upgrade process, it's not designed to be used by end-user (except when directed by tech support to "properly" upgrade device).

Further more cable boxes are often owned by the cable company, not the subscriber, and they certainly do test and read changelogs before updating their own hardware.

Does not make any difference in all cases I've encountered: yes, you can buy the box - but this only affects you monthly payments, if you buy it you can do whatever you want with it till you connect it to cable network. But if you do want to connect it to cable network - you automatically agree to install network's firmware.

Who owns the system, who is responsible for doing the maintenance?

These are two different question - and they naturally have different answers. That's true for plumbing, electric wiring, cars and so on... so why should it be the same for computers and gadgets?

That's the person who wants access to manage revisions and read changelogs although certainly they decide to just apply changes as they become available. Having a robust mechanism makes that easier.

Right, but how to help the guy who actually does maintenance is separate issue from how to push updates to end-user. Even there changelogs are not all that helpful: someone who's responsible for pushing updates to end-user does not need to know what changes are there, s/he only needs to know what can go wrong and how to fix the problems.

Bwahahah

Posted Nov 1, 2011 1:49 UTC (Tue) by foom (subscriber, #14868) [Link]

Sorry, but your example doesn't work for me. The number of people I know who have had a software update to their leased DVR set-top-box erase all their stored shows is quite high.

Even though that's the case, people still don't tend to complain...very much...because it's just recorded TV programs, after all, not something *important*.

The embedded long-term support initiative

Posted Oct 30, 2011 16:59 UTC (Sun) by andreasb (guest, #80258) [Link] (1 responses)

BIOS flashing and recovery isn't as bad as it was 10 years ago. Just as an example, the Asus P5Q board I have here goes into an automatic recovery mode when it finds a bad BIOS checksum where it will attempt to find a BIOS image on CD, floppy or USB stick and flash it. Plus it has a menu driven BIOS flashing program in the main BIOS for regular upgrades.

Thankfully no comparison to the olden times when you had to boot DOS from floppy to do an upgrade and had to replace the flash chip (or at least reprogram it outside the system) when it got corrupted

The embedded long-term support initiative

Posted Oct 30, 2011 21:12 UTC (Sun) by nix (subscriber, #2304) [Link]

Unfortunately, those olden times are still here for pretty much everyone but Asus (who remain excellent). e.g. the Tyan mobo in my server has a DOS flashing program and can only boot from CD, hard drive, or floppy. The machine doesn't have a floppy drive, so in order to flash the BIOS I have to find a way to get DOS onto a bootable CD, then find a way to interact with a headless machine in order to run the flashing program and tell it to get going... and if it goes wrong there is of course no reversion mechanism at all.

But why would you want an error recovery mechanism on great big servers as good as the one on J Random Nobody's desktop? :(

The embedded long-term support initiative

Posted Oct 30, 2011 17:38 UTC (Sun) by raven667 (subscriber, #5198) [Link]

At least for servers I generally try to track the dell firmware. They don't seem to update unless there is a real reason so it's not so much breakage due to churn as it is avoiding already-fixed problems. There's not much worse than encountering a hardware crash bug that might have been fixed in a newer firmware revision.

The embedded long-term support initiative

Posted Oct 30, 2011 19:48 UTC (Sun) by epa (subscriber, #39769) [Link] (16 responses)

Surely the answer is to stop relying on updates and patching to fix security holes after the fact. The software needs to be designed so that you know with reasonable certainty that it is secure as shipped, and further measures need to be in place to make sure that even if there is a vulnerability in one part of the system, it doesn't matter much. We have grown inured to releasing software with serious flaws* and patching it later. This would not be acceptable in any other industry. Yes, recalls and field modifications do happen, but they are the exception and considered an embarassment for the company that shipped a faulty product.

* For the sake of argument, anything that lets an attacker get your credit card number without lots of social engineering may be considered a serious flaw.

The embedded long-term support initiative

Posted Oct 30, 2011 22:24 UTC (Sun) by tialaramex (subscriber, #21167) [Link] (4 responses)

I struggle to imagine how any other industries could begin to compare. We're not talking about something like the lever or wheel. Programmable computers escape our ability to reliably understand what they're doing long before the scale where they're of any use to us, as Radó noticed.

You can try Formal Methods. If you're building a bomb, or maybe the core control systems for a nuclear reactor, you might persuade someone to pay for that. For their extra money, they will get something far harder to use and far less capable than everything else in the world, but very slightly more reliable.

The embedded long-term support initiative

Posted Oct 31, 2011 16:11 UTC (Mon) by zlynx (guest, #2285) [Link] (3 responses)

Formal Methods.

Hahah.

With those, all you have done is written the program twice. Once in the language and once in the formal verification.

(Almost) all of the mistakes it is possible to make while programming are possible to duplicate in the formal methods.

So if you have a security flaw where you have forgotten to specify that Method A must not be called before passing Security Check A, or if the specification does not spell out that calling Method B invalidates the current security state. That flaw will exist no matter how thorough the formal methods are.

The embedded long-term support initiative

Posted Nov 1, 2011 11:57 UTC (Tue) by dps (guest, #5725) [Link] (2 responses)

Formal methods are much more reliable than most programming languages. Just writing an unambiguous specifications helps. Formal methods also allow non-negotiable proof that a proposed solution meets the specification.

You can also prove things like *any* network of the stated components with an infinitely readable external input is deadlock free. I am not aware of any programming language that can do that.

In generally the experience is that using formal methods makes the initial phases longer and the final phases shorter, partly because a lot of problems and discovered sooner. Moves like only proving critical safety properties or analysing a high level design are popular.

As with any tool formal methods can be misused. I not only know formal methods but believe I have the taste to know when *not* to apply them.

Secure Software and Formal Methods

Posted Nov 2, 2011 11:18 UTC (Wed) by abacus (guest, #49001) [Link] (1 responses)

As far as I know formal methods are fine for functional specifications. I'm still waiting for a formal specification of a secure operating system though. See e.g. Gerwin Klein e.a., seL4: formal verification of an operating-system kernel, Communications of the ACM, Volume 53 Issue 6, June 2010.

Secure Software and Formal Methods

Posted Nov 2, 2011 17:33 UTC (Wed) by zlynx (guest, #2285) [Link]

My complaint about formal methods is that they aren't any kind of cure-all. They rely on the specifications being complete and that the specifications are correctly translated into the formal verification method.

In almost all programming, the specifications are incomplete or incorrect. Programmers, even without realizing what they are doing, fill in the gaps in incomplete specifications and may not even realize the specification was missing information.

Many bugs are caused by programmers on each side of an interface assuming different things about the specification. An example of this is the POSIX select() timeout parameter.

To fully specify the behavior of a system under all conditions is in my opinion, as prone to oversights and misunderstanding (aka bugs) as writing the program itself in assembly or C. (Well, you can make a lot more typo type mistakes writing in C, but I think it holds for logic errors.)

Reading through some of that seL4 paper you linked, I see they had to make about 300 changes in the spec during the process. In other words, they had to debug it. I wonder how they know if they found all the problems in their spec?

Who will pay for it?

Posted Oct 31, 2011 7:59 UTC (Mon) by khim (subscriber, #9252) [Link] (10 responses)

Surely the answer is to stop relying on updates and patching to fix security holes after the fact.

I doubt it.

The software needs to be designed so that you know with reasonable certainty that it is secure as shipped, and further measures need to be in place to make sure that even if there is a vulnerability in one part of the system, it doesn't matter much.

Do you propose a new law? What measures do you propose to make sure software development will not move to other, more liberal, countries?

Because without government mandate any company which will try this approach will just go bancrupt (by the time it'll release anything market will move to the "next big thing") so you'll need something big to make it happen. Do you really believe bureaucracy struggles over such mandates (this what will actually happen, I doubt quiality of the software itself will grow all that much) will actually make your life easier or better?

This would not be acceptable in any other industry.

Are you sure? From what I'm seeing other industries view this as a problem not as an achievement and move to "built-in computer with updates" model where they can. Sure, centuries-old industries are too conservative to eploy such ideas, but other, newer, industries... mobile phones, TV sets, blu-ray players and so on: they all switched from "what you buy is what you'll use till device will die" to "we'll fix any bugs later" model - and I'm sure other industries will follow.

Yes, recalls and field modifications do happen, but they are the exception and considered an embarassment for the company that shipped a faulty product.

They were exceptions and were considered as embarassment because they severely affected the bottom line. Now, when they are cheap... situation is changing. Sure, we'll not see upgradeable car computers any time soon (because cars have a lot of legal requirements around them), but "entertainment centers" in cars soon will surely follow the same model. Actually they were replaceable for a very long time so you can view them as precursors of today's "ship then fix" model...

Who will pay for it?

Posted Oct 31, 2011 15:19 UTC (Mon) by epa (subscriber, #39769) [Link] (6 responses)

Relying on updates to fix security holes does not work! It hasn't worked to keep us secure over the past twenty years, what reason is there to suppose it will work for the next twenty?

You mention mobile phones, Blu-ray players and so on. Those are all parts of the software industry. When I said 'this would not be acceptable in any other industry' I was referring to industries other than software. It is not good enough to put up an unsound building and return to fix it later when flaws are discovered. (It does happen, but is rare and shaming for the architects and builders involved.) You can't sell a desk lamp with unsafe wiring and rely on asking people to take it back to the store later. Only in the software industry do we try to get away with such practices. And as you say, the market usually tolerates it.

Who will pay for it?

Posted Oct 31, 2011 16:38 UTC (Mon) by mpr22 (subscriber, #60784) [Link]

In the building case: Correctness is feasible to achieve and validate, and the cost of remedying errors is relatively large.

In the desk lamp case: Correctness is not merely feasible but trivial to achieve and validate.

Software that does what there is currently perceived to be a demand for often lies somewhere between insanely hard and mathematically impossible to achieve and validate correctness of. The cost of remedying an error, on the other hand, is not strongly related to the severity of its consequences. Many disastrous software errors turn out to have trivial fixes.

(And, of course, even if your software is provable, all you can prove is that it conforms to the provided specification. Proving that the specification is a correct statement of the requirements, or that the requirements were well-formed in the first place, is a separate problem.)

What are you talking about?

Posted Oct 31, 2011 16:39 UTC (Mon) by khim (subscriber, #9252) [Link]

Relying on updates to fix security holes does not work!

On the contrary: it works very well indeed. The companies who use this approach survive and thrive. The companies who lost the wind and tried to fix all the bugs before shipment are history.

You mention mobile phones, Blu-ray players and so on.

Yup.

Those are all parts of the software industry.

Not even close. First mobile network started operating back in 1979, it was analogue and had nothing to do with software. First LD player was on sale year before that - and Bly-ray is it's direct descendant (from end-user POV). And first TVs were created even earlier: it was introduced back in 1928 and most definitely had nothing to do with software.

Only in the software industry do we try to get away with such practices.

Again: not true at all. Lots of industries use this approach too: mobile phones, credit cards, etc. Initially they had pathetic security but since they were convenient they were used anyway. Later, when frauds become a problem additional layers of security were added. The same happened with printed banknotes few centuries before. You can go back few thousands years (when first stamps and other similar tools were invented) - and see the very same picture. Again: special inks, papers and procedure and so on followed, not preceded.

In fact where information is exchanged "rely on updates to fix security holes" is typical approach, not an exception. The only thing software introduced is "fast" updates. When you introduce new, more protected, banknote you must to this in slow, very spread-out manner. But when new software is created to patch vulnerability... you can push it in hurry.

So no, I don't believe in "fix all the bugs before shipment" approach. It failed us for thousands of years - why do you think it can be fixed now?

Who will pay for it?

Posted Oct 31, 2011 16:58 UTC (Mon) by tialaramex (subscriber, #21167) [Link]

But we aren't talking about unsafe wiring in a desk lamp. Even if we were I'd argue that spending two hours travelling on the bus with a microwave (it caught fire spontaneously after ~8 months use) to get it replaced was a lot more hassle than any software update I've ever undertaken.

Your building example was better. But in fact it's completely routine to "snag" large office or industrial buildings.

When I helped take possession of a four story building in 1998 we were all issued with a sheet of orange stickers labelled "snag". Each problem we discovered was to be marked with a label, and added to a list maintained by the liaison to the building contractor. Some problems were corrected in a few days to our satisfaction, as well as if they'd been corrected during construction. Many were "bodged" so that they met the strict letter of the requirements, but were not really adequate (e.g. it's much cheaper to fiddle with the hinges on a door to make it close, for a few days until it settles again, than to buy and install a new door which fits properly). A few simply couldn't be fixed at all, no way around it without tearing down the building and beginning anew. Some orange stickers for those last problems remained as visible sores on the pristine new building until their adhesive failed years later.

If we added to the "snags" every conceivable way a resourceful and determined attacker could get into the building, we'd never have finished. What if they just drive a truck through the large glass frontage? What if they tailgate a legitimate employee? What if they pay someone to pull the fire alarm, dress as firemen, and just walk in?

Who will pay for it?

Posted Nov 1, 2011 2:02 UTC (Tue) by foom (subscriber, #14868) [Link]

> It is not good enough to put up an unsound building and return to fix it later when flaws are discovered. (It does happen, but is rare and shaming for the architects and builders involved.)

On the contrary, putting up a building with serious "bugs" is exceedingly common. Where it matters most (building will fall down), extra care is taken to make sure it won't fail in that way, but, for all the other ways a building can be broken, it's quite common for a new building to in fact *be* broken in all of those ways.

E.g. leaking excessive amounts of water, light switches are in stupid places that don't make any sense, HVAC doesn't work right (hot/cold areas), architect decided not to put stairwells in elevator lobby (but rather halfway across the building in a supposed-to-be-locked area), so it turns out it's illegal to actually lock the entrance to the floor from the (public) elevator lobby, etc...

Who will pay for it?

Posted Nov 2, 2011 8:15 UTC (Wed) by ekj (guest, #1524) [Link]

Writing bug-free software, does not work. As in, literally nobody has figured out how to do it, not even with near-infinite budgets for near-trivial computations.

Even those situations where we use the very strictest of quality-controls, and as a result end up paying orders of magnitude more than we would with "normal" software, we still get banal, -stupid- bugs like the Mars Climate Orbiter doing lithobraking due to one software-module using imperial units rather than metric like the rest of the software. (i.e. bugs not unlike those typical of normal off-the-shelf software)

In most lines of bussiness, simply *trying* to do software like that, would guarantee bankruptcy. There's a reason things are done the way things are done.

Who will pay for it?

Posted Nov 3, 2011 9:53 UTC (Thu) by jschrod (subscriber, #1646) [Link]

> It is not good enough to put up an unsound building and return to fix it
> later when flaws are discovered. (It does happen, but is rare and shaming
> for the architects and builders involved.)

I wish you lots of luck if/when you'll build your first house and that fantasy will be rather rudely destroyed.

Who will pay for it?

Posted Oct 31, 2011 15:22 UTC (Mon) by epa (subscriber, #39769) [Link] (2 responses)

...and obviously, an entertainment centre that has bugs is fine, because it is not safety critical and cannot be used to steal money from you. My point is that things which are important must be held to a higher standard than that used for video games.

Who will pay for it?

Posted Oct 31, 2011 15:35 UTC (Mon) by KSteffensen (guest, #68295) [Link]

An entertainment center used to show movies from some Netflix-like service could conceivably hold credit card information. so even things that don't immediately seem critical might be.

And some of us certainly wouldn't mind video games being held to a higher standard than what is currently the case.

Actually it CAN steal money from you - and that's the point

Posted Oct 31, 2011 16:48 UTC (Mon) by khim (subscriber, #9252) [Link]

...and obviously, an entertainment centre that has bugs is fine, because it is not safety critical and cannot be used to steal money from you....

It can be used for that - and that's the point.

Money themselves (coins and banknotes) were one of the first objects which employed "security updates" in it's design.

"Security updates" are not something invented with software - on the contrary, they have centuries long history! And yes, they worked. Frauds happened all the time, but as long as they were rare enough "security patches" worked. They worked for so long - why do you think tomorrow everything will suddenly collapse? What exactly changed today?