Jones: Future development of Trinity [LWN.net]

Here's a discouraging blog post from Dave Jones on why he will no longer be developing the Trinity fuzz tester. "It’s no coincidence that the number of bugs reported found with Trinity have dropped off sharply since the beginning of the year, and I don’t think it’s because the Linux kernel suddenly got lots better. Rather, it’s due to the lack of real ongoing development to 'try something else' when some approaches dry up. Sadly we now live in a world where it’s easier to get paid to run someone else’s fuzzer these days than it is to develop one."

to post comments

Jones: Future development of Trinity

Posted Jul 12, 2015 23:37 UTC (Sun) by josh (subscriber, #17465) [Link] (1 responses)

Doesn't sound like he's discontinuing work entirely (though reducing it), but that he doesn't plan to continue publishing it:

> And that’s why for the foreseeable future, I’m unlikely to make public any further feature work I do on it.

That's somehow even more discouraging than if he simply didn't have time to work on it. It's sad that a handful of black-hats have apparently ruined the fun of working on an exceptionally useful tool.

Jones: Future development of Trinity

Posted Jul 13, 2015 11:38 UTC (Mon) by nim-nim (subscriber, #34454) [Link]

Black hats or management (cf first quote in the post)?

Jones: Future development of Trinity

Posted Jul 13, 2015 3:33 UTC (Mon) by pabs (subscriber, #43278) [Link] (23 responses)

This seems like an opportunity for the Linux Foundation to step in and provide some financial motivation to develop in public.

Jones: Future development of Trinity

Posted Jul 13, 2015 16:11 UTC (Mon) by k3ninho (subscriber, #50375) [Link] (22 responses)

> This seems like an opportunity for the Linux Foundation to step in and provide some financial motivation to develop in public.

I think that Dave's whole point is that 'in public' is a losing strategy for the white-hat and innocent bystander communities.

It's precisely the same situation as having a private mailing list to co-ordinate responses to zero-day exloits of security vulnerabilities. Why? Trinity gives bad and good actors some scaffolding to reach above the low-hanging fruit of known bugs, and every improvement to that scaffolding will find and allow security exploits not yet known. On the white-hat side of this, knowing and fixing those bugs is vital; on the black-hat side, that's a financial win for each zero-day vuln found.

For the health of the ecosystem, it makes sense to avoid empowering black-hat actors with more vulnerabilities or cash. That's to say that it makes sense to develop extensions to Trinity among trusted actors, where the ability to exploit the system isn't advanced without also advancing the available fixes to bugs found, and even to make a co-ordinated response where horrific security flaws are found.

K3n.

Jones: Future development of Trinity

Posted Jul 13, 2015 16:35 UTC (Mon) by martin.langhoff (subscriber, #61417) [Link] (17 responses)

So a good quality fuzzer should belong in trusted private circles, just as you'd share news of a serious kernel vulnerability privately with a trusted core kernel developer.

It makes sense. It also makes sense that this should get funding from trusted corporate player, or from the Linux Core Infra Initialive folks...

Jones: Future development of Trinity

Posted Jul 13, 2015 18:51 UTC (Mon) by corbet (editor, #1) [Link] (16 responses)

I think that's kind of a sad conclusion, so, for whatever reason, I find myself resisting it. A good fuzzer, like any other tool, will benefit from open development. The community as a whole will benefit from people running the fuzzer. For all the unpleasantness that has happened, I think we're better off for having had Trinity out there.

It sucks that unpleasant people are using tools like Trinity. But, then, remember that nastiness like FinFly is based on Linux; maybe we should keep the kernel to ourselves too :)

Jones: Future development of Trinity

Posted Jul 13, 2015 20:50 UTC (Mon) by emunson (subscriber, #44357) [Link] (13 responses)

What I find most sad is that it was used to find exploits in the Android kernel, suggesting that the Android project is not already running it regularly.

I think we could probably pull the changes made by Hacking Team to fuzz ioctls into Trinity after cleaning them up and making them suck less in general. At least we might see some benefit from the whole mess.

Jones: Future development of Trinity

Posted Jul 14, 2015 6:41 UTC (Tue) by salimma (subscriber, #34460) [Link] (9 responses)

Android suffers from another problem of manufacturers not releasing updates for their older devices, no? Makes me think that the best possible future is of Google taking more control of software updates, like they're already doing with Android One, Nexus and Wear ... manufacturers can customize but they have to distribute through Google's channels.

Jones: Future development of Trinity

Posted Jul 14, 2015 11:57 UTC (Tue) by anatolik (guest, #73797) [Link] (3 responses)

Agree. I wish Android will take ChromeOS model of distributing releases.

Jones: Future development of Trinity

Posted Jul 14, 2015 13:19 UTC (Tue) by pizza (subscriber, #46) [Link]

With very few exceptions, ChromeOS is running on commodity PC hardware well supported by Mainline Linux. Meanwhile, Android is anything but.

Jones: Future development of Trinity

Posted Jul 15, 2015 9:53 UTC (Wed) by Tjebbe (guest, #34055) [Link] (1 responses)

Getting totally offtopic here, but I wish people buying phones would demand the OS be uncoupled from the hardware, and the hardware specs be open. That way, the provided os is seen as 'just a default' and they can run and upgrade despite the vendor having no interest in aftermarket support. Seems much better to me in the long run than shifting full dependency onto the next organization that is well-known for dropping support to projects it no longer deems fruitful enough.

Jones: Future development of Trinity

Posted Jul 15, 2015 18:57 UTC (Wed) by Wol (subscriber, #4433) [Link]

Add to that, the amount of bloatware is awful. I've just upgraded my phone - didn't have much choice. With only about two or three (yes, really) apps of *MY* choice on the phone, updates kept crashing with "insufficient space on device" - yes the bloatware had chewed up pretty much *all* the memory.

It's a perfectly decent, usable phone, rendered useless by all the unwanted crap forced onto the device by Google and Samsung.

Cheers,
Wol

Jones: Future development of Trinity

Posted Jul 14, 2015 15:09 UTC (Tue) by raven667 (subscriber, #5198) [Link] (4 responses)

They have already done this, they used to have a problem with updates but they've moved most of their critical OS components into the Google Play Services application library which can be updated like any other without the carrier or handset maker interfering. One of the few components this doesn't cover is the kernel but because of the way handsets are developed, without discoverable busses and each being a one-off design with a one-off kernel, without direct upstream support, there is a lot of trouble with kernel updates anyway. Saying you want a Nexus-like update experience on the thousands of different kinds of Android systems out there is saying that you want Google to take over kernel development for thousands of incompatible hardware platforms which I don't think even they have the resources to do, a huge expense for very little gain in the bottom line.

Herding the vendors closer and closer together, with consortiums and compatibility certifications, to eventually get to where a small number of kernel binaries can directly boot on thousands of different devices will greatly reduce the amount of maintenance and porting done in the entire Android ecosystem, and the cost for most of the vendors in the long term, but it'll take a loooong time to get there.

Jones: Future development of Trinity

Posted Jul 15, 2015 21:01 UTC (Wed) by Wol (subscriber, #4433) [Link]

This sounds what device-tree is meant to address. Which isn't that far off, afaict. The big problem is the huge amount of duplicated crap in the arm tree, where the same chipset may have multiple drivers, each derived from a different starting point, and all replaceable by just one driver if only the human responsible was aware they are the same chipset.

Cheers,
Wol

Jones: Future development of Trinity

Posted Jul 16, 2015 14:47 UTC (Thu) by thestinger (guest, #91827) [Link] (2 responses)

> They have already done this, they used to have a problem with updates but they've moved most of their critical OS components into the Google Play Services application library which can be updated like any other without the carrier or handset maker interfering.

No, they haven't done that at all. AOSP still includes all of the critical OS components and is the basis for ROMs like CyanogenMod. Google Play Services only provides the proprietary Google services listed here:

https://en.wikipedia.org/wiki/Google_Play_Services

The only notable core OS features that are missing without it are gesture typing for the keyboard (depends on proprietary library) and the network location service which can be provided by the FOSS UnifiedNlp app (which supports local databases, so it can work without an active WiFi / data connection).

They *did* separate most of the user-facing apps from the core operating system and are able to update them independently. There are extended proprietary versions of some AOSP apps with Google integration and some of the core apps were pretty much killed off completely. The WebView widget was also split out into an app package and can be updated through the package manager.

The vast majority of the core operating system (95%+ of the 500+ Git repositories Android is composed of) can only be updated through an OTA update. WebView is one of the very few exceptions.

Jones: Future development of Trinity

Posted Jul 16, 2015 16:54 UTC (Thu) by raven667 (subscriber, #5198) [Link]

> They *did* separate most of the user-facing apps from the core operating system and are able to update them independently.

Yes, most of the new development is not in the core OS anymore but is available as updates from the Google Play Store by splitting out the applications and core libraries from the OS leaving OS components that don't have as much ongoing maintenance and new feature development.

The link you posted doesn't have a very complete list, this is more detailed

https://developers.google.com/android/guides/setup

which includes a huge amount of the new features advertised for new releases of Google Android applications, that are no longer part of the core OS.

Jones: Future development of Trinity

Posted Jul 16, 2015 19:04 UTC (Thu) by flussence (guest, #85566) [Link]

The loss of gesture typing is the only pain I've encountered after removing Play Services from my phone (though that in itself was quite a PITA). None of the alternative keyboards on F-Droid are as easy to use, and the basic stock keyboard rubs salt in the wound by recognising and drawing gesture trails anyway without acting on them.

Apart from that it's still a fully functional OS, like you've said. The slower user input is more than made up for by not having a giant proprietary library in the background eating a chunk of resources, and all the unwanted secondary effects that leads to.

Jones: Future development of Trinity

Posted Jul 14, 2015 21:41 UTC (Tue) by smoogen (subscriber, #97) [Link]

I think that is a faulty logic conclusion. The Android kernel may get all kinds of trinity testing.. the 4000+ vendor versions of the kernel are a completely different story since they may turn on all kinds of things that the 'main' Android kernel Google sends out 'off'. The fact that phone hardware changes so much so quickly that the phones at the beginning of a manufacturing run may be completely different from the ones at the end just make it an impossible job to know what is being used where/when and test it all.

Jones: Future development of Trinity

Posted Jul 14, 2015 22:23 UTC (Tue) by excors (subscriber, #95769) [Link] (1 responses)

The Android project is quite a long way removed from what actually gets shipped on devices, so I think they're not in a particularly good position to do that testing.

Google provides some patches that are required on top of the mainline kernel by their Android userspace code. Then sometimes Linaro is involved in porting those patches to other kernel versions and adding their own changes. Then the SoC vendor adds a load of drivers for their hardware, and optimisations and bug fixes. Often they'll integrate changes from third-party IP vendors, e.g. graphics drivers. Then the device manufacturer takes the SoC kernel and integrates drivers provided by the vendors of all the other chips in their device, and writes some of their own to glue everything together. Then eventually the device reaches customers, and the manufacturer dumps a tarball of the kernel onto their web site.

That whole process adds a lot of new non-upstreamed kernel code, perhaps on the order of a million lines in total per device (and many times more if you aggregate all the different devices in the world), often written by people who aren't experienced kernel developers and not reviewed by people who are.

So testing is difficult near the top of the chain (e.g. at Google's level) because they have no access to a lot of the buggy code that will end up in a device. It's probably easier to test near the bottom of the chain, e.g. by the manufacturers, but harder to do anything about failures - they're not experts in all the drivers they've integrated, and trying to get the bugs analysed and fixed by upstream vendors is likely to take months, and by then the device will have already shipped and the code branch will be frozen to avoid the risk of regressions.

I suppose the fundamental problem is that the manufacturers just don't particularly care about keeping the user's data secure, so they don't apply much pressure on the upstream vendors to provide more secure technology in the first place. There are some things they *do* care about keeping secure, like secret DRM keys and the decrypted video data, and then vendors spend a lot of resources on things like TrustZone to support that, because it directly affects sales of their chips. But the entire Linux kernel and an outdated web browser and all the dodgy apps the user has ever downloaded are stuffed into the non-secure zone alongside all the user's private data.

Jones: Future development of Trinity

Posted Jul 15, 2015 2:34 UTC (Wed) by nevets (subscriber, #11875) [Link]

> So testing is difficult near the top of the chain (e.g. at Google's level) because they have no access to a lot of the buggy code that will end up in a device. It's probably easier to test near the bottom of the chain, e.g. by the manufacturers, but harder to do anything about failures

I'm not so sure about that. As the black-hats have the same issue (they have access to the high level kernel, but probably not all the derivatives for the devices), testing Google's level of the kernel will most likely catch the same bugs that the black hats are finding.

Jones: Future development of Trinity

Posted Jul 13, 2015 22:38 UTC (Mon) by pjhacnau (subscriber, #4223) [Link]

I think the problem is not so much that black hats are taking Trinity and using it. The problem is that:
a) The bulk of people using Trinity are black hats who are extending/enhancing it to find new exploits to use (not fix)
b) The few non-black hats aren't sharing their changes either.

So you get all the bad points of a development-in-public model with none of the good points.

Jones: Future development of Trinity

Posted Jul 14, 2015 20:48 UTC (Tue) by k3ninho (subscriber, #50375) [Link]

That's not the conclusion I intended. It seems reasonable that the community runs the fuzzer, yes. Also reasonable that responsible disclosure of the vulnerabilities found is matched to bugs fixed so that the next round of improvements get shared after the issues have been fixed. It's not feasible that everyone is an expert on every subsystem or fuzzing approaches for every subsystem, so there will be waves of patches adding features and exposing bugs. Those waves can first go to a trusted community and then to the wider community.

K3n.

Jones: Future development of Trinity

Posted Jul 13, 2015 17:52 UTC (Mon) by droundy (guest, #4559) [Link] (3 responses)

The problem with this argument is that it places the burden on each possible white-hat developer to work on his or her own, which still leaves the black hats with the advantage. True, if you could effectively distinguish between good and bad people, you could enable sharing between white-hats, but I don't know anyone with that superpower.

I don't have a solution, but it seems like any solution needs to involve greater rewards (whether financial or otherwise) for constructive behavior. Or alternatively, reducing the rewards for destructive behavior, which seems less feasible.

Jones: Future development of Trinity

Posted Jul 13, 2015 23:30 UTC (Mon) by roc (subscriber, #30627) [Link] (2 responses)

People and organizations who are deeply invested in deploying Linux-based products have a powerful incentive to not collaborate with black-hats on exploit development. They can collaborate among themselves on that basis.

Jones: Future development of Trinity

Posted Jul 14, 2015 0:46 UTC (Tue) by andresfreund (subscriber, #69562) [Link] (1 responses)

Since Dave Jones for years worked for Redhat I'm not sure what your point is in the face of his obvious dissatisfaction with the contributions made?

Jones: Future development of Trinity

Posted Jul 14, 2015 4:31 UTC (Tue) by roc (subscriber, #30627) [Link]

That black-hat-excluding selective cooperation is possible --- at least, you can identify who you should cooperate with. Of course such cooperation could still break down due to selfishness and other reasons. Though in this case, it sounds like it wasn't even tried.

Jones: Future development of Trinity

Posted Jul 13, 2015 21:34 UTC (Mon) by lkundrak (subscriber, #43452) [Link]

Wouldn't it make sense for Google to enhance Trinity to fuzz their Android ioctls now that there's a proof of concept that the approach is useful?

It would enhance security of their product and it will definitely be appreciated by the community.

Jones: Future development of Trinity

Posted Jul 14, 2015 4:03 UTC (Tue) by zblaxell (subscriber, #26385) [Link] (4 responses)

When I first started working on Android devices I noticed this simple conflict of interest:

I want a smartphone that recognizes and supports my authority and excludes all others (especially the smartphone vendor, mobile carrier, thieves, and most other third parties) using assorted technical measures.
Most smartphone vendors want to build phones that recognize and support their authority and excludes all others (including the account owner and end user--in particular, me--but not including various third parties the vendor has service agreements with) using assorted technical measures.

A white-hat security researcher who wants to disclose a shiny new zero-day they've found now faces a problem: many smartphone vendors are acting in a manner indistinguishable from black-hats. An owner of a new smartphone may have to literally defeat the black-hats who work in the smartphone factory in order to correct (or even merely verify) the device's notion of who its owner is.

Even worse, these black-hats believe they are white-hats. They self-describe as white-hats in their press releases. They invite themselves to the white-hats-only parties, sometimes even host them. They seek social and legal protections normally accorded only to white-hats when their own customers deploy technical and social countermeasures (aka malware removal tools) against their software. They reach for white-hat tools like legislation, litigation and regulation to create random legal distinctions between their software and malware where no technical difference exists. Their perception is so distorted by self-delusion that they misrecognize other white-hats as black-hats and other black-hats as white-hats, and end up designing sophisticated security mechanisms that are effective only against white-hats.

This creates a perverse disincentive to disclosure of security defects. I can't tell a white-hat vendor how to fix a vulnerability in some Android device without also helping a black-hat vendor lock me out of a device I might own in the future. As a result I no longer tell anyone anything at all.

Jones: Future development of Trinity

Posted Jul 23, 2015 19:43 UTC (Thu) by Garak (guest, #99377) [Link] (3 responses)

Indeed. Somewhere in your answer is my explanation for why I don't see some decent smartphone models with simple physical switches for the microphone mute, main battery power, etc...

Jones: Future development of Trinity

Posted Jul 24, 2015 4:15 UTC (Fri) by bronson (subscriber, #4806) [Link] (2 responses)

The reason is threefold: size, cost, and reliability. Physical switches are huge, heavy, break all the time, and really expensive compared to the other parts in the phone. That's why everyone has moved to capacitive soft-buttons. I'm kinda surprised to still see the volume rocker on most phones, maybe hearing loss laws require it?

While I agree that companies trying to lock down my phone against my wishes are on the wrong side of history, calling them black hats is a bit rich.

Jones: Future development of Trinity

Posted Jul 24, 2015 4:44 UTC (Fri) by Garak (guest, #99377) [Link] (1 responses)

Note the _complete_ absence of security threat modeling in your analysis. The shear simplicity of how a reasonably verifiable physical switch can provide strong mitigation to a vast threat surface of complex software vulnerabilities. The emperor wears no clothes, but everyone still trumpets the splendor of the threads. Even Snowden couldn't unravel the web they weave.

Jones: Future development of Trinity

Posted Jul 24, 2015 5:57 UTC (Fri) by bronson (subscriber, #4806) [Link]

Correct. The truth is a lot simpler than what you're claiming: the switches disappeared because of their bulk and cost. It has nothing whatsoever to do with security or Snowden.

If you're unhappy that the general public doesn't care to pay for more hard switches, take it up with the Facebook crowd. Phone manufacturers are just responding to market demands.