[personal profile] mjg59
A large part of the internet became inaccessible today after a botnet made up of IP cameras and digital video recorders was used to DoS a major DNS provider. This highlighted a bunch of things including how maybe having all your DNS handled by a single provider is not the best of plans, but in the long run there's no real amount of diversification that can fix this - malicious actors have control of a sufficiently large number of hosts that they could easily take out multiple providers simultaneously.

To fix this properly we need to get rid of the compromised systems. The question is how. Many of these devices are sold by resellers who have no resources to handle any kind of recall. The manufacturer may not have any kind of legal presence in many of the countries where their products are sold. There's no way anybody can compel a recall, and even if they could it probably wouldn't help. If I've paid a contractor to install a security camera in my office, and if I get a notification that my camera is being used to take down Twitter, what do I do? Pay someone to come and take the camera down again, wait for a fixed one and pay to get that put up? That's probably not going to happen. As long as the device carries on working, many users are going to ignore any voluntary request.

We're left with more aggressive remedies. If ISPs threaten to cut off customers who host compromised devices, we might get somewhere. But, inevitably, a number of small businesses and unskilled users will get cut off. Probably a large number. The economic damage is still going to be significant. And it doesn't necessarily help that much - if the US were to compel ISPs to do this, but nobody else did, public outcry would be massive, the botnet would not be much smaller and the attacks would continue. Do we start cutting off countries that fail to police their internet?

Ok, so maybe we just chalk this one up as a loss and have everyone build out enough infrastructure that we're able to withstand attacks from this botnet and take steps to ensure that nobody is ever able to build a bigger one. To do that, we'd need to ensure that all IoT devices are secure, all the time. So, uh, how do we do that?

These devices had trivial vulnerabilities in the form of hardcoded passwords and open telnet. It wouldn't take terribly strong skills to identify this at import time and block a shipment, so the "obvious" answer is to set up forces in customs who do a security analysis of each device. We'll ignore the fact that this would be a pretty huge set of people to keep up with the sheer quantity of crap being developed and skip straight to the explanation for why this wouldn't work.

Yeah, sure, this vulnerability was obvious. But what about the product from a well-known vendor that included a debug app listening on a high numbered UDP port that accepted a packet of the form "BackdoorPacketCmdLine_Req" and then executed the rest of the payload as root? A portscan's not going to show that up[1]. Finding this kind of thing involves pulling the device apart, dumping the firmware and reverse engineering the binaries. It typically takes me about a day to do that. Amazon has over 30,000 listings that match "IP camera" right now, so you're going to need 99 more of me and a year just to examine the cameras. And that's assuming nobody ships any new ones.

Even that's insufficient. Ok, with luck we've identified all the cases where the vendor has left an explicit backdoor in the code[2]. But these devices are still running software that's going to be full of bugs and which is almost certainly still vulnerable to at least half a dozen buffer overflows[3]. Who's going to audit that? All it takes is one attacker to find one flaw in one popular device line, and that's another botnet built.

If we can't stop the vulnerabilities getting into people's homes in the first place, can we at least fix them afterwards? From an economic perspective, demanding that vendors ship security updates whenever a vulnerability is discovered no matter how old the device is is just not going to work. Many of these vendors are small enough that it'd be more cost effective for them to simply fold the company and reopen under a new name than it would be to put the engineering work into fixing a decade old codebase. And how does this actually help? So far the attackers building these networks haven't been terribly competent. The first thing a competent attacker would do would be to silently disable the firmware update mechanism.

We can't easily fix the already broken devices, we can't easily stop more broken devices from being shipped and we can't easily guarantee that we can fix future devices that end up broken. The only solution I see working at all is to require ISPs to cut people off, and that's going to involve a great deal of pain. The harsh reality is that this is almost certainly just the tip of the iceberg, and things are going to get much worse before they get any better.

Right. I'm off to portscan another smart socket.

[1] UDP connection refused messages are typically ratelimited to one per second, so it'll take almost a day to do a full UDP portscan, and even then you have no idea what the service actually does.

[2] It's worth noting that this is usually leftover test or debug code, not an overtly malicious act. Vendors should have processes in place to ensure that this isn't left in release builds, but ha well.

[3] My vacuum cleaner crashes if I send certain malformed HTTP requests to the local API endpoint, which isn't a good sign
Page 1 of 3 << [1] [2] [3] >>

Date: 2016-10-22 05:33 am (UTC)
From: (Anonymous)
Say we wanted to give the "make 99 more of Matthew Garrett" approach a try anyway. How does one learn to dump firmwares and reverse engineer binaries? Can you recommend any learning resources to train up a next generation of hobbyist IoT security auditors?

Is lesson one perhaps "don't live in a country with anti-circumvention laws?"

Cutting of single devices

Date: 2016-10-22 07:17 am (UTC)
From: (Anonymous)
IPv6 would probably make the problem worse, but it would enable ISPs to block single devices close to were they are in the network.

No incoming connections by default

Date: 2016-10-22 09:08 am (UTC)
From: (Anonymous)
You could go half way between doing nothing and cutting off people from the internet.

Given that the first step in infecting vulnerable devices is a port scan, by default ISPs could block all incoming connections until the client explicitly requests that incoming connections should be unblocked.

The catch is I expect ISPs would completely fsck this up and it would be a pain to get things working properly if you know what you are doing.

Date: 2016-10-22 09:21 am (UTC)
brainwane: several colorful scribbles in the vague shape of a jellyfish (jellyfish)
From: [personal profile] brainwane
Matthew will probably have a more substantive answer, but I wouldn't be surprised if this exercise is a good place to start.

Is lesson one perhaps "don't live in a country with anti-circumvention laws?"

Or, repeal them when you find them.

How about open source?

Date: 2016-10-22 09:47 am (UTC)
From: (Anonymous)
Coming from an opensource guy I'm confused why you didn't mention this alternative? Let government require by law that all the firmware and software of these devices has the sources published, and that all of them should have a way of being updated by their users.

My statement follows the logic that if there had been this law imposed much earlier, all these companies would have end up joining forces in trying to write joint-written firmware (so that the code is not maintained by a single company) to de-duplicate efforts and spread the responsability.

Re: How about open source?

Date: 2016-10-22 10:28 am (UTC)
From: (Anonymous)
That doesn't solve anything, since you can't force people to actually update their devices.

Re: How about open source?

Date: 2016-10-22 12:31 pm (UTC)
From: (Anonymous)
They would auto update by default. (A solution for the not yet infected ones, I know, but we have to start somewhere.)

Re: No incoming connections by default

Date: 2016-10-22 12:57 pm (UTC)
From: (Anonymous)
This will stop the most trivial attacks, and then the attackers will shift to hacking into supplier's infrastructure (remember how many devices rely on or provide additional features available via supplier's website).

Re: How about open source?

Date: 2016-10-22 01:01 pm (UTC)
From: (Anonymous)
How does on really verify that the software published matches software installed? The history of GPL enforcement shows that it is hard to get vendors compliant even if there is an active litigator determined to get all the source code.

Now imagine trying to pry the sources from some small China shop where there is no version control and firmware is produced by a guy in a corner who has three directories on his PC with some pieces of firmware built which he hacks on every time a new device is released. It's so much cheaper to dump _something_ then fold the shop and reopen under different name.

Re: How about open source?

Date: 2016-10-22 01:15 pm (UTC)
From: (Anonymous)
> How does on really verify that the software published matches software installed?


> The history of GPL enforcement shows that it is hard to get vendors
> compliant even if there is an active litigator determined to get all the
> source code

I'm sure it would get easier once that litigator happens to be the government/state.

> Now imagine trying to pry the sources from some small China shop where
> there is no version control and firmware is produced by a guy in a corner
> who has three directories on his PC with some pieces of firmware built
> which he hacks on every time a new device is released.

With a law requiring opensource, this practice would at least become illegal.

> It's so much cheaper to dump _something_ then fold the shop and reopen
> under different name.

Then we would be talking about how to improve law enforcement in these cases, but at least we need to make it illegal in the first place.

Home routers

Date: 2016-10-22 01:21 pm (UTC)
From: (Anonymous)
Seems for at least one class of devices, the manufacturer should have to warranty security updates - home routers/firewalls. The OnHub devices make a good example of what to do. And yes, companies that don't should be forced out of business. Now, the next thing to do is to force IoT device makers to create systems where the traffic is outgoing only from the device - so for example you don't connect directly to that new security cam, but instead connect to a cloud server which the security cam also connected to, and only after authenticating with the cloud server are you able to connect to your device. Then when bugs show up, it is fixing up a few cloud servers, not every cheap security cam. Device manufacturers should like this also, they can charge ongoing rent for access to the cloud server.

There's an economic solution to this

Date: 2016-10-22 01:36 pm (UTC)
From: (Anonymous)
Charging user's per MB for their internet service (http://fattuba.com/a-solution-for-ddos/) would fix this problem.

An idea

Date: 2016-10-22 02:03 pm (UTC)
From: (Anonymous)
Maybe it would be possible to combat this by creating proper incentives. However it doesn't need to be all-or-nothing; it would already do good to significantly reduce the number of vulnerable devices.

I haven't thought this through, but how about something like this:

* Forbid commercial import of devices that have been found to be vulnerable (unless the importer undertakes to update their firmware before selling them)
* Forbid the sale of new, already imported devices that have been found to be vulnerable (so, not targeted at consumers selling their used devices)

These I believe could create a few interesting incentives:

* Manufacturers would have an incentive to think of security if they run a risk of not being able to sell them to major markets
* There would be an incentive to find vulnerabilities in competitors' devices and give an anonymous tip
* Commercial importers would have an incentive to try and choose better devices or ones where the manufacturer somehow promises to rectify problems, since they run the risk of being stuck with unsaleable stock

This rule seems a bit harsh in some ways, especially for vendors who really try hard to secure their devices but fail because of, say, some Linux vulnerability. I think it should probably be enough if there are no known vulnerabilities older than, say, 45 days for which there is no firmware update available.

Date: 2016-10-22 02:58 pm (UTC)
From: (Anonymous)
Where have all the grey hats gone? We can develop a worm that "shuts the door" behind itself after it infects a vulnerable device, and "patches" the insecure devices to prevent subsequent attacks and exploits.

Re: Cutting of single devices

Date: 2016-10-22 03:03 pm (UTC)
From: (Anonymous)
Temporary, maybe.
Devices will be directly exposed to internet more often (either by misconfiguration or voluntarily) and they will have a more complex network stack in many cases.
On the other hand, it will be much harder to scan the IPv6 address-space for vulnerable devices and there will be less complex network stack once we have abandoned IPv4 (no more NAT).

I think the benefits outweigh the cons by far with IPv6 since quickly scanning for vulnerable devices is a very strong reason why we see all the crap now. To have simpler protocols and simpler setups is also a good reason to strive for it in the long run as that benefits security as well. Temporary it will cause some issues (dual stacks and misconfigured setups due to lack of knowledge by the user/technicians) but that will quickly change when IPv6 become more common.

Re: There's an economic solution to this

Date: 2016-10-22 03:49 pm (UTC)
From: (Anonymous)
No it wouldn't. The actual amount of traffic from any given device is quite low; the problem is that there are hundreds of thousands of them.

Force internet connected devices to be rented

Date: 2016-10-22 04:52 pm (UTC)
From: (Anonymous)
Perhaps the end game is internet connected devices become leased rather than purchased outright (even your IoT light bulb). Consumers are forced to pay an ongoing fee and ISPs give the devices away to lure customers in. Devices expected to last a long time (e.g. TVs) are not allowed to ship Internet connectivity directly but must do so via USB add ons that can be replaced (vendor locked of course). Only giants (e.g. Google, Amazon) are allowed to ship direct to consumers under penalty that their devices have to be periodically patched and must auto update.

With this, the choice not to update the firmware is ripped out of the customer's hands. ISPs have an incentive not to let their networks participate in attacks and actually ensure updates make their way to customers so they actually pass updates for hardware they provide on to consumers.

A secondary effort is made to ensure everyone is forced to keep changing (e.g. by using a Linux-eseque upgrade all at once system for protocols). Soon old devices struggle to connect to the internet at all due to the never ending technological changes. All the updates burn through a limited amount of flash rewrite cycles thus creating planned obsolescence or forcing the customer to purchase a new USB key with the latest firmware on it. At best you can buy an auto-updated device for "legacy" devices but they're never allowed to connect to the real Internet only some faux house internal one.

Date: 2016-10-22 05:23 pm (UTC)
From: (Anonymous)

launch a counter-attack.
we know where the attacks are coming from, so break into those devices and either lock them or (with a dark-grey hat) brick them.

Re: How about open source?

Date: 2016-10-22 06:05 pm (UTC)
From: (Anonymous)
I'm talking about preventing this problem to happen in the future, not to fight the problem of currently-infected devices.

It's our responsibility

Date: 2016-10-22 07:46 pm (UTC)
From: [identity profile] https://openid-provider.appspot.com/me@jasonclinton.com

I know that this is not a popular view within the computer science and open source communities but: this is our problem, not the vendors'.

We ship open source software used by these vendors with unsafe defaults. We don't default closed or put giant PEOPLE_WILL_DIE_IF_YOU_DO_THIS text on our configuration variables. We ship compilers and languages that don't put safety and buffer overflow protection first. Hell, we don't even systematically regression test (let alone fuzz test) the software that our communities produce.

We ship crap code and we expect downmarket vendors to polish it.

We have to fix the problem; no one else will.

There are glimmers of hope: there's healthy projects like boringssl. There's basic language safety efforts like Rust. There's healthy discourse about handling security better within the Linux kernel community. And if that doesn't work out, there's long-shots efforts like user-mode drivers in Magenta.

The first place that any open source contributor can begin is to hold ourselves to a higher standard:

  • write units tests,
  • try to find a way to run continuous regression tests on your code,
  • think about the security costs of backward-incompatible API changes,
  • ask for help fuzz testing binaries,
  • and, above all, think about the engineer who will recycle your library under deadline pressure.

maybe we should build better platform ?

Date: 2016-10-22 09:18 pm (UTC)
From: (Anonymous)
Maybe we should:

1. Design a secure architecture - on that the vendor cannot mess up, and all updates are from providers we can count on.

2. Somewhat forcefully shove it in chips. silicon is cheap.

3. Make sure it's extremely cheap on the one hand, and on the other, find some carrot or stick to make sure only/mostly "good" devices enter the market.

Re: An idea

Date: 2016-10-22 10:16 pm (UTC)
From: (Anonymous)
Have you seen Kees Cook's data on Linux kernel security flaw lifetimes? Your proposed demands would effectively prevent selling any Linux-based hardware, and would effectively promote closed-source security-by-obscurity (if it hasn't been found vulnerable yet, it's OK to sell).

Something more realistic would be to provide an easy-to-use, secure by default platform that chip vendors could base their BSPs on. A lot of the cheap products base their software more or less directly on the vendor's BSP, and those are frequently some crap thrown together just to get a booting system. It's not unusual to see BSPs still based on some 2.4 kernel, and never updated. Something like Yocto just has a too steep learning curve, and doesn't solve the problem of vendors sticking to outdated versions.

Date: 2016-10-23 12:06 am (UTC)
From: (Anonymous)
All of the DVR-type appliances I've ever examined have been one of two "ODM" boards and software stacks, minimally rebranded - consequently, I would be willing to bet that it's not quite as bad of a hydra to get the common manufacturers to fix this in newer devices as it seems.

No experience with actual IP cameras (versus dumb cameras that only knew how to communicate over closed circuit to a DVR), but I would expect the DVR case to be the better bang/buck (there are probably more distinct internet connections with just one or two IP cameras instead of something managing them, but even if they're mostly the same stock board, there's probably many more vendors to convince to update their shit).
Page 1 of 3 << [1] [2] [3] >>


Matthew Garrett

About Matthew

Power management, mobile and firmware developer on Linux. Security developer at Google. Member of the Free Software Foundation board of directors. Ex-biologist. @mjg59 on Twitter. Content here should not be interpreted as the opinion of my employer.

Expand Cut Tags

No cut tags