[personal profile] mjg59
A large part of the internet became inaccessible today after a botnet made up of IP cameras and digital video recorders was used to DoS a major DNS provider. This highlighted a bunch of things including how maybe having all your DNS handled by a single provider is not the best of plans, but in the long run there's no real amount of diversification that can fix this - malicious actors have control of a sufficiently large number of hosts that they could easily take out multiple providers simultaneously.

To fix this properly we need to get rid of the compromised systems. The question is how. Many of these devices are sold by resellers who have no resources to handle any kind of recall. The manufacturer may not have any kind of legal presence in many of the countries where their products are sold. There's no way anybody can compel a recall, and even if they could it probably wouldn't help. If I've paid a contractor to install a security camera in my office, and if I get a notification that my camera is being used to take down Twitter, what do I do? Pay someone to come and take the camera down again, wait for a fixed one and pay to get that put up? That's probably not going to happen. As long as the device carries on working, many users are going to ignore any voluntary request.

We're left with more aggressive remedies. If ISPs threaten to cut off customers who host compromised devices, we might get somewhere. But, inevitably, a number of small businesses and unskilled users will get cut off. Probably a large number. The economic damage is still going to be significant. And it doesn't necessarily help that much - if the US were to compel ISPs to do this, but nobody else did, public outcry would be massive, the botnet would not be much smaller and the attacks would continue. Do we start cutting off countries that fail to police their internet?

Ok, so maybe we just chalk this one up as a loss and have everyone build out enough infrastructure that we're able to withstand attacks from this botnet and take steps to ensure that nobody is ever able to build a bigger one. To do that, we'd need to ensure that all IoT devices are secure, all the time. So, uh, how do we do that?

These devices had trivial vulnerabilities in the form of hardcoded passwords and open telnet. It wouldn't take terribly strong skills to identify this at import time and block a shipment, so the "obvious" answer is to set up forces in customs who do a security analysis of each device. We'll ignore the fact that this would be a pretty huge set of people to keep up with the sheer quantity of crap being developed and skip straight to the explanation for why this wouldn't work.

Yeah, sure, this vulnerability was obvious. But what about the product from a well-known vendor that included a debug app listening on a high numbered UDP port that accepted a packet of the form "BackdoorPacketCmdLine_Req" and then executed the rest of the payload as root? A portscan's not going to show that up[1]. Finding this kind of thing involves pulling the device apart, dumping the firmware and reverse engineering the binaries. It typically takes me about a day to do that. Amazon has over 30,000 listings that match "IP camera" right now, so you're going to need 99 more of me and a year just to examine the cameras. And that's assuming nobody ships any new ones.

Even that's insufficient. Ok, with luck we've identified all the cases where the vendor has left an explicit backdoor in the code[2]. But these devices are still running software that's going to be full of bugs and which is almost certainly still vulnerable to at least half a dozen buffer overflows[3]. Who's going to audit that? All it takes is one attacker to find one flaw in one popular device line, and that's another botnet built.

If we can't stop the vulnerabilities getting into people's homes in the first place, can we at least fix them afterwards? From an economic perspective, demanding that vendors ship security updates whenever a vulnerability is discovered no matter how old the device is is just not going to work. Many of these vendors are small enough that it'd be more cost effective for them to simply fold the company and reopen under a new name than it would be to put the engineering work into fixing a decade old codebase. And how does this actually help? So far the attackers building these networks haven't been terribly competent. The first thing a competent attacker would do would be to silently disable the firmware update mechanism.

We can't easily fix the already broken devices, we can't easily stop more broken devices from being shipped and we can't easily guarantee that we can fix future devices that end up broken. The only solution I see working at all is to require ISPs to cut people off, and that's going to involve a great deal of pain. The harsh reality is that this is almost certainly just the tip of the iceberg, and things are going to get much worse before they get any better.

Right. I'm off to portscan another smart socket.

[1] UDP connection refused messages are typically ratelimited to one per second, so it'll take almost a day to do a full UDP portscan, and even then you have no idea what the service actually does.

[2] It's worth noting that this is usually leftover test or debug code, not an overtly malicious act. Vendors should have processes in place to ensure that this isn't left in release builds, but ha well.

[3] My vacuum cleaner crashes if I send certain malformed HTTP requests to the local API endpoint, which isn't a good sign

Force internet connected devices to be rented

Date: 2016-10-22 04:52 pm (UTC)
From: (Anonymous)
Perhaps the end game is internet connected devices become leased rather than purchased outright (even your IoT light bulb). Consumers are forced to pay an ongoing fee and ISPs give the devices away to lure customers in. Devices expected to last a long time (e.g. TVs) are not allowed to ship Internet connectivity directly but must do so via USB add ons that can be replaced (vendor locked of course). Only giants (e.g. Google, Amazon) are allowed to ship direct to consumers under penalty that their devices have to be periodically patched and must auto update.

With this, the choice not to update the firmware is ripped out of the customer's hands. ISPs have an incentive not to let their networks participate in attacks and actually ensure updates make their way to customers so they actually pass updates for hardware they provide on to consumers.

A secondary effort is made to ensure everyone is forced to keep changing (e.g. by using a Linux-eseque upgrade all at once system for protocols). Soon old devices struggle to connect to the internet at all due to the never ending technological changes. All the updates burn through a limited amount of flash rewrite cycles thus creating planned obsolescence or forcing the customer to purchase a new USB key with the latest firmware on it. At best you can buy an auto-updated device for "legacy" devices but they're never allowed to connect to the real Internet only some faux house internal one.
From: (Anonymous)
"Devices expected to last a long time (e.g. TVs) are not allowed to ship Internet connectivity directly but must do so via USB add ons"

I think there is some confusion here. A TV does not need to be hosted (listening) publicly on the Internet. AFAIK, all Internet-enabled TVs (and devices of a similar ilk), simply make outbound connections and any inbound connections would be blocked by the home router by default.

The only devices that need to be hosted publicly on the Internet (ie. with a listening socket) are those that you need to connect to from outside your home network. So, IP security cameras, etc. Not home devices such as TV's or lightbulbs.

Ofc, I could have the wrong end of the stick here. Apologies if so :-)
From: (Anonymous)
> The only devices that need to be hosted publicly on the Internet (ie. with a listening socket) are those that you need to connect to from outside your home network. So, IP security cameras, etc. Not home devices such as TV's or lightbulbs.

I have a wifi gateway for some lightbulbs which connects out to a service and keeps that connection permanently open. This is to let you control your lights from outside your home (lets not ask why), but also gives a back-channel into your home network which the service could be compromised to use. It's an extra step, but a good target for attackers if they can get into many home networks at once.

(This is blocked from talking to the Internet and is on a separate IoT network in my case, because I investigated what it was doing and did't want that happening...)

So unfortunately
From: (Anonymous)
Ooops, didn't finish that before I hit submit, somehow...

So unfortunately it's not just things which explicitly listen externally which can be at risk.
From: (Anonymous)
Interesting. I would have thought you would be safe in that sense because the connection from the wifi gateway to the service would be outgoing connection only. Unless the wifi gateway actually listened on a port (and was forwarded from the firewall/router) then there would be no back-channel into your home wifi. At least thats my (limited) understanding.


Matthew Garrett

About Matthew

Power management, mobile and firmware developer on Linux. Security developer at Google. Ex-biologist. @mjg59 on Twitter. Content here should not be interpreted as the opinion of my employer.

Expand Cut Tags

No cut tags