Fixing the IoT isn't going to be easy
Oct. 21st, 2016 11:35 pm![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
mjg59A large part of the internet became inaccessible today after a botnet made up of IP cameras and digital video recorders was used to DoS a major DNS provider. This highlighted a bunch of things including how maybe having all your DNS handled by a single provider is not the best of plans, but in the long run there's no real amount of diversification that can fix this - malicious actors have control of a sufficiently large number of hosts that they could easily take out multiple providers simultaneously.
To fix this properly we need to get rid of the compromised systems. The question is how. Many of these devices are sold by resellers who have no resources to handle any kind of recall. The manufacturer may not have any kind of legal presence in many of the countries where their products are sold. There's no way anybody can compel a recall, and even if they could it probably wouldn't help. If I've paid a contractor to install a security camera in my office, and if I get a notification that my camera is being used to take down Twitter, what do I do? Pay someone to come and take the camera down again, wait for a fixed one and pay to get that put up? That's probably not going to happen. As long as the device carries on working, many users are going to ignore any voluntary request.
We're left with more aggressive remedies. If ISPs threaten to cut off customers who host compromised devices, we might get somewhere. But, inevitably, a number of small businesses and unskilled users will get cut off. Probably a large number. The economic damage is still going to be significant. And it doesn't necessarily help that much - if the US were to compel ISPs to do this, but nobody else did, public outcry would be massive, the botnet would not be much smaller and the attacks would continue. Do we start cutting off countries that fail to police their internet?
Ok, so maybe we just chalk this one up as a loss and have everyone build out enough infrastructure that we're able to withstand attacks from this botnet and take steps to ensure that nobody is ever able to build a bigger one. To do that, we'd need to ensure that all IoT devices are secure, all the time. So, uh, how do we do that?
These devices had trivial vulnerabilities in the form of hardcoded passwords and open telnet. It wouldn't take terribly strong skills to identify this at import time and block a shipment, so the "obvious" answer is to set up forces in customs who do a security analysis of each device. We'll ignore the fact that this would be a pretty huge set of people to keep up with the sheer quantity of crap being developed and skip straight to the explanation for why this wouldn't work.
Yeah, sure, this vulnerability was obvious. But what about the product from a well-known vendor that included a debug app listening on a high numbered UDP port that accepted a packet of the form "BackdoorPacketCmdLine_Req" and then executed the rest of the payload as root? A portscan's not going to show that up[1]. Finding this kind of thing involves pulling the device apart, dumping the firmware and reverse engineering the binaries. It typically takes me about a day to do that. Amazon has over 30,000 listings that match "IP camera" right now, so you're going to need 99 more of me and a year just to examine the cameras. And that's assuming nobody ships any new ones.
Even that's insufficient. Ok, with luck we've identified all the cases where the vendor has left an explicit backdoor in the code[2]. But these devices are still running software that's going to be full of bugs and which is almost certainly still vulnerable to at least half a dozen buffer overflows[3]. Who's going to audit that? All it takes is one attacker to find one flaw in one popular device line, and that's another botnet built.
If we can't stop the vulnerabilities getting into people's homes in the first place, can we at least fix them afterwards? From an economic perspective, demanding that vendors ship security updates whenever a vulnerability is discovered no matter how old the device is is just not going to work. Many of these vendors are small enough that it'd be more cost effective for them to simply fold the company and reopen under a new name than it would be to put the engineering work into fixing a decade old codebase. And how does this actually help? So far the attackers building these networks haven't been terribly competent. The first thing a competent attacker would do would be to silently disable the firmware update mechanism.
We can't easily fix the already broken devices, we can't easily stop more broken devices from being shipped and we can't easily guarantee that we can fix future devices that end up broken. The only solution I see working at all is to require ISPs to cut people off, and that's going to involve a great deal of pain. The harsh reality is that this is almost certainly just the tip of the iceberg, and things are going to get much worse before they get any better.
Right. I'm off to portscan another smart socket.
[1] UDP connection refused messages are typically ratelimited to one per second, so it'll take almost a day to do a full UDP portscan, and even then you have no idea what the service actually does.
[2] It's worth noting that this is usually leftover test or debug code, not an overtly malicious act. Vendors should have processes in place to ensure that this isn't left in release builds, but ha well.
[3] My vacuum cleaner crashes if I send certain malformed HTTP requests to the local API endpoint, which isn't a good sign
To fix this properly we need to get rid of the compromised systems. The question is how. Many of these devices are sold by resellers who have no resources to handle any kind of recall. The manufacturer may not have any kind of legal presence in many of the countries where their products are sold. There's no way anybody can compel a recall, and even if they could it probably wouldn't help. If I've paid a contractor to install a security camera in my office, and if I get a notification that my camera is being used to take down Twitter, what do I do? Pay someone to come and take the camera down again, wait for a fixed one and pay to get that put up? That's probably not going to happen. As long as the device carries on working, many users are going to ignore any voluntary request.
We're left with more aggressive remedies. If ISPs threaten to cut off customers who host compromised devices, we might get somewhere. But, inevitably, a number of small businesses and unskilled users will get cut off. Probably a large number. The economic damage is still going to be significant. And it doesn't necessarily help that much - if the US were to compel ISPs to do this, but nobody else did, public outcry would be massive, the botnet would not be much smaller and the attacks would continue. Do we start cutting off countries that fail to police their internet?
Ok, so maybe we just chalk this one up as a loss and have everyone build out enough infrastructure that we're able to withstand attacks from this botnet and take steps to ensure that nobody is ever able to build a bigger one. To do that, we'd need to ensure that all IoT devices are secure, all the time. So, uh, how do we do that?
These devices had trivial vulnerabilities in the form of hardcoded passwords and open telnet. It wouldn't take terribly strong skills to identify this at import time and block a shipment, so the "obvious" answer is to set up forces in customs who do a security analysis of each device. We'll ignore the fact that this would be a pretty huge set of people to keep up with the sheer quantity of crap being developed and skip straight to the explanation for why this wouldn't work.
Yeah, sure, this vulnerability was obvious. But what about the product from a well-known vendor that included a debug app listening on a high numbered UDP port that accepted a packet of the form "BackdoorPacketCmdLine_Req" and then executed the rest of the payload as root? A portscan's not going to show that up[1]. Finding this kind of thing involves pulling the device apart, dumping the firmware and reverse engineering the binaries. It typically takes me about a day to do that. Amazon has over 30,000 listings that match "IP camera" right now, so you're going to need 99 more of me and a year just to examine the cameras. And that's assuming nobody ships any new ones.
Even that's insufficient. Ok, with luck we've identified all the cases where the vendor has left an explicit backdoor in the code[2]. But these devices are still running software that's going to be full of bugs and which is almost certainly still vulnerable to at least half a dozen buffer overflows[3]. Who's going to audit that? All it takes is one attacker to find one flaw in one popular device line, and that's another botnet built.
If we can't stop the vulnerabilities getting into people's homes in the first place, can we at least fix them afterwards? From an economic perspective, demanding that vendors ship security updates whenever a vulnerability is discovered no matter how old the device is is just not going to work. Many of these vendors are small enough that it'd be more cost effective for them to simply fold the company and reopen under a new name than it would be to put the engineering work into fixing a decade old codebase. And how does this actually help? So far the attackers building these networks haven't been terribly competent. The first thing a competent attacker would do would be to silently disable the firmware update mechanism.
We can't easily fix the already broken devices, we can't easily stop more broken devices from being shipped and we can't easily guarantee that we can fix future devices that end up broken. The only solution I see working at all is to require ISPs to cut people off, and that's going to involve a great deal of pain. The harsh reality is that this is almost certainly just the tip of the iceberg, and things are going to get much worse before they get any better.
Right. I'm off to portscan another smart socket.
[1] UDP connection refused messages are typically ratelimited to one per second, so it'll take almost a day to do a full UDP portscan, and even then you have no idea what the service actually does.
[2] It's worth noting that this is usually leftover test or debug code, not an overtly malicious act. Vendors should have processes in place to ensure that this isn't left in release builds, but ha well.
[3] My vacuum cleaner crashes if I send certain malformed HTTP requests to the local API endpoint, which isn't a good sign
![[identity profile]](https://www.dreamwidth.org/img/silk/identity/openid.png){kind=small}
no subject
Date: 2016-10-22 05:33 am (UTC)Is lesson one perhaps "don't live in a country with anti-circumvention laws?"
Cutting of single devices
Date: 2016-10-22 07:17 am (UTC)No incoming connections by default
Date: 2016-10-22 09:08 am (UTC)Given that the first step in infecting vulnerable devices is a port scan, by default ISPs could block all incoming connections until the client explicitly requests that incoming connections should be unblocked.
The catch is I expect ISPs would completely fsck this up and it would be a pain to get things working properly if you know what you are doing.
no subject
Date: 2016-10-22 09:21 am (UTC)Or, repeal them when you find them.
How about open source?
Date: 2016-10-22 09:47 am (UTC)My statement follows the logic that if there had been this law imposed much earlier, all these companies would have end up joining forces in trying to write joint-written firmware (so that the code is not maintained by a single company) to de-duplicate efforts and spread the responsability.
Re: How about open source?
Date: 2016-10-22 10:28 am (UTC)Re: How about open source?
Date: 2016-10-22 12:31 pm (UTC)Re: No incoming connections by default
Date: 2016-10-22 12:57 pm (UTC)Re: How about open source?
Date: 2016-10-22 01:01 pm (UTC)Now imagine trying to pry the sources from some small China shop where there is no version control and firmware is produced by a guy in a corner who has three directories on his PC with some pieces of firmware built which he hacks on every time a new device is released. It's so much cheaper to dump _something_ then fold the shop and reopen under different name.
Re: How about open source?
Date: 2016-10-22 01:15 pm (UTC)https://gitian.org/
> The history of GPL enforcement shows that it is hard to get vendors
> compliant even if there is an active litigator determined to get all the
> source code
I'm sure it would get easier once that litigator happens to be the government/state.
> Now imagine trying to pry the sources from some small China shop where
> there is no version control and firmware is produced by a guy in a corner
> who has three directories on his PC with some pieces of firmware built
> which he hacks on every time a new device is released.
With a law requiring opensource, this practice would at least become illegal.
> It's so much cheaper to dump _something_ then fold the shop and reopen
> under different name.
Then we would be talking about how to improve law enforcement in these cases, but at least we need to make it illegal in the first place.
Home routers
Date: 2016-10-22 01:21 pm (UTC)There's an economic solution to this
Date: 2016-10-22 01:36 pm (UTC)An idea
Date: 2016-10-22 02:03 pm (UTC)I haven't thought this through, but how about something like this:
* Forbid commercial import of devices that have been found to be vulnerable (unless the importer undertakes to update their firmware before selling them)
* Forbid the sale of new, already imported devices that have been found to be vulnerable (so, not targeted at consumers selling their used devices)
These I believe could create a few interesting incentives:
* Manufacturers would have an incentive to think of security if they run a risk of not being able to sell them to major markets
* There would be an incentive to find vulnerabilities in competitors' devices and give an anonymous tip
* Commercial importers would have an incentive to try and choose better devices or ones where the manufacturer somehow promises to rectify problems, since they run the risk of being stuck with unsaleable stock
This rule seems a bit harsh in some ways, especially for vendors who really try hard to secure their devices but fail because of, say, some Linux vulnerability. I think it should probably be enough if there are no known vulnerabilities older than, say, 45 days for which there is no firmware update available.
no subject
Date: 2016-10-22 02:58 pm (UTC)Re: Cutting of single devices
Date: 2016-10-22 03:03 pm (UTC)Devices will be directly exposed to internet more often (either by misconfiguration or voluntarily) and they will have a more complex network stack in many cases.
On the other hand, it will be much harder to scan the IPv6 address-space for vulnerable devices and there will be less complex network stack once we have abandoned IPv4 (no more NAT).
I think the benefits outweigh the cons by far with IPv6 since quickly scanning for vulnerable devices is a very strong reason why we see all the crap now. To have simpler protocols and simpler setups is also a good reason to strive for it in the long run as that benefits security as well. Temporary it will cause some issues (dual stacks and misconfigured setups due to lack of knowledge by the user/technicians) but that will quickly change when IPv6 become more common.
Re: How about open source?
Date: 2016-10-22 03:04 pm (UTC)Re: There's an economic solution to this
Date: 2016-10-22 03:49 pm (UTC)Force internet connected devices to be rented
Date: 2016-10-22 04:52 pm (UTC)With this, the choice not to update the firmware is ripped out of the customer's hands. ISPs have an incentive not to let their networks participate in attacks and actually ensure updates make their way to customers so they actually pass updates for hardware they provide on to consumers.
A secondary effort is made to ensure everyone is forced to keep changing (e.g. by using a Linux-eseque upgrade all at once system for protocols). Soon old devices struggle to connect to the internet at all due to the never ending technological changes. All the updates burn through a limited amount of flash rewrite cycles thus creating planned obsolescence or forcing the customer to purchase a new USB key with the latest firmware on it. At best you can buy an auto-updated device for "legacy" devices but they're never allowed to connect to the real Internet only some faux house internal one.
no subject
Date: 2016-10-22 05:23 pm (UTC)launch a counter-attack.
we know where the attacks are coming from, so break into those devices and either lock them or (with a dark-grey hat) brick them.
Re: How about open source?
Date: 2016-10-22 06:05 pm (UTC)Re: How about open source?
Date: 2016-10-22 06:11 pm (UTC)It's our responsibility
Date: 2016-10-22 07:46 pm (UTC)I know that this is not a popular view within the computer science and open source communities but: this is our problem, not the vendors'.
We ship open source software used by these vendors with unsafe defaults. We don't default closed or put giant PEOPLE_WILL_DIE_IF_YOU_DO_THIS text on our configuration variables. We ship compilers and languages that don't put safety and buffer overflow protection first. Hell, we don't even systematically regression test (let alone fuzz test) the software that our communities produce.
We ship crap code and we expect downmarket vendors to polish it.
We have to fix the problem; no one else will.
There are glimmers of hope: there's healthy projects like boringssl. There's basic language safety efforts like Rust. There's healthy discourse about handling security better within the Linux kernel community. And if that doesn't work out, there's long-shots efforts like user-mode drivers in Magenta.
The first place that any open source contributor can begin is to hold ourselves to a higher standard:
maybe we should build better platform ?
Date: 2016-10-22 09:18 pm (UTC)1. Design a secure architecture - on that the vendor cannot mess up, and all updates are from providers we can count on.
2. Somewhat forcefully shove it in chips. silicon is cheap.
3. Make sure it's extremely cheap on the one hand, and on the other, find some carrot or stick to make sure only/mostly "good" devices enter the market.
Re: An idea
Date: 2016-10-22 10:16 pm (UTC)Something more realistic would be to provide an easy-to-use, secure by default platform that chip vendors could base their BSPs on. A lot of the cheap products base their software more or less directly on the vendor's BSP, and those are frequently some crap thrown together just to get a booting system. It's not unusual to see BSPs still based on some 2.4 kernel, and never updated. Something like Yocto just has a too steep learning curve, and doesn't solve the problem of vendors sticking to outdated versions.
no subject
Date: 2016-10-23 12:06 am (UTC)No experience with actual IP cameras (versus dumb cameras that only knew how to communicate over closed circuit to a DVR), but I would expect the DVR case to be the better bang/buck (there are probably more distinct internet connections with just one or two IP cameras instead of something managing them, but even if they're mostly the same stock board, there's probably many more vendors to convince to update their shit).