mjg59 | Fixing the IoT isn't going to be easy

Entry tags:

advogato,
fedora

Fixing the IoT isn't going to be easy

A large part of the internet became inaccessible today after a botnet made up of IP cameras and digital video recorders was used to DoS a major DNS provider. This highlighted a bunch of things including how maybe having all your DNS handled by a single provider is not the best of plans, but in the long run there's no real amount of diversification that can fix this - malicious actors have control of a sufficiently large number of hosts that they could easily take out multiple providers simultaneously.

To fix this properly we need to get rid of the compromised systems. The question is how. Many of these devices are sold by resellers who have no resources to handle any kind of recall. The manufacturer may not have any kind of legal presence in many of the countries where their products are sold. There's no way anybody can compel a recall, and even if they could it probably wouldn't help. If I've paid a contractor to install a security camera in my office, and if I get a notification that my camera is being used to take down Twitter, what do I do? Pay someone to come and take the camera down again, wait for a fixed one and pay to get that put up? That's probably not going to happen. As long as the device carries on working, many users are going to ignore any voluntary request.

We're left with more aggressive remedies. If ISPs threaten to cut off customers who host compromised devices, we might get somewhere. But, inevitably, a number of small businesses and unskilled users will get cut off. Probably a large number. The economic damage is still going to be significant. And it doesn't necessarily help that much - if the US were to compel ISPs to do this, but nobody else did, public outcry would be massive, the botnet would not be much smaller and the attacks would continue. Do we start cutting off countries that fail to police their internet?

Ok, so maybe we just chalk this one up as a loss and have everyone build out enough infrastructure that we're able to withstand attacks from this botnet and take steps to ensure that nobody is ever able to build a bigger one. To do that, we'd need to ensure that all IoT devices are secure, all the time. So, uh, how do we do that?

These devices had trivial vulnerabilities in the form of hardcoded passwords and open telnet. It wouldn't take terribly strong skills to identify this at import time and block a shipment, so the "obvious" answer is to set up forces in customs who do a security analysis of each device. We'll ignore the fact that this would be a pretty huge set of people to keep up with the sheer quantity of crap being developed and skip straight to the explanation for why this wouldn't work.

Yeah, sure, this vulnerability was obvious. But what about the product from a well-known vendor that included a debug app listening on a high numbered UDP port that accepted a packet of the form "BackdoorPacketCmdLine_Req" and then executed the rest of the payload as root? A portscan's not going to show that up[1]. Finding this kind of thing involves pulling the device apart, dumping the firmware and reverse engineering the binaries. It typically takes me about a day to do that. Amazon has over 30,000 listings that match "IP camera" right now, so you're going to need 99 more of me and a year just to examine the cameras. And that's assuming nobody ships any new ones.

Even that's insufficient. Ok, with luck we've identified all the cases where the vendor has left an explicit backdoor in the code[2]. But these devices are still running software that's going to be full of bugs and which is almost certainly still vulnerable to at least half a dozen buffer overflows[3]. Who's going to audit that? All it takes is one attacker to find one flaw in one popular device line, and that's another botnet built.

If we can't stop the vulnerabilities getting into people's homes in the first place, can we at least fix them afterwards? From an economic perspective, demanding that vendors ship security updates whenever a vulnerability is discovered no matter how old the device is is just not going to work. Many of these vendors are small enough that it'd be more cost effective for them to simply fold the company and reopen under a new name than it would be to put the engineering work into fixing a decade old codebase. And how does this actually help? So far the attackers building these networks haven't been terribly competent. The first thing a competent attacker would do would be to silently disable the firmware update mechanism.

We can't easily fix the already broken devices, we can't easily stop more broken devices from being shipped and we can't easily guarantee that we can fix future devices that end up broken. The only solution I see working at all is to require ISPs to cut people off, and that's going to involve a great deal of pain. The harsh reality is that this is almost certainly just the tip of the iceberg, and things are going to get much worse before they get any better.

Right. I'm off to portscan another smart socket.

[1] UDP connection refused messages are typically ratelimited to one per second, so it'll take almost a day to do a full UDP portscan, and even then you have no idea what the service actually does.

[2] It's worth noting that this is usually leftover test or debug code, not an overtly malicious act. Vendors should have processes in place to ensure that this isn't left in release builds, but ha well.

[3] My vacuum cleaner crashes if I send certain malformed HTTP requests to the local API endpoint, which isn't a good sign

Threaded | Flat

Say we wanted to give the "make 99 more of Matthew Garrett" approach a try anyway. How does one learn to dump firmwares and reverse engineer binaries? Can you recommend any learning resources to train up a next generation of hobbyist IoT security auditors?

Is lesson one perhaps "don't live in a country with anti-circumvention laws?"

IPv6 would probably make the problem worse, but it would enable ISPs to block single devices close to were they are in the network.

You could go half way between doing nothing and cutting off people from the internet.

Given that the first step in infecting vulnerable devices is a port scan, by default ISPs could block all incoming connections until the client explicitly requests that incoming connections should be unblocked.

The catch is I expect ISPs would completely fsck this up and it would be a pain to get things working properly if you know what you are doing.

Coming from an opensource guy I'm confused why you didn't mention this alternative? Let government require by law that all the firmware and software of these devices has the sources published, and that all of them should have a way of being updated by their users.

My statement follows the logic that if there had been this law imposed much earlier, all these companies would have end up joining forces in trying to write joint-written firmware (so that the code is not maintained by a single company) to de-duplicate efforts and spread the responsability.

Seems for at least one class of devices, the manufacturer should have to warranty security updates - home routers/firewalls. The OnHub devices make a good example of what to do. And yes, companies that don't should be forced out of business. Now, the next thing to do is to force IoT device makers to create systems where the traffic is outgoing only from the device - so for example you don't connect directly to that new security cam, but instead connect to a cloud server which the security cam also connected to, and only after authenticating with the cloud server are you able to connect to your device. Then when bugs show up, it is fixing up a few cloud servers, not every cheap security cam. Device manufacturers should like this also, they can charge ongoing rent for access to the cloud server.

Charging user's per MB for their internet service (http://fattuba.com/a-solution-for-ddos/) would fix this problem.

Maybe it would be possible to combat this by creating proper incentives. However it doesn't need to be all-or-nothing; it would already do good to significantly reduce the number of vulnerable devices.

I haven't thought this through, but how about something like this:

* Forbid commercial import of devices that have been found to be vulnerable (unless the importer undertakes to update their firmware before selling them)
* Forbid the sale of new, already imported devices that have been found to be vulnerable (so, not targeted at consumers selling their used devices)

These I believe could create a few interesting incentives:

* Manufacturers would have an incentive to think of security if they run a risk of not being able to sell them to major markets
* There would be an incentive to find vulnerabilities in competitors' devices and give an anonymous tip
* Commercial importers would have an incentive to try and choose better devices or ones where the manufacturer somehow promises to rectify problems, since they run the risk of being stuck with unsaleable stock

This rule seems a bit harsh in some ways, especially for vendors who really try hard to secure their devices but fail because of, say, some Linux vulnerability. I think it should probably be enough if there are no known vulnerabilities older than, say, 45 days for which there is no firmware update available.

Where have all the grey hats gone? We can develop a worm that "shuts the door" behind itself after it infects a vulnerable device, and "patches" the insecure devices to prevent subsequent attacks and exploits.

Perhaps the end game is internet connected devices become leased rather than purchased outright (even your IoT light bulb). Consumers are forced to pay an ongoing fee and ISPs give the devices away to lure customers in. Devices expected to last a long time (e.g. TVs) are not allowed to ship Internet connectivity directly but must do so via USB add ons that can be replaced (vendor locked of course). Only giants (e.g. Google, Amazon) are allowed to ship direct to consumers under penalty that their devices have to be periodically patched and must auto update.

With this, the choice not to update the firmware is ripped out of the customer's hands. ISPs have an incentive not to let their networks participate in attacks and actually ensure updates make their way to customers so they actually pass updates for hardware they provide on to consumers.

A secondary effort is made to ensure everyone is forced to keep changing (e.g. by using a Linux-eseque upgrade all at once system for protocols). Soon old devices struggle to connect to the internet at all due to the never ending technological changes. All the updates burn through a limited amount of flash rewrite cycles thus creating planned obsolescence or forcing the customer to purchase a new USB key with the latest firmware on it. At best you can buy an auto-updated device for "legacy" devices but they're never allowed to connect to the real Internet only some faux house internal one.

I know that this is not a popular view within the computer science and open source communities but: this is our problem, not the vendors'.

We ship open source software used by these vendors with unsafe defaults. We don't default closed or put giant PEOPLE_WILL_DIE_IF_YOU_DO_THIS text on our configuration variables. We ship compilers and languages that don't put safety and buffer overflow protection first. Hell, we don't even systematically regression test (let alone fuzz test) the software that our communities produce.

We ship crap code and we expect downmarket vendors to polish it.

We have to fix the problem; no one else will.

There are glimmers of hope: there's healthy projects like boringssl. There's basic language safety efforts like Rust. There's healthy discourse about handling security better within the Linux kernel community. And if that doesn't work out, there's long-shots efforts like user-mode drivers in Magenta.

The first place that any open source contributor can begin is to hold ourselves to a higher standard:

write units tests,
try to find a way to run continuous regression tests on your code,
think about the security costs of backward-incompatible API changes,
ask for help fuzz testing binaries,
and, above all, think about the engineer who will recycle your library under deadline pressure.

Maybe we should:

1. Design a secure architecture - on that the vendor cannot mess up, and all updates are from providers we can count on.

2. Somewhat forcefully shove it in chips. silicon is cheap.

3. Make sure it's extremely cheap on the one hand, and on the other, find some carrot or stick to make sure only/mostly "good" devices enter the market.

All of the DVR-type appliances I've ever examined have been one of two "ODM" boards and software stacks, minimally rebranded - consequently, I would be willing to bet that it's not quite as bad of a hydra to get the common manufacturers to fix this in newer devices as it seems.

No experience with actual IP cameras (versus dumb cameras that only knew how to communicate over closed circuit to a DVR), but I would expect the DVR case to be the better bang/buck (there are probably more distinct internet connections with just one or two IP cameras instead of something managing them, but even if they're mostly the same stock board, there's probably many more vendors to convince to update their shit).

Really well written article. Ty, Matthew.

I was surprised that the solutions section didn't mention alternate Internet infrastructure that are more resilient. The first one I think of is the Interplanetary File System.

The answer is the same as it is for things plugged into the electricity sockets in your home. At least in Australia they are required by law to meet certain standards before you can legally plug them in:

https://ablis.business.gov.au/VIC/pages/1e057cce-d835-4a4b-91b7-d733fda05ec4.aspx

Hobbyists are allowed to do some stuff themselves, but not allowed to sell it.

Likewise, home routers could be legally required to meet certain standards before being allowed to be plugged in, and that could include having a separate network for iOT devices in which they are required to pre-register hosts they wish to communicate with and be approved before they could be used.

And use the existing legal framework that we have here and probably in most countries for fines and recalls of faulty equipment which doesn't meet the standards, and fines for hobbyists who build their own without it being up to scratch.

As for countries which don't enforce protections against DDoS. Yes, the only solution is to cut them off or rate limit their egress. The current situation where a single actor can, with complete anonymity, take anyone but a handful of giant organisations entirely off the air is broken. When they get powerful enough to take all the DNS roots out at the same time, we're really screwed.

> If we can't stop the vulnerabilities getting into people's homes in the first place, can we at least fix them afterwards?

Yes. You do this by not-having anything for which the software _cannot_ be maintained. It doesn't necessarily have to _be_ maintained or have updates be available from any particular party (e.g. the vendor or manufacturer). There only needs to be a total and complete lack of barriers which might _prevent_ any interested party from performing the software maintenance. Then, if people want the maintenance enough, someone will do it. (Maybe for money, maybe not.)

If barriers to maintenance exist, you use government force to remove the barrier. Nobody can make a case for lack of software maintenance being a desirable, or even tolerable, thing.

there are certainly a number of people willing to know and take down their hacked devices themselves. Looks like the only thing missing is a notification system from your ISP, looks like an easy solution that will help at least partially.

int_ua

One simple thing that will change this in a big way, is to always have IoT devices act a a client instead of a server when being accessed remotely. They talk to a broker/server and then that server manages access control. Most of the latest IoT devices that I have seen adhere to this architectural model. Yes, things become a bit difficult for devices with multimedia, but that should also be possible to accomplish.
No mucking around, opening ports on your gateway/firewall.

Kedar.

I think the solution is to have an architecture for the IoT, and home/business networks in general, where devices are not internet addressable by default. If you can't send it packets, you can't attack it. All IoT devices should be required to talk to the Internet via an IoT hub. The fact that the hub mediates all traffic means that you can have a lot of cheap crap devices and use the hub to make sure if one of them is vulnerable, an attack can't get through. It could also do traffic limiting (your thermometer should never need to send more than 10kbps of traffic to the outside world), SSL inspection and plenty of other useful things.

See http://blog.gerv.net/2016/03/an-iot-vision/ , https://blog.gerv.net/2016/10/security-updates-not-needed/ and https://blog.gerv.net/2016/10/no-default-passwords/ .

They might not want to update if you tell them their security camera is attacking Twitter. But what if you point out that the same bad guys who hijack their security camera to attack Twitter can also see their pictures for industrial espionage or whatever. Maybe that's something which might count as a more relevant incentive?

"Why the IoT security nightmare could be a dream for Ubuntu"

http://arstechnica.co.uk/business/2016/10/ubuntu-snap-packaging-internet-of-things-security/

Hello
I really wonder about this discussion.
The problem is, that the manufacturers did not even tried to make their devices secure.
Fixed passwords?
20 years ago - ok.
Today? - Just let them pay the invoice.
For example, if a device is misused in an attack, then the Manufacturer has to pay 10$ per sold device for the 1st attack. 20$ for the 2nd attack.

I'm pretty sure they'll follow basic security rules extremely fast and they will offer patches in no time.

Kind regards, Tom

All my devices, except the ISP connected router, connect via NAT. Unless I explicitly configure some tunnels on that router, the internet cannot initiate connections to those devices. In the case of all these IoT, how is NAT being circumvented? Is NAT not how typical consumer things connect to the internet?

Build a bot net to shut down insecure devices when found. The resulting failures would force security and password upgrades by the public. And the DOS that hits manufacturers CS lines with irate customers would force them to fix their security flaws FAST. Someone will cry illegal, but there is no other way to effectively end the threat. It would just take some bot net building knowledge and some courage.

More and more I think the answer to "But if the ISPs cut off the installed base of crappy IoT, all those people will be inconvenienced" is "Yes. Yes. That's sort of the point."

A big part of the problem is that it's possible to ignore the problem right up until your house bursts into flames or the police show up. A policy of NO LOLCAT NO PORN NO NETFLIX until you unplug that damn thing would dramatise the whole issue and probably precipitate a political solution.

Threaded | Flat

no subject

Cutting of single devices

No incoming connections by default

How about open source?

Home routers

There's an economic solution to this

An idea

no subject

Force internet connected devices to be rented

It's our responsibility

maybe we should build better platform ?

no subject

what about ipfs etc?

no subject

Allow maintenance

opt-in subscription on notifications about your devices being used in attacks

Use the client mode

Gatewayed networks

Scare people into updating their devices

no subject

It's extremely easy.

How is NAT failing?

Build a bot net to shut down insecure devices

no subject