Fixing the IoT isn't going to be easy

[mjg59]

A large part of the internet became inaccessible today after a botnet made up of IP cameras and digital video recorders was used to DoS a major DNS provider. This highlighted a bunch of things including how maybe having all your DNS handled by a single provider is not the best of plans, but in the long run there's no real amount of diversification that can fix this - malicious actors have control of a sufficiently large number of hosts that they could easily take out multiple providers simultaneously.

To fix this properly we need to get rid of the compromised systems. The question is how. Many of these devices are sold by resellers who have no resources to handle any kind of recall. The manufacturer may not have any kind of legal presence in many of the countries where their products are sold. There's no way anybody can compel a recall, and even if they could it probably wouldn't help. If I've paid a contractor to install a security camera in my office, and if I get a notification that my camera is being used to take down Twitter, what do I do? Pay someone to come and take the camera down again, wait for a fixed one and pay to get that put up? That's probably not going to happen. As long as the device carries on working, many users are going to ignore any voluntary request.

We're left with more aggressive remedies. If ISPs threaten to cut off customers who host compromised devices, we might get somewhere. But, inevitably, a number of small businesses and unskilled users will get cut off. Probably a large number. The economic damage is still going to be significant. And it doesn't necessarily help that much - if the US were to compel ISPs to do this, but nobody else did, public outcry would be massive, the botnet would not be much smaller and the attacks would continue. Do we start cutting off countries that fail to police their internet?

Ok, so maybe we just chalk this one up as a loss and have everyone build out enough infrastructure that we're able to withstand attacks from this botnet and take steps to ensure that nobody is ever able to build a bigger one. To do that, we'd need to ensure that all IoT devices are secure, all the time. So, uh, how do we do that?

These devices had trivial vulnerabilities in the form of hardcoded passwords and open telnet. It wouldn't take terribly strong skills to identify this at import time and block a shipment, so the "obvious" answer is to set up forces in customs who do a security analysis of each device. We'll ignore the fact that this would be a pretty huge set of people to keep up with the sheer quantity of crap being developed and skip straight to the explanation for why this wouldn't work.

Yeah, sure, this vulnerability was obvious. But what about the product from a well-known vendor that included a debug app listening on a high numbered UDP port that accepted a packet of the form "BackdoorPacketCmdLine_Req" and then executed the rest of the payload as root? A portscan's not going to show that up[1]. Finding this kind of thing involves pulling the device apart, dumping the firmware and reverse engineering the binaries. It typically takes me about a day to do that. Amazon has over 30,000 listings that match "IP camera" right now, so you're going to need 99 more of me and a year just to examine the cameras. And that's assuming nobody ships any new ones.

Even that's insufficient. Ok, with luck we've identified all the cases where the vendor has left an explicit backdoor in the code[2]. But these devices are still running software that's going to be full of bugs and which is almost certainly still vulnerable to at least half a dozen buffer overflows[3]. Who's going to audit that? All it takes is one attacker to find one flaw in one popular device line, and that's another botnet built.

If we can't stop the vulnerabilities getting into people's homes in the first place, can we at least fix them afterwards? From an economic perspective, demanding that vendors ship security updates whenever a vulnerability is discovered no matter how old the device is is just not going to work. Many of these vendors are small enough that it'd be more cost effective for them to simply fold the company and reopen under a new name than it would be to put the engineering work into fixing a decade old codebase. And how does this actually help? So far the attackers building these networks haven't been terribly competent. The first thing a competent attacker would do would be to silently disable the firmware update mechanism.

We can't easily fix the already broken devices, we can't easily stop more broken devices from being shipped and we can't easily guarantee that we can fix future devices that end up broken. The only solution I see working at all is to require ISPs to cut people off, and that's going to involve a great deal of pain. The harsh reality is that this is almost certainly just the tip of the iceberg, and things are going to get much worse before they get any better.

Right. I'm off to portscan another smart socket.

[1] UDP connection refused messages are typically ratelimited to one per second, so it'll take almost a day to do a full UDP portscan, and even then you have no idea what the service actually does.

[2] It's worth noting that this is usually leftover test or debug code, not an overtly malicious act. Vendors should have processes in place to ensure that this isn't left in release builds, but ha well.

[3] My vacuum cleaner crashes if I send certain malformed HTTP requests to the local API endpoint, which isn't a good sign

Comments

[(Anonymous)]

Say we wanted to give the "make 99 more of Matthew Garrett" approach a try anyway. How does one learn to dump firmwares and reverse engineer binaries? Can you recommend any learning resources to train up a next generation of hobbyist IoT security auditors?

Is lesson one perhaps "don't live in a country with anti-circumvention laws?"

[brainwane]

Matthew will probably have a more substantive answer, but I wouldn't be surprised if this exercise is a good place to start.

Is lesson one perhaps "don't live in a country with anti-circumvention laws?"

Or, repeal them when you find them.

Cutting of single devices

[(Anonymous)]

IPv6 would probably make the problem worse, but it would enable ISPs to block single devices close to were they are in the network.

Re: Cutting of single devices

[(Anonymous)]

Temporary, maybe.
Devices will be directly exposed to internet more often (either by misconfiguration or voluntarily) and they will have a more complex network stack in many cases.
On the other hand, it will be much harder to scan the IPv6 address-space for vulnerable devices and there will be less complex network stack once we have abandoned IPv4 (no more NAT).

I think the benefits outweigh the cons by far with IPv6 since quickly scanning for vulnerable devices is a very strong reason why we see all the crap now. To have simpler protocols and simpler setups is also a good reason to strive for it in the long run as that benefits security as well. Temporary it will cause some issues (dual stacks and misconfigured setups due to lack of knowledge by the user/technicians) but that will quickly change when IPv6 become more common.

No incoming connections by default

[(Anonymous)]

You could go half way between doing nothing and cutting off people from the internet.

Given that the first step in infecting vulnerable devices is a port scan, by default ISPs could block all incoming connections until the client explicitly requests that incoming connections should be unblocked.

The catch is I expect ISPs would completely fsck this up and it would be a pain to get things working properly if you know what you are doing.

Re: No incoming connections by default

[(Anonymous)]

This will stop the most trivial attacks, and then the attackers will shift to hacking into supplier's infrastructure (remember how many devices rely on or provide additional features available via supplier's website).

How about open source?

[(Anonymous)]

Coming from an opensource guy I'm confused why you didn't mention this alternative? Let government require by law that all the firmware and software of these devices has the sources published, and that all of them should have a way of being updated by their users.

My statement follows the logic that if there had been this law imposed much earlier, all these companies would have end up joining forces in trying to write joint-written firmware (so that the code is not maintained by a single company) to de-duplicate efforts and spread the responsability.

Re: How about open source?

[(Anonymous)]

That doesn't solve anything, since you can't force people to actually update their devices.

Home routers

[(Anonymous)]

Seems for at least one class of devices, the manufacturer should have to warranty security updates - home routers/firewalls. The OnHub devices make a good example of what to do. And yes, companies that don't should be forced out of business. Now, the next thing to do is to force IoT device makers to create systems where the traffic is outgoing only from the device - so for example you don't connect directly to that new security cam, but instead connect to a cloud server which the security cam also connected to, and only after authenticating with the cloud server are you able to connect to your device. Then when bugs show up, it is fixing up a few cloud servers, not every cheap security cam. Device manufacturers should like this also, they can charge ongoing rent for access to the cloud server.

There's an economic solution to this

[(Anonymous)]

Charging user's per MB for their internet service (http://fattuba.com/a-solution-for-ddos/) would fix this problem.

Re: There's an economic solution to this

[(Anonymous)]

No it wouldn't. The actual amount of traffic from any given device is quite low; the problem is that there are hundreds of thousands of them.

An idea

[(Anonymous)]

Maybe it would be possible to combat this by creating proper incentives. However it doesn't need to be all-or-nothing; it would already do good to significantly reduce the number of vulnerable devices.

I haven't thought this through, but how about something like this:

* Forbid commercial import of devices that have been found to be vulnerable (unless the importer undertakes to update their firmware before selling them)
* Forbid the sale of new, already imported devices that have been found to be vulnerable (so, not targeted at consumers selling their used devices)

These I believe could create a few interesting incentives:

* Manufacturers would have an incentive to think of security if they run a risk of not being able to sell them to major markets
* There would be an incentive to find vulnerabilities in competitors' devices and give an anonymous tip
* Commercial importers would have an incentive to try and choose better devices or ones where the manufacturer somehow promises to rectify problems, since they run the risk of being stuck with unsaleable stock

This rule seems a bit harsh in some ways, especially for vendors who really try hard to secure their devices but fail because of, say, some Linux vulnerability. I think it should probably be enough if there are no known vulnerabilities older than, say, 45 days for which there is no firmware update available.

Re: An idea

[(Anonymous)]

Have you seen Kees Cook's data on Linux kernel security flaw lifetimes? Your proposed demands would effectively prevent selling any Linux-based hardware, and would effectively promote closed-source security-by-obscurity (if it hasn't been found vulnerable yet, it's OK to sell).

Something more realistic would be to provide an easy-to-use, secure by default platform that chip vendors could base their BSPs on. A lot of the cheap products base their software more or less directly on the vendor's BSP, and those are frequently some crap thrown together just to get a booting system. It's not unusual to see BSPs still based on some 2.4 kernel, and never updated. Something like Yocto just has a too steep learning curve, and doesn't solve the problem of vendors sticking to outdated versions.

[(Anonymous)]

Where have all the grey hats gone? We can develop a worm that "shuts the door" behind itself after it infects a vulnerable device, and "patches" the insecure devices to prevent subsequent attacks and exploits.

[(Anonymous)]

this!

launch a counter-attack.
we know where the attacks are coming from, so break into those devices and either lock them or (with a dark-grey hat) brick them.

Force internet connected devices to be rented

[(Anonymous)]

Perhaps the end game is internet connected devices become leased rather than purchased outright (even your IoT light bulb). Consumers are forced to pay an ongoing fee and ISPs give the devices away to lure customers in. Devices expected to last a long time (e.g. TVs) are not allowed to ship Internet connectivity directly but must do so via USB add ons that can be replaced (vendor locked of course). Only giants (e.g. Google, Amazon) are allowed to ship direct to consumers under penalty that their devices have to be periodically patched and must auto update.

With this, the choice not to update the firmware is ripped out of the customer's hands. ISPs have an incentive not to let their networks participate in attacks and actually ensure updates make their way to customers so they actually pass updates for hardware they provide on to consumers.

A secondary effort is made to ensure everyone is forced to keep changing (e.g. by using a Linux-eseque upgrade all at once system for protocols). Soon old devices struggle to connect to the internet at all due to the never ending technological changes. All the updates burn through a limited amount of flash rewrite cycles thus creating planned obsolescence or forcing the customer to purchase a new USB key with the latest firmware on it. At best you can buy an auto-updated device for "legacy" devices but they're never allowed to connect to the real Internet only some faux house internal one.

Re: Force internet connected devices to be rented

[(Anonymous)]

"Devices expected to last a long time (e.g. TVs) are not allowed to ship Internet connectivity directly but must do so via USB add ons"

I think there is some confusion here. A TV does not need to be hosted (listening) publicly on the Internet. AFAIK, all Internet-enabled TVs (and devices of a similar ilk), simply make outbound connections and any inbound connections would be blocked by the home router by default.

The only devices that need to be hosted publicly on the Internet (ie. with a listening socket) are those that you need to connect to from outside your home network. So, IP security cameras, etc. Not home devices such as TV's or lightbulbs.

Ofc, I could have the wrong end of the stick here. Apologies if so :-)

It's our responsibility

[https://openid-provider.appspot.com/me@jasonclinton.com]

I know that this is not a popular view within the computer science and open source communities but: this is our problem, not the vendors'.

We ship open source software used by these vendors with unsafe defaults. We don't default closed or put giant PEOPLE_WILL_DIE_IF_YOU_DO_THIS text on our configuration variables. We ship compilers and languages that don't put safety and buffer overflow protection first. Hell, we don't even systematically regression test (let alone fuzz test) the software that our communities produce.

We ship crap code and we expect downmarket vendors to polish it.

We have to fix the problem; no one else will.

There are glimmers of hope: there's healthy projects like boringssl. There's basic language safety efforts like Rust. There's healthy discourse about handling security better within the Linux kernel community. And if that doesn't work out, there's long-shots efforts like user-mode drivers in Magenta.

The first place that any open source contributor can begin is to hold ourselves to a higher standard:

• write units tests,
• try to find a way to run continuous regression tests on your code,
• think about the security costs of backward-incompatible API changes,
• ask for help fuzz testing binaries,
• and, above all, think about the engineer who will recycle your library under deadline pressure.

maybe we should build better platform ?

[(Anonymous)]

Maybe we should:

1. Design a secure architecture - on that the vendor cannot mess up, and all updates are from providers we can count on.

2. Somewhat forcefully shove it in chips. silicon is cheap.

3. Make sure it's extremely cheap on the one hand, and on the other, find some carrot or stick to make sure only/mostly "good" devices enter the market.

Re: maybe we should build better platform ?

[(Anonymous)]

"Secure" is harder to do than you might think if you don't control any side of the equation - you can't guarantee the hardware because that shit is going to get cloned and hacked up as soon as you release it, and people are going to buy the device that supports more things (even if it does this by silently ignoring the invalid crypto bits on some things), and we're back to square one.

A closed system runs into problems with lock-out, forcing people to upgrade their old devices without a legal stick isn't going to work (and people are not going to VLAN off their crappy cameras if they haven't already), and all of our solutions for distributed webs of trust are vulnerable to majority takeover (say, if you compromise the largest vendor in the space).

It's not hopeless, it's just all imperfect solutions.

[(Anonymous)]

All of the DVR-type appliances I've ever examined have been one of two "ODM" boards and software stacks, minimally rebranded - consequently, I would be willing to bet that it's not quite as bad of a hydra to get the common manufacturers to fix this in newer devices as it seems.

No experience with actual IP cameras (versus dumb cameras that only knew how to communicate over closed circuit to a DVR), but I would expect the DVR case to be the better bang/buck (there are probably more distinct internet connections with just one or two IP cameras instead of something managing them, but even if they're mostly the same stock board, there's probably many more vendors to convince to update their shit).

what about ipfs etc?

[(Anonymous)]

Really well written article. Ty, Matthew.

I was surprised that the solutions section didn't mention alternate Internet infrastructure that are more resilient. The first one I think of is the Interplanetary File System.

[brong]

The answer is the same as it is for things plugged into the electricity sockets in your home. At least in Australia they are required by law to meet certain standards before you can legally plug them in:

https://ablis.business.gov.au/VIC/pages/1e057cce-d835-4a4b-91b7-d733fda05ec4.aspx

Hobbyists are allowed to do some stuff themselves, but not allowed to sell it.

Likewise, home routers could be legally required to meet certain standards before being allowed to be plugged in, and that could include having a separate network for iOT devices in which they are required to pre-register hosts they wish to communicate with and be approved before they could be used.

And use the existing legal framework that we have here and probably in most countries for fines and recalls of faulty equipment which doesn't meet the standards, and fines for hobbyists who build their own without it being up to scratch.

As for countries which don't enforce protections against DDoS. Yes, the only solution is to cut them off or rate limit their egress. The current situation where a single actor can, with complete anonymity, take anyone but a handful of giant organisations entirely off the air is broken. When they get powerful enough to take all the DNS roots out at the same time, we're really screwed.

[(Anonymous)]

Many countries have had legislation only allowed connecting certified equipment to the telephone network. That was a terrible situation (ask your nearest old-timer), and the legislation will be certain to include anti-competitive elements. It also doesn't address the issue of whole classes of vulnerabilities discovered after the certification requirements were written, and extant certified devices that are now found to be vulnerable after all. Plus, it takes a long time to push through changes in laws and regulations.

What about connected devices? Does the hardware and server have to be certified as one system? That shifts the burden (and cost!) of certification to the customer, which can be enormously expensive and time-consuming. Want to set up a Linux media server in your home? Not until you pay to get your home network and TV certified! And don't even think about changing the configuration or installing updates later, either.

Allow maintenance

[(Anonymous)]

> If we can't stop the vulnerabilities getting into people's homes in the first place, can we at least fix them afterwards?

Yes. You do this by not-having anything for which the software _cannot_ be maintained. It doesn't necessarily have to _be_ maintained or have updates be available from any particular party (e.g. the vendor or manufacturer). There only needs to be a total and complete lack of barriers which might _prevent_ any interested party from performing the software maintenance. Then, if people want the maintenance enough, someone will do it. (Maybe for money, maybe not.)

If barriers to maintenance exist, you use government force to remove the barrier. Nobody can make a case for lack of software maintenance being a desirable, or even tolerable, thing.

Re: Allow maintenance

[(Anonymous)]

The people who could control updates - the users and the manufacturers - are not the people suffering from DDoS attacks. Even if someone else can do the work, one of those groups has to approve the update, and they've got no incentive to do so. Unless you create a new government agency that has the keys to update any IOT device. Luckily that's unlikely to happen, because government control of cameras inside your house does not sound like a great idea.

Also, it would still require a massive effort, because to be effective, you have to find and patch the security holes *before* malicious actors exploit them; as Matthew mentioned, competent attackers will disable update mechanisms if at all possible.

opt-in subscription on notifications about your devices being used in attacks

[(Anonymous)]

there are certainly a number of people willing to know and take down their hacked devices themselves. Looks like the only thing missing is a notification system from your ISP, looks like an easy solution that will help at least partially.

int_ua

Use the client mode

[(Anonymous)]

One simple thing that will change this in a big way, is to always have IoT devices act a a client instead of a server when being accessed remotely. They talk to a broker/server and then that server manages access control. Most of the latest IoT devices that I have seen adhere to this architectural model. Yes, things become a bit difficult for devices with multimedia, but that should also be possible to accomplish.
No mucking around, opening ports on your gateway/firewall.

Kedar.

Re: Use the client mode

[(Anonymous)]

Yes!

I've been thinking this ever since IoT started to take off. I suspect the main issue of a central server is the cost of setting up and maintaining it.

Also, it goes without saying that the server itself will need to be bulletproof, security-wise. As it will be an obvious target for attacks itself.

Gatewayed networks

[gerv.net]

I think the solution is to have an architecture for the IoT, and home/business networks in general, where devices are not internet addressable by default. If you can't send it packets, you can't attack it. All IoT devices should be required to talk to the Internet via an IoT hub. The fact that the hub mediates all traffic means that you can have a lot of cheap crap devices and use the hub to make sure if one of them is vulnerable, an attack can't get through. It could also do traffic limiting (your thermometer should never need to send more than 10kbps of traffic to the outside world), SSL inspection and plenty of other useful things.

See http://blog.gerv.net/2016/03/an-iot-vision/ , https://blog.gerv.net/2016/10/security-updates-not-needed/ and https://blog.gerv.net/2016/10/no-default-passwords/ .

Scare people into updating their devices

[(Anonymous)]

They might not want to update if you tell them their security camera is attacking Twitter. But what if you point out that the same bad guys who hijack their security camera to attack Twitter can also see their pictures for industrial espionage or whatever. Maybe that's something which might count as a more relevant incentive?

[(Anonymous)]

"Why the IoT security nightmare could be a dream for Ubuntu"

http://arstechnica.co.uk/business/2016/10/ubuntu-snap-packaging-internet-of-things-security/

It's extremely easy.

[(Anonymous)]

Hello
I really wonder about this discussion.
The problem is, that the manufacturers did not even tried to make their devices secure.
Fixed passwords?
20 years ago - ok.
Today? - Just let them pay the invoice.
For example, if a device is misused in an attack, then the Manufacturer has to pay 10$ per sold device for the 1st attack. 20$ for the 2nd attack.

I'm pretty sure they'll follow basic security rules extremely fast and they will offer patches in no time.

Kind regards, Tom

Re: It's extremely easy.

[mjg59]

How do you make them pay?

How is NAT failing?

[(Anonymous)]

All my devices, except the ISP connected router, connect via NAT. Unless I explicitly configure some tunnels on that router, the internet cannot initiate connections to those devices. In the case of all these IoT, how is NAT being circumvented? Is NAT not how typical consumer things connect to the internet?

Build a bot net to shut down insecure devices

[(Anonymous)]

Build a bot net to shut down insecure devices when found. The resulting failures would force security and password upgrades by the public. And the DOS that hits manufacturers CS lines with irate customers would force them to fix their security flaws FAST. Someone will cry illegal, but there is no other way to effectively end the threat. It would just take some bot net building knowledge and some courage.

Re: Build a bot net to shut down insecure devices

[(Anonymous)]

You have to find the flaws before the black hats, who have commercial motivation (people pay to use botnets). Assume smart attackers will close the door behind themselves, so you can't shut down devices that are already hacked.

Also, it's not just someone 'crying illegal'. Your proposal is to destroy goods that people have bought fair and square. Why is that morally better than the botnets you're trying to prevent? The owners of the hardware would probably not think it is.

[(Anonymous)]

More and more I think the answer to "But if the ISPs cut off the installed base of crappy IoT, all those people will be inconvenienced" is "Yes. Yes. That's sort of the point."

A big part of the problem is that it's possible to ignore the problem right up until your house bursts into flames or the police show up. A policy of NO LOLCAT NO PORN NO NETFLIX until you unplug that damn thing would dramatise the whole issue and probably precipitate a political solution.