Piston, an Openstack-in-a-box vendor[1] are a sponsor of the Red Hat[2] Summit this year. Last week they briefly ceased to be for no publicly stated reason, although it's been sugggested that this was in response to Piston winning a contract that Red Hat was also bidding on. This situation didn't last for long - Red Hat's CTO tweeted that this was an error and that Red Hat would pay Piston's sponsorship fee for them.

To Red Hat's credit, having the CTO immediately and publicly accept responsibility and offer reparations seems like the best thing they could possibly do in the situation and demonstrates that there are members of senior management who clearly understand the importance of community collaboration to Red Hat's success. But that leaves open the question of how this happened in the first place.

Red Hat is big on collaboration. Workers get copies of the Red Hat Brand Book, an amazingly well-written description of how Red Hat depends on the wider community. New hire induction sessions stress the importance of open source and collaboration. Red Hat staff are at the heart of many vital free software projects. As far as fundamentally Getting It is concerned, Red Hat are a standard to aspire to.

Which is why something like this is somewhat unexpected. Someone in Red Hat made a deliberate choice to exclude Piston from the Summit. If the suggestion that this was because of commercial concerns is true, it's antithetical to the Red Hat Way. Piston are a contributor to upstream Openstack, just as Red Hat are. If Piston can do a better job of selling that code than Red Hat can, the lesson that Red Hat should take away is that they need to do a better job - not punish someone else for doing so.

However, it's not entirely without precedent. The most obvious example is the change to kernel packaging that happened during the RHEL 6 development cycle. Previous releases had included each individual modification that Red Hat made to the kernel as a separate patch. From RHEL 6 onward, all these patches are merged into one giant patch. This was intended to make it harder for vendors like Oracle to compete with RHEL by taking patches from upcoming RHEL point releases, backporting them to older ones and then selling that to Red Hat customers. It obviously also had the effect of hurting other distributions such as Debian who were shipping 2.6.32-based kernels - bugs that were fixed in RHEL had to be separately fixed in Debian, despite Red Hat continuing to benefit from the work Debian put into the stable 2.6.32 point releases.

It's almost three years since that argument erupted, and by and large the community seems to have accepted that the harm Oracle were doing to Red Hat (while giving almost nothing back in return) justified the change. The parallel argument in the Piston case might be that there's no reason for Red Hat to give advertising space to a company that's doing a better job of selling Red Hat's code than Red Hat are. But the two cases aren't really equal - Oracle are a massively larger vendor who take significantly more from the Linux community than they contribute back. Piston aren't.

Which brings us back to how this could have happened in the first place. The Red Hat company culture is supposed to prevent people from thinking that this kind of thing is acceptable, but in this case someone obviously did. Years of Red Hat already having strong standing in a range of open source communities may have engendered some degree of complacency and allowed some within the company to lose track of how important Red Hat's community interactions are in perpetuating that standing. This specific case may have been resolved without any further fallout, but it should really trigger an examination of whether the reality of the company culture still matches the theory. The alternative is that this kind of event becomes the norm rather than the exception, and it takes far less time to lose community goodwill than it takes to build it in the first place.

[1] And, in the spirit of full disclosure, a competitor to my current employer
[2] Furthering the spirit of full disclosure, a former employer
I spent last week looking into the firmware configuration protocol used on current IBM system X servers. IBM provide a tool called ASU for configuring firmware settings, either in-band (ie, running on the machine you want to reconfigure) or out of band (ie, running on a remote computer and communicating with the baseboard management controller - IMM in IBM-speak). I'm not a fan of using vendor binaries for this kind of thing. They tend to be large (ASU is a 20MB executable) and difficult to script, so I'm gradually reimplementing them in Python. Doing that was fairly straightforward for Dell and Cisco, both of whom document their configuration protocol. IBM, on the other hand, don't. My first plan was to just look at the wire protocol, but it turns out that it's using IPMI 2.0 to communicate and that means that the traffic is encrypted. So, obviously, time to spend a while in gdb.

The most important lesson I learned last time I did this was "Check whether the vendor tool has a debug option". ASU didn't mention one in its help text and there was no sign of a getopt string, so this took a little longer - but nm helpfully showed a function called DebugLog and setting a breakpoint on that in gdb showed it being called a bunch of the time, so woo. Stepping through the function revealed that it was checking the value of a variable, and looking for other references to that variable in objdump revealed a -l argument. Setting that to 255 gave me rather a lot of output. Nearby was a reference to --showsptraffic, and passing that as well gave me output like:

BMC Sent: 2e 90 4d 4f 00 06 63 6f 6e 66 69 67 2e 65 66 69
BMC Recv: 00 4d 4f 00 21 9e 00 00

which was extremely promising. 2e is IPMI-speak for an OEM specific command, which seemed pretty plausible. 4d 4f 00 is 20301 in decimal, which is the IANA enterprise number for IBM's system X division (this is required by the IPMI spec, it wasn't an inspired piece of deduction on my behalf). 63 6f 6e 66 69 67 2e 65 66 69 is ASCII for config.xml. That left 90 and 06 to figure out. 90 appeared in all of the traces, so it appears to be the command to indicate that the remaining data is destined for the IMM. The prior debug output indicated that we were in the QuerySize function, so 06 presumably means… query the size. And this is supported by the response - 00 (a status code), 4d 4f 00 (the IANA enterprise number again, to indicate that the responding device is speaking the same protocol as you) and 21 9e 00 00 - or, in rational endianness, 40481, an entirely plausible size.

Once I'd got that far the rest started falling into place fairly quickly. 01 rather than 06 indicated a file open request, returning a four byte file handle of some description. 02 was read, 03 was write and 05 was close. Hacking this together with pyghmi meant I could open a file and read it. Success!

Well, kind of. I was getting back a large blob of binary. The debug trace showed calls to an EfiDecompress function, so on a whim I tried just decompressing it using the standard UEFI compression format. Shockingly, it worked and now I had a 345K XML blob and presumably many more problems than I'd previously had. Parsing the XML was fairly straightforward, and now I could retrieve the full set of config options, along with the default, current and possible values for them.

Making changes was pretty much just the reverse of this. A small XML blob containing the new values is compressed and written to asu_update.efi. One of the elements is a random identifier, which will then appear in another file called config_log along with a status. Just keep re-reading this until the value changes to CM_DONE and we're good.

The in-band configuration appears to be identical, but rather than sending commands over the wire they're send through the system's IPMI controller directly. Yes, this does mean that it's sending compressed XML through a single io port. Yes, I'd rather be drinking.

I've documented the protocol here and hope to be able to release this code before too long - right now I'm trying to nail down the interface a little, but it's broadly working.
Contributor License Agreements ("CLAs") are a mechanism for an upstream software developer to insist that contributors grant the upstream developer some additional set of rights. These range in extent - some CLAs require that the contributor reassign their copyright over the contribution to the upstream developer, some merely provide the upstream developer with a grant of rights that aren't explicit in the software license (such as an explicit patent grant for a contribution licensed under a BSD-style license).

CLAs aren't new. FSF-copyrighted projects have been using copyright assignment since at least 1985 - in return, the FSF promise that the software will always be distributed under a copyleft-style license. For over a decade, Apache Software Foundation projects have required that contributors sign a CLA that allows them to retain copyright, but grants the ASF the right to relicense the work as it wishes. For the most part, this hasn't been terribly controversial.

So why do people object so much when Canonical do it? I've written about this in the context of Mir before, but it's worth expanding on the general case. The FSF's copyright assignment ensures that contributions to GPLed software will only be distributed under GPL-style licenses. The Apache CLA permits the ASF to relicense a contribution under a proprietary license, but the Apache license allows anyone to do that anyway. Going through Wikipedia's list of CLA users, the majority cover projects that are under BSD- or Apache-style licenses, with a couple of cases covering GPLed projects with a promise that any contributions will only be distributed under GPL-like licenses[1]. Either everyone can produce proprietary derivative works, or nobody can.

In contrast, Canonical ship software under the GPLv3 family of licenses (GPL, AGPL and LGPL) but require that contributors sign an agreement that permits Canonical to relicense their contributions under a proprietary license. This is a fundamentally different situation to almost all widely accepted CLAs, and it's disingenuous for Canonical to defend their CLA by pointing out the broad community uptake of, for instance, the Apache CLA.

Canonical could easily replace their CLA with one that removed this asymmetry - Project Harmony, the basis of Canonical's CLA, permits you to specify an "inbound equals outbound" agreement that prevents upstream from relicensing under a proprietary license[2]. Canonical's deliberate choice not to do so just strengthens the argument that the CLA is primarily about wanting to produce proprietary versions of software rather than wanting to strengthen their case in any copyright or patent disputes. It's unsurprising that people feel disinclined to contribute to projects under those circumstances, and it's difficult to understand why Canonical simultaneously insist on this hostile behaviour and bemoan the lack of community contribution to Canonical projects.

[1] The one major exception is the Digia/Qt project CLA, which covers an LGPLed work but makes it entirely clear that Digia will ship your contributions under proprietary licenses as well. At least they're honest.

[2] See the various options in section 2.1(d) here. Canonical chose option five. If they'd chosen option one instead, this wouldn't be a problem.
(Discussion of a presentation I gave at Kiwicon last month)

Kexec is a Linux kernel feature intended to allow the booting of a replacement kernel at runtime. There's a few reasons you might want to do that, such as using Linux as a bootloader[1], rebooting without having to wait for the firmware to reinitialise or booting into a minimal kernel and userspace that can be booted on crash in order to save system state for later analysis.

But kexec's significantly more flexible than this. The kexec system call interface takes a list of segments (ie, pointers to a userspace buffer and the desired target destination) and an entry point. The kernel relocates those segments and jumps to the entry point. That entry point is typically code referred to as purgatory, due to the fact that it lives between the world of the first kernel and the world of the second kernel. The purgatory code sets up the environment for the second kernel and then jumps to it. The first kernel doesn't need to know anything about what the second kernel is or does. While it's conventional to load Linux, you can load just about anything.

The most important thing to note here is that none of this is signed. In other words, despite us having a robust in-kernel mechanism for ensuring that only signed modules can be inserted into the kernel, root can still load arbitrary code via kexec and execute it. This seems like a somewhat irritating way to patch the running kernel, so thankfully there's a much more straightforward approach.

I modified kexec to add an additional loader and uploaded the code here. Build and install it. Make sure that /sys/module/module/parameters/sig_enforce on your system is "Y". Then, as root, do something like:
kexec --type="dummy" --address=`printf "0x%x" $(( $(grep "B sig_enforce" /proc/kallsyms | awk '{print "0x"$1}') & 0x7fffffff))` --value=0 --load-preserve-context --mem-max=0x10000 /bin/true
to load it[2]. Now do kexec -e and watch colours flash and check /sys/module/module/parameters/sig_enforce again.

The beauty of this approach is that it doesn't rely on any kernel bugs - it's using kernel functionality that was explicitly designed to let you do this kind of thing (ie, run arbitrary code in ring 0). There's not really any way to fix it beyond adding a new system call that has rather tighter restrictions on the binaries that can be loaded. If you're using signed modules but still permit kexec, you're not really adding any additional security.

But that's not the most interesting way to use kexec. If you can load arbitrary code into the kernel, you can load anything. Including, say, the Windows kernel. ReactOS provides a bootloader that's able to boot the Windows 2003 kernel, and it shouldn't be too difficult for a sufficiently enterprising individual to work out how to get Windows 8 booting. Things are a little trickier on UEFI - you need to tell the firmware which virtual→physical map to use, and you can only do it once. If Linux has already done that, it's going to be difficult to set up a different map for Windows. Thankfully, there's an easy workaround. Just boot with the "noefi" kernel argument and the kernel will skip UEFI setup, letting you set up your own map.

Why would you want to do this? The most obvious reason is avoiding Secure Boot restrictions. Secure Boot, if enabled, is explicitly designed to stop you booting modified kernels unless you've added your own keys. But if you boot a signed Linux distribution with kexec enabled (like, say, Ubuntu) then you're able to boot a modified Windows kernel that will still believe it was booted securely. That means you can disable stuff like the Early Launch Anti-Malware feature or driver signing, or just stick whatever code you want directly into the kernel. In most cases all you'd need for this would be a bootloader, kernel and an initrd containing support for the main storage, an ntfs driver and a copy of kexec-tools. That should be well under 10MB, so it'll easily fit on the EFI system partition. Copy it over the Windows bootloader and you should be able to boot a modified Windows kernel without any terribly obvious graphical glitches in the process.

And that's the story of why kexec is disabled on Fedora when Secure Boot is enabled.

[1] That way you only have to write most drivers once
[2] The address section finds the address of the sig_enforce symbol in the kernel, and the value argument tells the dummy code what value to set that address to. --load-preserve-context informs the kernel that it should save hardware state in order to permit returning to the original kernel. --mem-max indicates the highest address that the kernel needs to back up. /bin/true is just there to satisfy the argument parser.
According to its charter, the Linux Foundation Technical Advisory Board exists "to advise The Linux Foundation Board of Directors (Board) and the management of The Linux Foundation (Management) on matters related to supporting the technical agenda of The Linux Foundation". It consists of 10 members, with 5 seats up for election each year. Elections take place at an event at Kernel Summit, but are also open to attendees of whichever conference is colocated with the Kernel Summit that year. The election announcement is emailed to the Linux kernel list and tech-board-discuss, a mostly moribund list that springs into life once a year for election announcements.

This arrangement seemed odd to me even back in 2007. Back then the Linux Foundation was already sponsoring development of certain non-kernel components, and now that list is even larger. While nominally open to all, nominations for the TAB tend to end up being people actively involved in the kernel community. That's probably better than people limited to any other single technical community (kernel developers tend to end up dealing with bugs from a fairly wide range of projects, so they're not entirely unaware of what other people have to deal with), but it still seems suboptimal. The current membership is mostly limited to people who spend little to no time working with userspace developers, let alone anyone active in other Linux Foundation projects. I don't think this is a good thing

So, after several years of considering it, I'm finally standing for the TAB. I've been an active developer at most levels of the Linux stack, from the kernel through to desktop environments. I've worked closely with distributions. I've even worked closely with firmware developers. I'm not intimately involved in any of the other Linux Foundation projects, but I have experience that allows me to better understand their needs and motivations than I'd have from having spent my entire life living in the kernel. Being on the TAB would make it easier to ensure that these projects are represented in a meaningful way.

Of the other people who I know are standing (and this list may well grow longer), I'm inclined to vote for:
  • Greg Kroah Hartman - while we've disagreed on a range of points, Greg's worked closely with userspace developers and even represents the Linux Foundation on the UEFI Forum. He's able to provide a wide range of expertise to the TAB and it benefits from that.
  • Jon Corbet - Jon's work on LWN should need no introduction. He's done an amazing job of keeping track of a range of technical developments across the entire Linux community, so is uniquely well suited to making sure that a range of opinions is represented.
  • Sarah Sharp - Sarah's certainly primarily a kernel developer, but she's been the loudest voice calling for the kernel community to spend some time thinking about whether it's as welcoming as it could be. Increasing the diversity of the kernel community allows us to hear a wider range of technical viewpoints, which benefits both the kernel and everything that depends on it.

The election is currently scheduled to be National Museum in Edinburgh on the evening of the 23rd of October. If you're attending Linuxcon or one of the colocated events, please do come along and vote.

(Updated to correct a typo)
Hacker News is a fairly influential link aggregation site, with stories submitted and voted on by users. As explained in the FAQ, the ranking of stories is roughly determined by the number of votes divided by a function of the time since submission. It's not a huge traffic driver (my personal experience of stories on the front page is on the order of 30,000 hits), but it's notable because the demographic tends to include a more technically literate and influential set of readers than most other sites. The discussion that ensues from technical posts often includes meaningful feedback from domain experts. Stories that appear there are likely to be noted by technology workers, especially in the Silicon Valley startup field[1].

That rather specific demographic appears to correlate with other traits. There's a rather more techno-libertarian bias on Hacker News than on most general discussion forums, which is consistent with the startup-oriented culture that it springs from - the desire to provide disruptive solutions to real world problems tends to collide with existing regulatory frameworks, so it's unsurprising that a belief in individual rights and small government would overlap with US startup culture. There's a leaning towards the use of web technologies rather than traditional client applications, which matches what people are doing in the rest of the world. And there's more enthusiasm for liberal open-source licenses over Copyleft licenses, which makes sense in a web-focused environment (as I wrote about here).

Now, personally I'm a big-government, client-app, Copyleft kind of person, but for the most part I don't think the above is actively dangerous. It's inevitable that political views will vary, we'll probably continue to cycle between thick and thin clients for generations and nobody's ever going to demonstrably prove that one licensing model deserves to win over another. But what is important is that the ongoing debates between these opinions be driven by facts, and that it remain obvious that these disagreements exist. As far as technical (and even political) discussion goes, Hacker News doesn't seem to
have a problem with that. Disagreeing with the orthodoxy is tolerated.

This seems less true when it comes to social issues. When a posting discussing the myth of the natural born programmer[2] hit the front page, the top rated comment is Paul Graham[3] off-handedly discounting the conclusions drawn. The original story linked to a review of peer-reviewed scientific research. Graham simply discounts it on the basis of his preconceptions. Shortly afterwards, the story plummeted off the front page, now surrounded by stories posted around the same time but with much lower scores.

How does this happen? There's two publicised methods which can result in stories dropping down the order. Users with high karma scores (either via submitting popular stories or writing popular comments) are able to flag submissions, and if enough do so then a negative weighting is applied to the story. There's also a flamewar detector, a heuristic that attempts to detect contentious subjects and pushes them off the front page.

The effect of both is to enforce the status quo of social beliefs. Stories that appear to challenge the narrative that good programmers are just naturally talented tend to vanish. Stories that discuss the difficulties faced by minorities in our field are summarily disappeared. There are no social problems in the technology industry. We have always been at war with Eastasia.

This isn't healthy. We don't improve the state of the software industry by hiding stories that expose conflicts. Flamewars don't solve problems, but without them we'd be entirely unaware of how much of a victim blaming mentality exists amongst our peers. It's true that conflict may reinforce preconceptions, causing people to dig in as they defend their beliefs. However, the absence of conflict does nothing to counteract that. If you're never exposed to opinions you disagree with, you'll never question your existing beliefs.

Hacker News is a privately run site and nobody's under any obligation to change how they choose to run it. But the focus on avoiding conflict to such an extent that controversial stories receive less exposure than ones that fit people's existing beliefs doesn't enhance our community. If we want to be able to use technology as an instrument of beneficial change to society as a whole, we benefit from building a diverse and welcoming community and questioning our preconceptions. Building a social echo chamber risks marginalising us from the rest of society, gradually becoming ignored and irrelevant as our self-reinforcing opinions drift ever further away from the mainstream. It doesn't help anybody.

[1] During the batch of interviews I did last year, two separate interviewers both mentioned a story they'd read on Hacker News that turned out to have been written by me. I'm not saying that that's what determined a hire/don't hire decision, but it seems likely that it helped.
[2] The article in question discusses the pervasive idea that some people are inherently good programmers. It turns out that perpetuating the idea that some people are just born good at a particular skill actually discourages others from even trying to learn it, even those who are just as capable.
[3] One of the co-founders of Y-Combinator and creator of Hacker News.

(Edited to fix a footnote reference)
XMir's been delayed from Ubuntu 13.10. The stated reason is that multi-monitor support isn't sufficiently reliable. That's true, but it's far from the only problem that XMir still has:

  • It's still broken on some single-monitor systems

  • Some machines have display corruption. There are writes to freed memory. To be fair, some of the behaviour that's been seen has been down to underlying bugs in the Xorg drivers that were never triggered under normal use but are hit by XMir. Others are down to implicit assumptions made in the drivers that XMir happens to violate. The problem is that there doesn't appear to have been enough room in the schedule to deal with these interactions, presumably because nobody accounted for the inevitable "This thing we thought would be easy turns out to be difficult" part of the project.

  • The input driver bug still isn't really fixed

  • I've mentioned this bug before. It's now marked "Fix released" which is kind of true but not really. Mir now tells XMir that the VT is changing before the VT is changed, but it doesn't wait for that to complete before switching the VTs. Until XMir is scheduled and runs, it's still receiving input events. In most circumstances that window is small, so there's no real risk of triggering it accidentally.

    There's one corner case where it might still cause problems. Simply running isn't enough - XMir has to make progress through its event loop. That'll only happen if the X server is processing its wakeup events, and it's possible to effectively stall that by submitting a sufficiently awkward drawing request to the server. The X server will appear to freeze, and if the user then attempts to work out what's going on by switching to a VT and logging in, those input events will still be going to XMir. It's left as an exercise to the reader to construct ways to take advantage of this.

    This can't happen in Xorg because the VT switch is blocked until the input devices have been released. Mir could have done the same, but doesn't because of a conscious design decision - in the Ubuntu Phone world, clients stop doing things when they're told to. Ubuntu Desktop is expected to behave the same way.

    This is an unfortunate situation to be in. Ubuntu Desktop was told that they were switching to XMir, but Mir development seems to be driven primarily by the needs of Ubuntu Phone. XMir has to do things that no other Mir client will ever cope with, and unless Mir development takes that into account it's inevitably going to suffer breakage like this. Canonical management needs to resolve this if there's any hope of ever shipping XMir as the default desktop environment.

  • It's still missing features

  • XMir doesn't support colour profiles. XRandR properties aren't exposed, so there's no way to control TV output encoding or overscan. There's still no hardware cursor support. Switching to XMir now would reduce functionality without providing any user-visible gain.

It's clear that XMir has turned into a larger project than Canonical had originally anticipated, but that's hardly surprising. There's only one developer with previous X experience working on it full-time, and the announced schedule provided no opportunity to deal with unexpected problems. As if that weren't enough, there's obvious conflict between Ubuntu Desktop and Ubuntu Phone when it comes to developer time and required functionality. The one hardware vendor who's willing to take a public stand has demonstrated that they have no interest in supporting XMir, despite Canonical assuring people that they were already engaging in productive negotiations with GPU manufacturers.

Multiple monitor support may be the single most obvious feature where XMir is lacking, but it's not the only one. Technically and politically, this code is a long way from being ready for prime time. Without significant community contribution, Canonical need to focus on prioritising it appropriately rather than expecting a tiny number of developers to perform miracles. Or, alternatively, they could just drop XMir entirely and focus on Unity 8.
One of the earlier examples of students using MIT computer resources to lay the groundwork for a later commercial endeavour, Zork was originally written in a LISP derivative called MDL. This was later tuned into the Zork Implementation Language, a domain specific language that was compiled to target the Z-machine rather than a specific piece of hardware. Combined with machine-specific Z-machine interpreters, this allowed rapid porting of games to a wide range of platforms - the only thing that needed to be rewritten was the interpreter, and that could be reused for any future games running on the same hardware.

Infocom were eventually acquired and killed off, but fan interest in their games continued. New Z-machine interpreters were written in order to allow their games (including Zork) to be run on platforms that Infocom had never targetted. One of the best known is Frotz. This has the advantage of being (a) portable and (b) including a "dumb" UI that makes no assumptions about the availablity of any vaguely useful functionality. Like, say, a Curses library.

So, Frotz seemed like the natural choice when this happened. But despite having a set of functionality that makes it look much more like an OS than a boot environment, UEFI doesn't actually expose a standard C library. The EFI Application Development Kit solves this particular design decision. Porting Frotz ended up involving far more fixing up of Frotz bugs that tripped up -Werror than anything else. One note, though - make sure you include DevShell in the list of required packages at build time, otherwise file i/o will mysteriously fail.

The tying of file i/o to the shell protocol unfortunately means that Frotz can't be directly launched by the firmware. The Boot to Zork images therefore contain a UEFI shell in the standard boot location (\EFI\BOOT\BOOTX64.EFI) which is executed when the firmware attempts to boot the device. The shell then looks for a file called "startup.nsh" in the root directory of the boot device and executes it. Unfortunately this doesn't actually set the shell equivalent of the current device, and so just launching Frotz from startup.nsh fails when Frotz can't open the Zork data file. The solution for this is simple, if ugly - the script walks through the list of devices, looking for one that contains ZORK1.DAT in the root directory. It then changes to that device and launches Frotz. If Frotz exits, it then resets the system.

This could be avoided by doing some more work and turning Frotz into a more UEFI-native application. Teaching Frotz to make native UEFI calls would avoid the requirement for the shell protocols, and the firmware provides a mechanism to obtain the path of the currently running executable which would avoid the need to explicitly locate the device. But I'm lazy and this was a "I'm spending the day on a plane" project initially inspired by a Sazerac-fuelled conversation during the UEFI plugfest, not a demonstration of UEFI best practices.

UEFI Boot To Zork and the source code to the modified Frotz can be downloaded here.

Eric (a fellow Fedora board member) has a post describing his vision for what Fedora as an end goal should look like. It's essentially an assertion that since we have no idea who our users are or what they want, we should offer them everything on an equal footing.

Shockingly enough, I disagree.

At the most basic level, the output of different Special Interest Groups is not all equal. We've had issues over the past few releases where various spins have shipped in a broken state, because the SIG responsible for producing them doesn't have the resources to actually test them. We're potentially going to end up shipping F20 with old Bluetooth code because the smaller desktops aren't able to port to the new API in time[1]. Promoting these equally implies that they're equal, and doing so when we know it isn't the case is a disservice to our users.

But it's not just about our users. Before I joined the Fedora project, I'd worked on both Debian and Ubuntu. Debian is broadly similar to the current state of Fedora - no strong idea about what is actually being produced, and a desire among many developers to cater to every user's requirements. Ubuntu's pretty much the direct opposite, with a strongly defined goal and a willingness to sacrifice some use cases in order to achieve that goal.

This leads to an interestingly different social dynamic. Ubuntu contributors know what they're working on. If a change furthers the well-defined aim of the project, that change happens. Moving from Ubuntu to Fedora was a shock to me - there were several rough edges in Fedora that simply couldn't be smoothed out because fixing them for one use case would compromise another use case, and nobody could decide which was more important[2]. It's basically unthinkable that such a situation could arise in Ubuntu, not just because there was a self appointed dictator but because there was an explicit goal and people could prioritise based on that[3].

Bluntly, if you have a well-defined goal, people are more likely to either work towards that goal or go and do something else. If you don't, people will just do whatever they want. The risk of defining that goal is that you'll lose some of your existing contributors, but the benefit is that the existing contributors will be more likely to work together rather than heading off in several different directions.

But perhaps more importantly, having a goal can attract people. Ubuntu's Bug #1 was a solid statement of intent. Being freer than Microsoft wasn't enough. Ubuntu had to be better than Microsoft products on every axis, and joining Ubuntu meant that you were going to be part of that. Now it's been closed and Ubuntu's wandered off into convergence land, and signing up to spend your free time on producing something to help someone sell phones is much less compelling than doing it to produce a product you can give to your friends.

Fedora should be the obvious replacement, but it's not because it's unclear to a casual observer what Fedora actually is. The website proudly leads with a description of Fedora as a fast, stable and powerful operating system, but it's obvious that many of the community don't think of Fedora that way - instead it's a playground to produce a range of niche derivatives, with little consideration as to whether contributing to Fedora in that way benefits the project as a whole. Codifying that would actively harm our ability to produce a compelling product, and in turn reduce our ability to attract new contributors even further.

Which is why I think the current proposal to produce three first-class products is exciting. Offering several different desktops on the download page is confusing. Offering distinct desktop, server and cloud products isn't. It makes it clear to our users what we care about, and in turn that makes it easier for users to be excited about contributing to Fedora. Let's not make the mistake of trying to be all things to all people.

[1] Although clearly in this case the absence of a stable ABI in BlueZ despite it having had a dbus interface for the best part of a decade is a pretty fundamental problem.
[2] See the multi-year argument over default firewall rules and the resulting lack of working SMB browsing or mDNS resolving
[3] To be fair, one of the reasons I was happy to jump ship was because of the increasingly autocratic way Ubuntu was being run. By the end of my involvement, significant technical decisions were being made in internal IRC channels - despite being on the project's Technical Board, I had no idea how or why some significant technical changes were being made. I don't think this is a fundamental outcome of having a well-defined goal, though. A goal defined by the community (or their elected representatives) should function just as well.
Update: While this isn't strictly fixed (Mir will still perform a VT switch without waiting to ensure that XMir has released the input devices), the time window for console input events to end up in XMir is now small enough that it's probably not a security problem any more.

It'd be easy to assume that in a Mir-based world, the Mir server receives input events and hands them over to Mir clients. In fact, as I described here, XMir uses standard Xorg input drivers and so receives all input events directly. This led to issues like the duplicate mouse pointer seen in earlier versions of XMir - as well as the pointer being drawn by XMir, Mir was drawing its own pointer.

But there's also some more subtle issues. Mir recently gained a fairly simple implementation of VT switching, simply listening for input events where a function key is hit while the ctrl and alt modifiers are set[1]. It then performs the appropriate ioctl on /dev/console and the kernel switches the VT. The problem here is that Mir doesn't tell XMir that this has happened, and so XMir still has all of its input devices open and still pays attention to any input events.

This is pretty easy to demonstrate. Open a terminal or text editor under Xmir and make sure it has focus. Hit ctrl+alt+f1 and log in. Hit ctlr+alt+f7 again. Your username and password will be sitting in the window.

This is Launchpad bug 1192843, filed on the 20th of June. A month and a half later, Mir was added to the main Ubuntu repositories. Towards the bottom, there's a note saying "XMir always listening to keyboard, passwords may appear in other X sessions". This is pretty misleading, since "other X sessions" implies that it's only going to happen if you run multiple X sessions. Regardless, it's a known bug that can potentially leak user passwords.

So it's kind of odd that that's the only mention of it, hidden in a disused toilet behind a "Doesn't work on VESA" sign. If you follow the link to installation instructions you get this page which doesn't mention the problem at all. Now, to be fair, it doesn't mention any of the other problems with Mir either, but the other problems merely result in things not working rather than your password ending up in IRC.

This being developmental software isn't an excuse. There's been plenty of Canonical-led publicity about Mir and people are inevitably going to test it out. The lack of clear and explicit warnings is utterly inexcusable, and these packages shouldn't have landed in the archive until the issue was fixed. This is brutally irresponsible behaviour on the part of Canonical.

So, if you ever switch to a text VT, either make sure you're not running XMir at the moment or make sure that you never leave any kind of network client focused when you switch away from X. And you might want to check IRC and IM logs to make sure you haven't made a mistake already.

[1] One lesser-known feature of X is that the VT switching events are actually configured in the keymap. ctrl+alt+f1 defaults to switching to VT1, but you can remap any key combination to any VT switch event. Except, of course, this is broken in XMir because Mir catches the keystroke and handles it anyway.
If you're tempted to add a platform-specific quirk to a Linux driver, pause and do the following:

  1. Check whether the platform works correctly with the generic Windows driver for the hardware in question. If it requires a platform-specific driver rather than the generic one, adding a quirk is probably ok.
  2. If the generic Windows driver works, check whether there's any evidence of platform-specific code in the Windows driver. This will typically be in the .inf file, but occasionally you'll want to run strings against the Windows driver and see whether any functions or strings match the platform in question. If there's evidence of special-casing in the generic Windows driver, adding a quirk is probably ok
  3. If the generic Windows driver works and doesn't appear to have any platform-specific special casing, don't add a quirk. You'll plausibly fix the machine you care about, but you won't fix any others that have the same behaviour. Even worse, if someone does eventually fix the problem properly, there's a risk that your special-casing will now break your system.

The moral to this story is: if you think adding a quirk is the right solution, you're almost certainly wrong.
Shipping a UEFI-bootable Linux distribution is a touch awkward[1], with the main sticking point being the necessity to produce boot media with multiple El-Torito images. An El-Torito image is either an image of a floppy disk or a small hard drive, embedded into the ISO. This allows BIOS systems to look for an El Torito image, hook some interrupts and then boot without the BIOS having to care about the fact that the image is embedded in an ISO-9660 filesystem. UEFI systems will look for an El-Torito image with a special tag - if they find it they'll mount it as a FAT filesystem and look for a bootloader, and if not they'll fall back to BIOS compatibility.

So, if you want a CD that'll boot on both BIOS and UEFI systems, you need two El-Torito images - one for BIOS, one for UEFI. Unfortunately, various BIOSes deal exceptionally badly with CD images that contain more than one El-Torito image. The most common failure case is to print a menu asking you to choose an option without labelling the options, but some will just fail outright. Thankfully, this is pretty much exclusively limited to 32-bit systems.

Things get irritatingly more complicated due to a quirk of UEFI. UEFI is based on executing code in native mode. That means that 32-bit UEFI systems can't execute 64-bit code in firmware, even if the CPU is capable of it. A 64-bit OS can only boot on 32-bit UEFI if it has very ugly compatibility hacks, including having to rewrite structures and register state every time it makes a UEFI call. The only OS I'm aware of that implements this is MacOS X. Having looked into what it'd take to implement it in Linux, I decided that hammering rusty nails through my feet would be a preferable use of time. Thankfully, I went drinking instead.

So distributions have a choice. They can either produce UEFI-bootable CD images for 32-bit x86 and risk failures on actual 32-bit systems, or they can ignore UEFI entirely on 32-bit and succeed in booting on all the hardware that people actually own[2]. Unsurprisingly, they tend to choose the second option.

So, if you're building an x86 hardware platform, don't ship with 32-bit UEFI. If you're stuck with a 32-bit CPU then just ship BIOS. If you have a 64-bit CPU then ship a 64-bit UEFI. If you ship with 32-bit UEFI, no significant existing Linux distribution will support you, and you'll face an uphill battle to convince them to do so.

32-bit UEFI. Just say what on earth were you thinking, please, no, can't you find a solution that doesn't involve me getting tetanus jabs.

(If you're worried about the extra memory consumption of 64-bit OSes, just encourage 32-bit userspace on a 64-bit kernel. Or just boot via BIOS)

[1] See the number that still don't manage it despite having had several years to adapt
[2] Until recently, the only vendor to ship 32-bit UEFI firmware on volume hardware was Apple. This was fine on their 32-bit systems, but on their 64-bit systems with 32-bit UEFI booting a 64-bit version of Windows would result in boot failure. Apple rectified this by stating that 64-bit Windows wasn't supported on platforms with 32-bit UEFI, which is a neat trick if you can manage it. 32-bit Windows (and Linux) was fine because it didn't include a UEFI boot image and so didn't trigger the bug.
Jon Masters, Chief ARM Architect at Red Hat, recently posted a description of his expectations for baseline arm64 servers. The quick summary is that systems should implement UEFI and ACPI, and any more traditional ARM boot mechanisms should be ignored. This is an interesting departure from the status quo in the ARM world, and it's worth thinking about the benefits and drawbacks of this approach.

It's very easy to build a generic kernel for most x86 systems, since the PC platform is fairly well defined even if not terribly well specified. Where system hardware does vary, it's almost always exposed on an enumerable bus (such as PCI or USB) which allows the OS to bind appropriate drivers. Things are different in the ARM world. Even once you're past the point of different SoC vendors requiring different kernel setup code and drivers, you still have to cope with the fact that system vendors can wire these SoCs up very differently. Hardware is often attached via GPIO lines without any means to enumerate them. The end result is that you've traditionally needed a different kernel for every ARM board. This is viable if you're selling the OS and hardware as a single product, but less viable if there's any desire to run a generic OS on the hardware.

The solution that's been adopted for this in the Linux world is called Device Tree. Device Tree actually has significant history, having been used as the device descriptor format in Open Firmware. Since there was already support for it in the Linux kernel, adapting it for use in ARM devices was straightforward. Device Tree aware devices can pass a descriptor blob to the kernel at startup[1], and devices without that knowledge can have a blob build into the kernel.

So, if this problem is already solved, why the push to move to UEFI and ACPI? This push didn't actually originate in the Linux world - Microsoft mandate that Windows RT devices implement UEFI and ACPI, and were they to launch a Windows ARM server product would probably carry that over. That makes sense for Microsoft, since recent versions of Windows have been x86 only and so have grown all kinds of support for ACPI and UEFI. Supporting Device Tree would require Microsoft to rewrite large parts of Windows, whereas mandating UEFI and ACPI allowed them to reuse most of their existing Windows boot and driver code. As a result, largely at Microsoft's behest, ACPI 5 has grown a range of additional features for describing things like GPIO pinouts and I2C connections. Whatever your weird device layout, you can probably express it via ACPI.

This argument works less well for Linux. Linux already supports Device Tree, whereas it currently doesn't support ACPI or UEFI on ARM[2]. Hardware vendors are already used to working with Device Tree. Moving to UEFI and ACPI has the potential to uncover a range of exciting new kernel issues and vendor bugs. It's not obviously an engineering win.

So how about users? There's an argument that since server vendors are now mostly shipping ACPI and UEFI systems, having ARM support these technologies makes it easier for customers to replace x86 systems with ARM systems. This really doesn't fly for ACPI, which is entirely invisible to the user. There are no standard ACPI entry points for system configuration, and the bits of ACPI that are generically useful (such as configuring system wakeup times) are already abstracted away to a standard interface by the kernel. It's somewhat more compelling for UEFI. UEFI supports a platform-independent bytecode language (EFI Byte Code, or EBC), which means that customers can write their own system management utilities, build them for EBC and then deploy them to their servers without caring about whether they're x86 or ARM. Want a bootloader that'll hit an internal HTTP server in order to determine which system image to deploy, and which works on both x86 and ARM? Straightforward.

Arnd Bergmann has a interesting counterargument. In a nutshell, ARM servers aren't currently aiming for the same market as x86 servers, and as a result customers are unlikely to gain any significant benefit from shared functionality between the two.

So if there's no real benefit to users, and if there's no benefit to kernel developers, what's the point? The main one that springs to mind is that there is a benefit to distributions. Moving to UEFI means that there's a standard mechanism for distributions to interact with the firmware and configure the bootloader. The traditional ARM approach has been for vendors to ship their own version of u-boot. If that's in flash then it's not much of a problem[3], but if it's on disk then you have to ship a range of different bootloaders and know which one to install (and let's not even talk about initial bootstrapping).

This seems like the most compelling argument. UEFI provides a genuine benefit for distributions, and long term it probably provides some benefit to customers. The question is whether that benefit is worth the flux. The same distribution benefit could be gained by simply mandating a minimum set of u-boot functionality, which would seem much more straightforward. The customer benefit is currently unclear.

In the end it'll probably be a market decision. If Red Hat produce an ARM product that has these requirements, and if Suse produce an ARM product that will work with u-boot and Device Tree, it'll be up to vendors to decide whether the additional work to support UEFI/ACPI is worth it in order to be able to sell to customers who want Red Hat. I expect that large vendors like HP and Dell will probably do it, but the smaller ones may not. The customer demand issue is also going to be unclear until we learn whether using UEFI is something that customers actually care about, rather than a theoretical benefit.

Overall, I'm on the fence as to whether a UEFI requirement is going to stick, and I suspect that the ACPI requirement is tilting at windmills. There's nothing stopping vendors from providing a Device Tree blob from UEFI, and I can't think of any benefits they gain from using ACPI instead. Vendor interest in the generic parts of the ACPI spec has been tepid even in the x86 world (the vast majority of ACPI spec updates come from Microsoft and Intel, not any of the system vendors), and I don't see that changing with the introduction of a range of ARM vendors who are already happy with Device Tree.

We'll see. Linux is going to need to gain the support for UEFI and ACPI on ARM in any case, since there's already hardware shipping in that configuration. But with ARM vendors still getting to grips with Device Tree, forcing them to go through another change in how they do things is going to be hard work. Red Hat may be successful in enforcing these requirements at the cost of some vendor unhappiness, or Red Hat may find that their product doesn't boot on most of the available hardware. It's an aggressive gamble, and while it'll be interesting to see how it plays out, I'm not that optimistic.

[1] The blob could be pulled from the firmware, but it's not uncommon for it to be built into u-boot instead. This does mean that you have a device-specific u-boot even if you have a generic kernel, but that's typically true anyway.
[2] Patches have been posted for ARM UEFI support. They're not mergeable in their current form, but they should be in the near future. ACPI support is in development.
[3] Although not all u-boots are created equal - some vendors ship versions that will only boot off FAT, some vendors ship versions that will only boot off ext2. Having to special case this stuff in your installer is a pain.
Mir is Canonical's new display server. It fulfils a broadly similar role to Wayland and Android's Surfaceflinger, in that it takes final responsibility for getting pixels onto the screen. XMir is an X server that runs on top of Mir. It permits applications that know how to speak the X protocol but don't know how to speak to Mir (ie, approximately all of them at present) to run in a Mir-based environment.

For Ubuntu 13.10, Canonical are proposing to use Mir by default. This doesn't mean that most applications will be using Mir, though - instead, the default session will run XMir as a full-screen client and a normal X environment will be run on that. This lets Canonical deploy Mir without forcing anyone to update their applications, allowing them to take a gradual approach. By 14.10, Canonical expect the default Unity session to be a Mir client rather than an X client. In theory it will then be possible to run an Ubuntu system without any X applications at all, leaving XMir to do nothing other than run legacy applications.

Of course, this requires a certain amount of replumbing. X would normally be responsible for doing things like setting up the screen and pushing the pixels out to the hardware, but this is now handled by Mir instead. Where a native X server would allocate a framebuffer in video memory and render into it, XMir asks Mir for a window corresponding to the size of the screen and renders into that and then simply asks Mir to display it. This step is actually more interesting than it sounds.

Unless you're willing to throw lots of CPU at them, unaccelerated graphics are slow. Even if you are, you're going to end up consuming more power for the same performance, so XMir would be impractical if it didn't provide access to accelerated hardware graphics functions. It makes use of the existing Xorg accelerated X drivers to do this, which is as simple as telling the drivers to render into the window that XMir requested from Mir rather than into the video framebuffer directly. In other words, when displaying through XMir, you're using exactly the same display driver stack as you would be if you were using Xorg. In theory you'd expect identical performance - in practice there's a 10-20% performance hit right now, but that's being actively worked on. Fullscreen 3D apps will also currently take a hit due to there being no support for skipping compositing, which is being fixed. XMir should certainly be capable of performing around as well as native X, but there's no reason for it to be any faster.

The output drivers aren't the only part of the stack. XMir still needs some way to get input events. You might naively expect that Mir would forward input events to XMir, but the current state of affairs is that XMir loads the existing X input drivers which in turn open the input devices themselves. This explains why XMir currently shows two cursors[1] - the first cursor is the Mir cursor that's being driven by the input events Mir is receiving, while the second is the cursor being drawn by XMir in response to the input events that it's receiving[2].

The final main piece of duplicated functionality is output management. In the case where X is its own display server, it can simply ask the kernel to program the outputs. XMir can't do that, so would have to ask Mir to do it instead. Sadly, that functionality's not implemented yet. Running xrandr on Xmir gives:
Screen 0: minimum 320 x 320, current 1366 x 768, maximum 8192 x 8192
XMIR-1 connected primary 1366x768+0+0 (normal left inverted right x axis y axis) 0mm x 0mm
   XMIR mode of death[3]   60.0*+
which translates as "I have a screen that can be any size from 320x320 to 8192x8192, and have one output of unknown physical size that can only display 1366x768 at 60Hz". Other than the 1366x768, the result will be identical no matter how many outputs you have - XMir currently has no support for configuring multiple displays, and will always report 60Hz. It also has no support for placing monitors into any kind of DPMS state.

Obviously, this is still software under active development. There's three months from now until until Ubuntu is due to release with this functionality enabled by default, and the only significant missing features are xrandr and DPMS support. As long as Mir exposes interfaces for controlling those, they shouldn't be a problem to implement. To most users, the transition from native Xorg to XMir should be completely invisible.

Which is kind of the problem. What benefits does the user gain from using XMir on Mir rather than native Xorg?

Features? XMir is a total of around 1000 lines of code on top of Xorg. It implements no functionality that isn't present in Xorg.

Performance? XMir effectively is Xorg - it's the same code running the same drivers. It's not going to be any faster. At the moment it's actually slower, but that's probably fixable.

Security? Again, XMir is effectively Xorg. It still runs as root. It has the same privileges. There's no security benefit to using XMir instead of Xorg.

Like I said, the transition should be completely invisible - no drawbacks, but no benefits. What it does do is get Mir deployed on a larger number of systems, which means Canonical can both get wider testing of the basic Mir functionality and argue that the number of deployed Mir systems means hardware vendors should support it. Users won't see any of the benefits until Unity transitions to being a native Mir client, which is slated for 14.10 next year.

As a PR move, it's pretty great. The public perception is that Mir has gone from not existing to being sufficient to run a full desktop environment in very little time, and it's natural to compare it to Wayland which has been in development for longer and still isn't running a fully featured desktop session. The problem with that perception is that Mir is doing very little at the moment. It's setting an output mode (by asking the kernel to), allocating a window for XMir to draw into and then pushing that window onto the screen whenever XMir tells it that it's changed. This isn't advanced display server functionality, it's the bare minimum that a display server could possibly do[4]. Which is a shame, because Mir is significantly more functional than that, it's just not being used to do anything we couldn't already do.

In summary: XMir on Mir in Ubuntu provides no user benefits and isn't a compelling technology demo. Mir itself will permit a range of additional features, but isn't slated to be running a user session itself until 14.10. The only obvious benefit to Canonical in shipping XMir on Mir is to gain additional testing, which makes using it in 14.04, a supposedly stable and long term release, a somewhat surprising choice.

[1] Although this is in the process of being fixed
[2] This also explains why, on systems with a trackpoint and a touchpad, only the trackpoint moves the Mir cursor while both move the XMir cursor. The X touchpad driver has put the touchpad in absolute mode, which means it's now reporting events that Mir currently ignores.
[3] "XMIR mode of death" is hardcoded into XMir. I'm really hoping it's a Regular Show reference.
[4] So why hasn't anyone done this with Wayland? Mostly because, as described above, there's no technical benefit in doing so. Wayland does have an X server for compatibility purposes, and if you wanted you could run it as a full screen client and run an entire session underneath it. But you'd gain nothing by doing so. The Wayland X server is intended for running individual clients rather than an entire X session. Run an X client under Wayland and it'll pop up in its own individual window and managed by your Wayland session's window manager, just like it would under X. XMir currently has no support for this "rootless" mode - right now if you want to run X apps under Mir, you'll need to launch an entire X session with its own window manager.
Recent Intel-based systems often implement something called Intel Rapid Start Technology. Like many things with the word "Technology" in the name, there's a large part of this that's marketing. The relatively small amount of technical documentation available implies that it's tied to your motherboard chipset and CPU, but as far as I can tell it's entirely implemented in firmware and could work just as well on, say, a Cyrix on a circa 1996 SIS-based motherboard if someone wrote the BIOS code[1]. But since nobody has, we're stuck with the vendors who've met Intel's requirements and licensed the code.

The concept of IRST is pretty simple. There's a firmware mechanism for setting a sleep timeout. If you suspend your computer and this timeout expires, it'll resume. However, instead of handing control back to the OS, the firmware just copies the entire contents of RAM to a special partition and turns the computer off. Next time you hit the power button, the firmware dumps the partition contents back into RAM and resumes as if nothing had changed. This takes a few seconds longer than resume from S3 but is far faster than resume from hibernation since it starts the moment the system gets power.

At a more technical level, it's a little more complicated. The first thing to know about this feature is that it's entirely invisible unless your hard drive is set up correctly. There needs to be a partition that's at least the size of your system's physical RAM. For GPT systems, this needs to have a type GUID of D3BFE2DE-3DAF-11DF-BA-40-E3A556D89593. For MBR systems, you need a partition type of 0x84[2]. If the firmware doesn't find an appropriate partition then the OS will get no indication that the firmware supports it. Boo.

(The second thing is that it seems like it really does have to be on an SSD, and if you try to do this on spinning media your firmware will ignore it anyway)

If all the prerequisites are in place, an ACPI device with an HID of INT3392 will exist. It has four methods associated with it: GFFS, SFFS, GFTV and SFTV. GFFS returns an integer representing the events that will cause the system to wake up from S3 and suspend to SSD. The system will wake after the timeout expires if bit 0 is set, and will wake when the battery becomes critically low if bit 1 is set. The other bits appear to be unused at the moment[3]. SFFS sets the wakeup events, using the same bit values as GFFS. GFTV returns an integer containing the current wakeup timeout in minutes. SFTV sets it. Values above 1440 (ie, 24 hours) seem to be considered invalid - if I set them the value instead ends up as 10 and the timeout flag gets cleared from the wakeup events field.

I've submitted a patch that adds a sysfs interface for setting these values, and unless anyone objects it'll probably end up in 3.11. There's still the remaining question of how userspace should make use of these, and also how installers should behave when it comes to systems that support IRST. As previously mentioned, there's no obvious indication to the OS that the feature is supported unless the appropriate partition already exists. The easiest way to deal with this is for installers to default to retaining any partitions with the magic IDs, but I'm still looking into whether it's possible to get the firmware to cough up some more information so it can be created automatically even if the drive's entirely blank.

And now, having got this working on a test machine, I just need to split my Thinkpad's swap partition in half and make sure it works here as well. Woo.

[1] Note: I am not going to do this.
[2] Conveniently, the same as the partition type that APM systems used for suspend to disk back when dubstep hadn't been invented yet
[3] At least, if you attempt to set them they get ignored.
One of the goals of our work at Nebula is making it as easy as possible for someone to set up a private cloud. In an ideal world that would involve being able to just plug in the controller hardware, wire up a rack of servers and turn everything on, but right now there are cases where the default firmware configuration on the servers doesn't match our desired configuration. Plugging a console into individual servers just to set some BIOS options is (based on personal experience) about as much fun as writing a doctoral thesis on the experience of watching paint dry, so it seemed worth trying to find a way to avoid people having to deal with that. Thankfully, it turns out that the industry has come to a similar set of conclusions. Recent Dell hardware lets you use WS-MAN, which makes it easy to do things like enable security features as long as you have an authenticated connection to the iDRAC management system. This actually works out wonderfully - the first time a Dell node sends a PXE request to the Cloud Controller, it can push back some configuration changes, reboot the system and then boot it in a known good configuration. Thanks, Dell![1].

HP's slightly trickier. As far as I've been able to work out, the remotely-available programmatic interfaces only provide configuration interfaces to the iLO device itself, along with a range of chassis-level monitoring. Instead, HP provide a couple of utilities that allow the OS to change values. The first of these is called conrep. Conrep is able to modify BIOS settings, but needs an XML file which tells it which configuration values are at which addresses. That's a bit of a pain. Thankfully it's being replaced with a new tool called hprcu which is able to directly query the firmware in order to figure out which configuration options are available and which values have to be stored where in order to set them. You run hprcu once and it spits out a file containing your existing settings. You modify that, feed it back into hprcu, reboot and you're set.

Sounds pretty ideal. What's the problem? The first is that hprcu is currently only shipped as a 32-bit binary, and right now we don't deploy any 32-bit support code on our nodes. It'd be irritating to have to do so just to configure the firmware. The second is that it works by doing raw port IO, and that's not going to be possible from userspace once we've moved to a UEFI Secure Boot setup.

So, obviously, I've been working on reimplementing it. The first step was to figure out what it was actually doing. The first step was to strace it to see if it was using the kernel IPMI interface. strace showed no accesses to /dev/ipmi*. My next thought was that it could be using some sort of shared memory segment with the iLO hardware, so I used MMIOTrace to dump the memory accesses it made. Turned out there weren't many, and certainly not enough to do anything interesting. Then I went back to the strace logs and saw that it was calling iopl() a bunch, and then I got sad.

iopl() allows a userspace application to change its io privilege levels. The Linux manpage for the iopl() call is amazingly helpful, informing us with a straight face that "This call is necessary to allow 8514-compatible X servers to run under Linux." What it actually does is grant userspace access to the full range of io ports.

So, what's an io port? The x86 architecture (and some others) provide two ways to communicate with hardware. The one that's mostly used these days is to map the hardware into the same address space as memory (memory-mapped io), but x86 has an entirely separate address range that can be used. This is significantly more limited, as only 65,536 addresses are available for the entire system. Further, you can only read or write a maximum of 4 bytes in a single transaction. However, while slower and less generally useful than memory-mapped io, port io has the advantage that it's much simpler to implement. As a result, a lot of the original PC hardware was intended to be accessed via port io, and a bunch of that's still present. Want to reprogram your real-time clock? Port io. Want to read from the legacy keyboard controller? Port io. For low-bandwidth transactions, it's a completely reasonable way to implement hardware.

Applications[2] perform port io by executing in and out instructions. Since these instructions allow you to do things like, say, hit the keyboard controller directly[3], userspace is generally forbidden from calling them. Sufficiently privileged applications can call iopl() to raise their privileges and gain access to the full set of io ports. But, since in and out are CPU-level instructions, the actual port io accesses won't show up in strace. So I had to take another approach.

The correct way of doing this would be to LD_PRELOAD something that intercepted iopl(), didn't call it, let the application perform an in or out instruction, caught the fault, looked at the stack frame, called iopl(), performed the access, dumped debug details, dropped iopl() again and restored state. Because that seemed like a bunch of work, I took advantage of the fact that hprcu had separate in and out functions and used gdb. I told gdb to break on every call to in or out, dump the register state and then continue. Then I hacked up a script to parse the register dumps and tell me where the reads and writes were going. Astonishingly, it worked. And then I read the results and got really sad.

PCI setup is hard, if you're a BIOS. You've got PCI devices with large memory windows and you've got to arrange them all somehow and look you've only got 640K of RAM you can access while you're doing this and come on, seriously? So PC-type systems define a port IO interface to performing PCI configuration. Each PCI device has an address made up of a bus, a device and a function. You take a 32-bit integer, set the top bit to indicate that you want PCI access, put the bus in bits 16-23, the device in bits 11-15, the function in bits 8-10 and the configuration register on that device you want to access in bits 0-7. Then you write that to io port 0xcf8. Reads or writes to io port 0xcfc then read or write from that configuration register on that device. This lets the BIOS tell each PCI device where it's going to live in mmio address space without having to actually perform any mmio. Good work, BIOS developers.

This works fine in the BIOS, because nothing else is running while the BIOS is. It even works fine in the kernel, which uses this approach on older hardware that doesn't provide a memory-mapped mechanism to access PCI configuration registers. But it works really badly if you're doing it in userspace, because it shares a problem with all other indexed accesses. You write an address to 0xcf8. You write a value to 0xcfc. What happens if the kernel writes to 0xcf8 before you write to 0xcfc? Your access goes to some other register on some other device and suddenly you've just mapped a PCI device over the top of RAM and well I sure hope you saved all your work.

So, I wasn't thrilled to discover that hprcu was communicating with the iLO by using 0xcf8/0xcfc port io accesses. Not only was it going to stop working once we started deploying UEFI Secure Boot, it had the potential to cause really annoyingly hard to debug problems. Working out how to reimplement it became something of a priority.

By looking at the XML files it generated, and by following the port IO accesses it was performing, I could figure out pretty much how it worked. Some configuration values are stored in the real-time clock CMOS. These were just done in the standard way - write the address to port 0x70, read or write the value to port 0x71[4]. Other configuration values were stored in NVRAM, with accesses going via PCI. This was a little trickier to figure out, because sometimes different NVRAM addresses went to the same PCI address. I finally figured it out - there's a 48 byte window into NVRAM via the PCI configuration registers, and another register which chooses which set of NVRAM is visible. So, take the NVRAM address, divide by 48, write that value to register 0xa6, take the modulus, add it to 0x80 and write the desired value to that address.

So now I knew enough to be able to take an existing XML file and deploy it. Definitely progress, and I could even add support to the kernel. But I still wanted to know how hprcu generated that XML file in the first place. Running strings over the binary showed a bunch of debug output that it never actually printed, but immediately after the help text it also printed SsLlFf:HhAaTtOoDd. I've lost enough years of my life to this kind of thing to be able to identity that as a getopt format string, and there was that tantalising D option at the end that went entirely unmentioned in the help text. Re-running hprcu with the -D argument gave me huge piles of debug output, including references to a DMI entry and an $RBS table.

DMI is a standard for exposing system information, and one aspect of it is providing data from the firmware to the operating system. The OS can look for a defined signature in a fixed area of memory and then pull out a bunch of data about the hardware, including things like the vendor name, model, serial number, BIOS version and more. Some of these information tables are standardised and some are vendor-specific. hprcu was looking for a vendor-specific table and then scanning for it looking for a known signature. That table turned out to to contain a set of signatures and addresses. The address corresponding to the signature hprcu was contained a bunch of data starting with "$RBS". It also contained a huge quantity of ASCII that looked awfully like the strings in the XML file that hprcu had written. Success!

So, the rest of today was spent on working out the format of this table. This was only marginally less tedious than setting up BIOS settings on 20 servers by hand, but I've made enough progress to be able to figure out how to write a kernel-level driver for this. That's about half-done now - the parsing code is all implemented, I just need to add the sysfs glue and I am nowhere near drunk enough for that right now[5], so for now you're just getting a format description.

To start with, look for a DMI table with a type of 0xe5. This contains several entries of 4 bytes of signature, 8 bytes of address and 4 bytes of length. Find the entry with a signature of "$CRQ" and map the corresponding address. The first four bytes of the mapped region should be "$RBS". The first record is 21 bytes into the file. Each record starts with a single byte representing the type and two bytes representing the record length. Records of type 1 contain an ascii string that starts 8 bytes into the record, and have a further type number 6 bytes into the record that tells you what type of string it is. 0x05 indicates that it's a "Feature string", which defines the overall classification of the following features. 0x06 is an "Option string", which provides a human readable explanation of what this configuration value corresponds to. 0x60 is an "Optional warning string", which tells the user which further configuration may be required before the configuration change takes effect.

All other types appear to be configuration options. Bytes 3 to 6 provide the little-endian name of the configuration option. Bytes 7 and 8 provide a unique numberical identifier for the configuration option. Byte 13 indicates the type. 0x03 is stored in CMOS. 0x04 is stored in NVRAM. 0x05 appears to be some different kind of CMOS store. I haven't figured out the others. There then follows a set of choices. These vary in length depending on the type - 0x03 are 14 bytes long, 0x04 are 6 bytes long, 0x05 are 5 bytes long. Type 0x03 choices have the choice id at byte 0, the CMOS offset at byte 2, the mask at byte 3 and the value at byte 4. Type 0x04 choices have the choice id at byte 0, the nvram offset at bytes 1 and 2, the mask at byte 3 and the value at byte 4. Type 0x05 choices have the choice id at byte 0, the cmos offset at byte 1, the mask at byte 2 and the value at byte 3. There are optional flags following each choice - if the final byte of the choice isn't 0, there then follow 6 bytes of flags. The first four bytes provide the name of a configuration option (little-endian, again), the fifth byte refers to a choice on that option and the sixth byte indicates what kind of flag. 0x1 appears to indicate that the option is mutually exclusive with that other option. Examples include the embedded serial port and virtual serial port options, where it's impossible to map both to the same address at once. A choice can have multiple flags.

After a configuration option, there'll be a set of option string records. In turn, these correspond to the previous choices - if a configuration option had three choices, there'll be three option string records. I haven't figured out whether there's a stronger way of binding the option string records to the configuration choice yet. The other thing I haven't entirely figured out are the details of configuring the platform to perform a cold reboot on the next power cycle, but the io traces for that don't look too bad.

The aim is to provide a kernel driver that exposes all these configuration options via sysfs, including indicating the current value, the available values and letting the user set a new value. In an ideal world we'd have a wonderfully generic interface to this kind of functionality, but I'm (sadly) not sure that that's possible.

And that's how I spent the past two days.

[1] (Thell)
[2] Or, indeed, kernels
[3] And, hence, read your keystrokes
[4] Which has exactly the same problems as the PCI access, in that if you happen to ask the clock what the time is while hprcu is accessing CMOS, at least one of you is going to get very confused
[5] Obviously, I would never write kernel code while drunk. You're certainly not running any of it in your enterprise kernel.
Mir is Canonical's equivalent to Wayland - a display server, responsible for getting application pixmaps onto a screen. It's intended to scale from mobile devices to the desktop, and as such is expected to turn up in Ubuntu Phone before too long[1]. There's already plenty of discussion about whether the technical differences between Wayland and Mir are sufficient to justify Canonical going their own way, so I'm not planning on talking about that.

Like many Canonical-led projects, Mir is under GPLv3 - a strong copyleft license. There's a couple of aspects of GPLv3 that are intended to protect users from being unable to make use of the rights that the license grants them. The first is that if GPLv3 code is shipped as part of a user product, it must be possible for the user to replace that GPLv3 code. That's a problem if your device is intended to be locked down enough that it can only run vendor code. The second is that it grants an explicit patent license to downstream recipients, permitting them to make use of those patents in derivative works.

One of the consequences of these obligations is that companies whose business models depend on either selling locked-down devices or licensing patents tend to be fairly reluctant to ship GPLv3 software. In effect, this is GPLv3 acting entirely as intended - unless you're willing to guarantee that a user can exercise the freedoms defined by the free software definition, you don't get to ship GPLv3 material. Some companies have decided that shipping GPLv3 code would be more expensive than either improving existing code under a more liberal license or writing new code from scratch. Android's a pretty great example of this - it contains no GPLv3 code, and even GPLv2 code (outside the kernel) is kept to a minimum.

Which, given Canonical's focus on pushing Ubuntu into GPLv3-hostile markets, makes the choice of GPLv3 an odd one. This isn't a problem as long as they're the sole copyright holder, because the copyright holder is obviously free to ship their code under as many licenses as they want. But Canonical still aim to foster community involvement, and ideally that includes accepting external contributions to their code. If Canonical simply accepted those contributions under GPLv3 then they'd no longer have the right to relicense the entire codebase, so any contributions are only accepted if the contributor has signed a Contributor License Agreement.

Canonical's CLA is pretty simple. In essence, it grants Canonical the right to use, modify and distribute your code, and it grants Canonical a patent license under any patents you own that may cover the code in question. But, most importantly, it grants Canonical the right to relicense your contribution under their choice of license. This means that, despite not being the sole copyright holder, Canonical are free to relicense your code under a proprietary license.

Given Canonical's market goals, this makes sense. They can relicense Mir (and any other GPLv3 projects they own) under licenses that keep their hardware partners happy, and they can ship in the phone market. Everyone's a winner.

Except, if Canonical want to ship proprietary versions, why not just license Mir under a license that permits that in the first place? This is where the asymmetry comes in. The Android userland is released under a permissive license that allows anyone to take Google's code, modify it as they wish and ship it on whatever hardware they want. I could legally start a company that provided customised versions of Android to phone vendors without them having any GPLv3 concerns. I won't be able to do that with Ubuntu Phone.

I'm a fan of GPLv3. I think the provisions it contains to support user freedom are important. I hate the growing trend of using free software to build devices that are, effectively, impossible for the end user to modify. If Canonical were releasing software under GPLv3 because of a commitment to free software then that would be an amazing thing. But it's pretty much impossible to square the CLA's requirement that contributors grant Canonical the right to ship under a proprietary license with a commitment to free software. Instead you end up with a situation that looks awfully like Canonical wanting to squash competition by making it impossible for anyone else to sell modified versions of Canonical's software in the same market.

Canonical aren't doing anything illegal or immoral here. They're free to run their projects in any way they choose. But retaining the right to produce proprietary versions of external contributions without granting equivalent reciprocal rights isn't consistent with caring about free software or contributing to the wider Linux community, especially if it means you get to exclude those external contributors from the market you're selling their code into.

(Edit to add: a friend in the contracting industry points out that it also prevents vendors who won't ship GPLv3 from using external contractors to work on Mir - they have to go to Canonical, because only Canonical can relicense contributions under a proprietary license.)

[1] Right now Ubuntu Phone is using Surfaceflinger, the Android display server, but that's apparently just an interim solution.
Since I wrote this, we've made some worthwhile progress on avoiding damaging Samsung hardware. The first is that the samsung-laptop driver appeared to be causing the firmware to attempt to write to an area of memory that was marked in the chipset, triggering a Machine Check Exception. That was what generated the pstore output that caused the problem originally. The driver now refuses to load if EFI is enabled, which avoids the problem. It's not ideal, since it's currently the only mechanism we have for certain functionality on Samsung laptops, but there you go.

The second problem was that avoiding crashing on boot didn't actually fix the problem in any fundamental way. Even with pstore disabled, it was possible for userspace to fill the nvram and trigger the same problem. Our first approach to this was to prevent any writes to nvram if the UEFI QueryVariableInfo() call reported that more than 50% of the nvram storage space would be used. That was safe, but led to another issue. The nvram storage area is typically implemented as part of the same flash chip as the firmware. Flash isn't arbitrarily accessible - changing the contents of a block typically involves rewriting the entire block. It's impractical to rewrite the entire nvram area on every write, so what actually happens is that deleting variables just results in them being marked as inactive but doesn't actually free up the space. The firmware can later perform some sort of garbage collection to free it up.

This caused us problems, since inactive space that hasn't been garbage collected yet isn't actually available, and as a result firmware implementations tend to count it as used. Say you had 64KB of nvram and wrote 32KB of variables. We'd then refuse to write any more because you'd drop below 50%. So you delete 16KB of the variables you've created and try again. Unfortunately, the firmware still thinks that there's 32KB in use and Linux would still refuse.

If you were lucky, rebooting would trigger a garbage collection run. If you weren't, it wouldn't. Problematic. Our next approach was to try to account for the space actually actively used by the variables, rather than relying on what the firmware told us via QueryVariableInfo(). This seems simple enough - just add up the size of all the variables and subtract that from the overall size to determine how much of the "used" space is actually just old inactive variables that can be ignored. However, there's still some problems there. The first is that each variable has some additional overhead associated with it, and the size of that overhead varies depending on the system vendor. We had to make a conservative guess, which could cause problems if systems had large numbers of small variables. The second is that the only variables the kernel can see are those that are flagged as runtime-visible. There may also be a significant quantity of nvram used to store variables that are only visible in boot services code. We could work around this by adding up sizes while we're still in boot services code, but on some systems calling QueryVariableInfo() before ExitBootServices() results in later calls to GetNextVariable() jumping to invalid addresses and crashing the kernel. Not a great approach.

Meanwhile, Samsung got back to us and let us know that their systems didn't require more than 5KB of nvram space to be available, which meant we could get rid of the 50% value and replace it with 5KB. The hope was that any system that booted with only 5KB of space available in nvram would trigger a garbage collection run. Unfortunately, it turned out that that wasn't true - some systems will only trigger garbage collection if the OS actually makes an attempt to write a variable that won't otherwise fit.

Hence this patch. The new approach is to ask the firmware how much space is available. If the size of the new variable would reduce this to less than 5K, we attempt to create a variable bigger than the remaining space. This should cause the firmware to realise that it's out of room and either (depending on implementation) perform a garbage collection run at runtime or set a flag that will cause the system to perform garbage collection on the next reboot. We then call QueryVariableInfo() again to see whether a garbage collection run actually happened, and if so check whether we now have enough space. If so, we go ahead and write the variable. If not, we tell userspace that there's not enough space.

This seems to work in all the situations I've tested, and it should avoid ending up in a situation where a Samsung can end up bricked. However, it's firmware, so who knows whether it's going to break things for someone else.
There's now no shortage of Linux distributions that support Secure Boot out of the box, so that's a mostly solved problem. But even if your distribution supports it entirely you still need to boot your install media in the first place.

Hardware initialisation is a slightly odd thing. There's no specification that describes the state ancillary hardware has to be in after firmware→OS handover, so the OS effectively has to reinitialise it again. This means that certain bits of hardware end up being initialised twice, and that's slow in some cases. The most obvious is probably USB, which has various timeouts as you wait for hardware to settle. Full USB support in the firmware probably adds a couple of seconds to boot time, and it's arguably wasted because the OS then has to do the same thing (but, thankfully, can at least do other things at the same time). So, looking for USB boot media takes time, and since the overwhelmingly common case is that users don't want to boot off USB, it's time that's almost always wasted.

One of the requirements for Windows 8 certified hardware is that it must complete firmware initialisation within a specific amount of time, something that Microsoft refer to as "Fast Boot". Meeting these requirements effectively makes it impossible to initialise USB, and it's likely that certain other things will also be skipped. If you've got a USB keyboard then this obviously means that your keyboard won't work until the OS starts, but even i8042 setup takes time and so some laptops with traditional PS/2-style keyboards may not set it up. That means the system will ignore the keyboard no matter how much you hammer it at boot, and the firmware will boot whichever OS it finds.

For a newly purchased device, that's going to be Windows 8. It's not too much of a problem with a fully installed Windows 8, since you can hold down shift while clicking the reboot icon and get a menu that lets you reboot into the firmware menu. Windows sets a flag in a UEFI variable and reboots the system, the firmware sees that flag and does full hardware initialisation and then drops you into the setup environment. It takes slightly longer to get into the firmware, but that's countered by the time you save every time you don't want to get into the firmware on boot.

So what's the problem? Well, the Windows 8 setup environment doesn't offer that reboot icon. Turn on a brand new Windows 8 system and you have two choices - agree to the Windows 8 license, or power the machine off. The only way to get into the firmware menu is to either agree to the Windows 8 license or to disassemble the machine enough that you can unplug the hard drive[1] and force the system to fall back to offering the boot menu.

I understand the commercial considerations that result in it ranging from being difficult to impossible to buy new hardware without Windows pre-installed, but up until now it was still straightforward to install an alternative OS without agreeing to the Windows license. Now, installing alternative operating systems on many new systems will require you to give up certain rights even if you want nothing other than to reach the system firmware menu.

I'm firmly of the opinion that there are benefits to Secure Boot. I'm also in favour of setups like Fast Boot. But I don't believe that anyone should be forced to agree to a EULA purely in order to be able to boot their own choice of OS on a system that they've already purchased.

[1] Which is a significant and probably warranty-voiding exercise on many systems, and that's assuming that it's not an SSD soldered to the motherboard…
I've been working on TPMs lately. It turns out that they're moderately awful, but what's significantly more awful is basically all the existing documentation. So here's some of what I've learned, presented in the hope that it saves someone else some amount of misery.

What is a TPM?

TPMs are devices that adhere to the Trusted Computing Group's Trusted Platform Module specification. They're typically microcontrollers[1] with a small amount of flash, and attached via either i2c (on embedded devices) or LPC[2] (on PCs). While designed for performing cryptographic tasks, TPMs are not cryptographic accelerators - in almost all situations, carrying out any TPM operations on the CPU instead would be massively faster[3]. So why use a TPM at all?

Keeping secrets with a TPM

TPMs can encrypt and decrypt things. They're not terribly fast at doing so, but they have one significant benefit over doing it on the CPU - they can do it with keys that are tied to the TPM. All TPMs have something called a Storage Root Key (or SRK) that's generated when the TPM is initially configured. You can ask the TPM to generate a new keypair, and it'll do so, encrypt them with the SRK (or another key descended from the SRK) and hand it back to you. Other than the SRK (and another key called the Endorsement Key, which we'll get back to later), these keys aren't actually kept on the TPM - the running OS stores them on disk. If the OS wants to encrypt or decrypt something, it loads the key into the TPM and asks it to perform the desired operation. The TPM decrypts the key and then goes to work on the data. For small quantities of data, the secret can even be stored in the TPM's nvram rather than on disk.

All of this means that the keys are tied to a system, which is great for security. An attacker can't obtain the decrypted keys, even if they have a keylogger and full access to your filesystem. If I encrypt my laptop's drive and then encrypt the decryption key with the TPM, stealing my drive won't help even if you have my passphrase - any other TPM simply doesn't have the keys necessary to give you access.

That's fine for keys which are system specific, but what about keys that I might want to use on multiple systems, or keys that I want to carry on using when I need to replace my hardware? Keys can optionally be flagged as migratable, which makes it possible to export them from the TPM and import them to another TPM. This seems like it defeats most of the benefits, but there's a couple of features that improve security here. The first is that you need the TPM ownership password, which is something that's set during initial TPM setup and then not usually used afterwards. An attacker would need to obtain this somehow. The other is that you can set limits on migration when you initially import the key. In this scenario the TPM will only be willing to export the key by encrypting it with a pre-configured public key. If the private half is kept offline, an attacker is still unable to obtain a decrypted copy of the key.

So I just replace the OS with one that steals the secret, right?

Say my root filesystem is encrypted with a secret that's stored on the TPM. An attacker can replace my kernel with one that grabs that secret once the TPM's released it. How can I avoid that?

TPMs have a series of Platform Configuration Registers (PCRs) that are used to record system state. These all start off programmed to zero, but applications can extend them at runtime by writing a sha1 hash into them. The new hash is concatenated to the existing PCR value and another sha1 calculated, and then this value is stored in the PCR. The firmware hashes itself and various option ROMs and adds those values to some PCRs, and then grabs the bootloader and hashes that. The bootloader then hashes its configuration and the files it reads before executing them.

This chain of trust means that you can verify that no prior system component has been modified. If an attacker modifies the bootloader then the firmware will calculate a different hash value, and there's no way for the attacker to force that back to the original value. Changing the kernel or the initrd will result in the same problem. Other than replacing the very low level firmware code that controls the root of trust, there's no way an attacker can replace any fundamental system components without changing the hash values.

TPMs support using these hash values to decide whether or not to perform a decryption operation. If an attacker replaces the initrd, the PCRs won't match and the TPM will simply refuse to hand over the secret. You can actually see this in use on Windows devices using Bitlocker - if you do anything that would change the PCR state (like booting into recovery mode), the TPM won't hand over the key and Bitlocker has to prompt for a recovery key. Choosing which PCRs to care about is something of a balancing act. Firmware configuration is typically hashed into PCR 1, so changing any firmware configuration options will change it. If PCR 1 is listed as one of the values that must match in order to release the secret, changing any firmware options will prevent the secret from being released. That's probably overkill. On the other hand, PCR 0 will normally contain the firmware hash itself. Including this means that the user will need to recover after updating their firmware, but failing to include it means that an attacker can subvert the system by replacing the firmware.

What about using TPMs for DRM?

In theory you could populate TPMs with DRM keys for media playback, and seal them such that the hardware wouldn't hand them over. In practice this is probably too easily subverted or too user-hostile - changing default boot order in your firmware would result in validation failing, and permitting that would allow fairly straightforward subverted boot processes. You really need a finer grained policy management approach, and that's something that the TPM itself can't support.

This is where Remote Attestation comes in. Rather than keep any secrets on the local TPM, the TPM can assert to a remote site that the system is in a specific state. The remote site can then make a policy determination based on multiple factors and decide whether or not to hand over session decryption keys. The idea here is fairly straightforward. The remote site sends a nonce and a list of PCRs. The TPM generates a blob with the requested PCR values, sticks the nonce on, encrypts it and sends it back to the remote site. The remote site verifies that the reply was encrypted with an actual TPM key, makes sure that the nonce matches and then makes a policy determination based on the PCR state.

But hold on. How does the remote site know that the reply was encrypted with an actual TPM? When TPMs are built, they have something called an Endorsement Key (EK) flashed into them. The idea is that the only way to have a valid EK is to have a TPM, and that the TPM will never release this key to anything else. There's a couple of problems here. The first is that proving you have a valid EK to a remote site involves having a chain of trust between the EK and some globally trusted third party. Most TPMs don't have this - the only ones I know of that do are recent Infineon and STMicro parts. The second is that TPMs only have a single EK, and so any site performing remote attestation can cross-correlate you with any other site. That's a pretty significant privacy concern.

There's a theoretical solution to the privacy issue. TPMs never actually sign PCR quotes with the EK. Instead, TPMs can generate something called an Attestation Identity Key (AIK) and sign it with the EK. The OS can then provide this to a site called a PrivacyCA, which verifies that the AIK is signed by a real EK (and hence a real TPM). When a third party site requests remote attestation, the TPM signs the PCRs with the AIK and the third party site asks the PrivacyCA whether the AIK is real. You can have as many AIKs as you want, so you can provide each service with a different AIK.

As long as the PrivacyCA only keeps track of whether an AIK is valid and not which EK it was signed with, this avoids the privacy concerns - nobody would be able to tell that multiple AIKs came from the same TPM. On the other hand, it makes any PrivacyCA a pretty attractive target. Compromising one would not only allow you to fake up any remote attestation requests, it would let you violate user privacy expectations by seeing that (say) the TPM being used to attest to HolyScriptureVideos.com was also being used to attest to DegradingPornographyInvolvingAnimals.com.

Perhaps unsurprisingly (given the associated liability concerns), there's no public and trusted PrivacyCAs yet, and even if they were (a) many computers are still being sold without TPMs and (b) even those with TPMs often don't have the EK certificate that would be required to make remote attestation possible. So while remote attestation could theoretically be used to impose DRM in a way that would require you to be running a specific OS, practical concerns make it pretty difficult for anyone to deploy that at any point in the near future.

Is this just limited to early OS components?

Nope. The Linux kernel has support for measuring each binary run or each module loaded and extending PCRs accordingly. This makes it possible to ensure that the running binaries haven't been modified on disk. There's not a lot of distribution infrastructure for setting this up, but in theory a distribution could deploy an entirely signed userspace and allow the user to opt into only executing correctly signed binaries. Things get more interesting when you add interpreted scripts to the mix, so there's still plenty of work to do there.

So what can I actually use a TPM for?

Drive encryption is probably the best example (Bitlocker does it on Windows, and there's a LUKS-based implementation for Linux here) - while in theory you could do things like use your TPM as a factor in two-factor authentication or tie your GPG key to it, there's not a lot of existing infrastructure for handling all of that. For the majority of people, the most useful feature of the TPM is probably the random number generator. rngd has support for pulling numbers out of it and stashing them in /dev/random, and it's probably worth doing that unless you have an Ivy Bridge or other CPU with an RNG.

Things get more interesting in more niche cases. Corporations can bind VPN keys to corporate machines, making it possible to impose varying security policies. Intel use the TPM as part of their anti-theft technology on education-oriented devices like the Classmate. And in the cloud, projects like Trusted Computing Pools use remote attestation to verify that compute nodes are in a known good state before scheduling jobs on them.

Is there a threat to freedom?

At the moment, probably not. The lack of any workable general purpose remote attestation makes it difficult for anyone to impose TPM-based restrictions on users, and any local code is obviously under the user's control - got a program that wants to read the PCR state before letting you do something? LD_PRELOAD something that gives it the desired response, or hack it so it ignores failure. It's just far too easy to circumvent.


TPMs are useful for some very domain-specific applications, drive encryption and random number generation. The current state of technology doesn't make them useful for practical limitations of end-user freedom.

[1] Ranging from 8-bit things that are better suited to driving washing machines, up to full ARM cores
[2] "Low Pin Count", basically ISA without the slots.
[3] Loading a key and decrypting a 5 byte payload takes 1.5 seconds on my laptop's TPM.