[personal profile] mjg59
Some of my recent time has been devoted to making our boot media more Mac friendly, which has entailed rather a lot of rebooting. This would have been fine, if tedious, except that some number of boots would fall over with either a clearly impossible kernel panic or userspace segfaulting in places that made no sense. Something was clearly wrong. Crashes that shouldn't happen are generally an indication of memory corruption. The question is how that corruption is being triggered. Hunting that down wasn't terribly easy.

My first thought was that we were possibly managing to load the kernel over a region used by UEFI code. UEFI defines two types of code - boot services and runtime services. While runtime services code and data must be preserved by the OS, in theory boot services code and data is available to the OS once the firmware has exited. In practice, that's not true. It seemed entirely possible that the kernel might be ending up on top of some of that boot services code or data and getting trodden on. Grub now has code to avoid putting the kernel on boot services, so testing the latest code seemed like a good plan. But no, crashes still happened.

That pretty much ruled out the bootloader. My next thought was that executing some of the firmware code was triggering a write to some other memory that contained the kernel. Josh Boyer suggested the next trick, which was to try marking the kernel read-only to see whether anything was hitting it. x86 lets you mark pages as read-only - any attempt to write to them should take a fault. UEFI functions are executed in the context of the kernel, so share the same page tables. That let me rule this out, since everything still went just as wrong and I wasn't taking an extra fault first.

However, at this point I was reasonably happy that it wasn't the kernel itself being overwritten - faults were occurring in userspace code as well. That was a pretty strong indication that what was happening was continuing to happen once userspace had started, so it wasn't a direct response to a firmware call. I made sure of that by stubbing out all the calls that could be triggered after kernel initialisation, and saw the same failures. Once all attempts to be clever have failed, it's time to just start using brute force. The kernel lets you reserve areas of RAM by passing arguments like memmap=0xlength$0xstart to block length bytes starting at start from being used. It took a while, but I finally found a 256MB range that made a difference - reserving it resulted in the machine booting reliably, letting the OS use it resulted in occasional crashes.

Definite progress. Comparing that memory range to the EFI memory map was helpful. There were several blocks of UEFI boot services data present there, which really seemed like too much of a coincidence. By reserving each of them in turn, I'd traced it down to a single 31MB region of boot service data - that is, memory reserved by the firmware for use by the UEFI boot services. Per spec, this is available to the OS once the boot environment has been exited. Nothing other than the OS should be touching this after boot, but something clearly was. Tracking down what was far easier than I expected, although the first attempt was a failure. Setting it read-only should have triggered a fault, but didn't. That was rather confusing. But, rather than give up, I patched the kernel to fill the region with 0xff at kernel init. Then I booted the system, read it back and looked for values that weren't 0xff. I got this:
00000000  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
*
001568a0  ff ff ff ff ff ff ff ff  ff ff ff ff 84 00 00 00  |................|
001568b0  00 20 a7 ac 46 00 00 00  00 00 00 00 00 00 06 01  |. ..F...........|
001568c0  c2 0b 0c 00 ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
001568d0  ff ff 0a 04 f0 03 82 0d  40 00 00 00 ff ff ff ff  |........@.......|
001568e0  ff ff 00 21 00 36 9a 80  ff ff ff ff ff ff 00 7e  |...!.6.........~|
001568f0  00 09 43 48 41 2d 47 75  65 73 74 01 04 02 04 0b  |..CHA-Guest.....|
00156900  16 32 08 0c 12 18 24 30  48 60 6c 2d 1a 0e 18 1a  |.2....$0H`l-....|
00156910  ff ff 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00156920  00 00 00 00 00 00 00 dd  09 00 10 18 02 00 10 01  |................|
00156930  00 00 dd 1e 00 90 4c 33  0e 18 1a ff ff 00 00 00  |......L3........|
00156940  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00156950  00 00 bd ea f8 b3 ff ff  ff ff ff ff ff ff ff ff  |................|
00156960  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
*
022fb000
That's a lot of 0xffs (around 31MB of them) with one small section that contains an 802.11 probe packet with the SSID of the hospital across the road from my house. Apple support network booting off wireless networks. It seems that the firmware brought up the wireless card, associated with this network (it's the only public one nearby) and then left the card DMAing packets into RAM. The read-only page attribute only applies to CPU-initiated accesses, so it could do this without triggering a page fault. It also explained why it was so random - whether memory corruption occurred would depend on whether a packet appeared between that memory being used by the OS and the kernel reinitialising the wireless card. It certainly explains why I couldn't reproduce it when I left the machine repeatedly rebooting on the bus home.

How do we fix this? Unsure. With luck disconnecting the UEFI driver in the bootloader should quiesce the hardware, but without testing I'm not sure of that yet. For now it's just another example of firmware managing to break expectations in deeply strange ways.

You always have the best war stories

Date: 2012-03-17 11:30 pm (UTC)
From: (Anonymous)
I'm so happy I don't have your job.

Re: You always have the best war stories

Date: 2012-03-18 06:17 pm (UTC)
From: (Anonymous)
Maybe this can be of interest to you:

http://os.inf.tu-dresden.de/vfiasco/
http://os.inf.tu-dresden.de/L4Re/overview.html
http://os.inf.tu-dresden.de/L4Re/doc/
http://demo.tudos.org/

enjoy :)
kind regards
Remo

Re: You always have the best war stories

Date: 2012-03-18 07:30 pm (UTC)
From: (Anonymous)
Because microkernels are so good at protecting you from HW DMA'ing all over your code...

Nasty

Date: 2012-03-18 12:14 am (UTC)
From: [identity profile] https://www.google.com/accounts/o8/id?id=AItOawnuDhEswDRWQ1vGgV6F7QVSJBWiiNLfXJk
Sounds like this might be a common problem in future. I suggest your solution is left in the kernel to assist anyone else trying to debug this type of problem.

Um...

Date: 2012-03-18 12:50 am (UTC)
dreamatdrew: The iconic all-in-one Apple computer icon, frowning and crying. (Sad Mac)
From: [personal profile] dreamatdrew
Ok, let me see if I'm getting this straight.

That section of memory is used by BootSrv (specifically part of the boot device chooser or the wireless firmware I'm guessing, I'm kinda curious to know if thats right), which gets data punted into it (by the wireless firmware?), but the boot-time wireless network handling doesn't necessarily notice that BootSrv has ended so it should cut that shit out right now, right?

Loading the wireless driver?

Date: 2012-03-18 01:50 am (UTC)
From: [identity profile] https://www.google.com/accounts/o8/id?id=AItOawlHalXbDc--PzIhlh_G6Hxl7obRoUuYwy4
Does loading the wireless driver make this go away (as long as it gets loaded before it dies...)? I guess if it does it's pretty counter-intuitive since it's normally a case of removing drivers to see if a problem goes away.

Truly stupid

Date: 2012-03-18 03:24 am (UTC)
From: (Anonymous)
Gotta love having a ton of spec when a captive hw maker doesn't give a darn.
My bet is that Apple simply 'knows' not to use any UEFI memory until all devices have been reinitialized by the OS - or they even have a way to pass that memory region back to the OS.
Can we have Linux reserve all such regions until we walked the list of known devices and put them in a known state? At least for Macs?

Dirk

Re: Truly stupid

Date: 2012-03-18 04:05 am (UTC)
From: (Anonymous)
We don't need to initialize the drivers, we just need to enumerate the bus and turn off stray bus masters.

-hpa

Bootloader really does not dig us out properly.

Date: 2012-03-18 10:03 pm (UTC)
From: (Anonymous)
There is a reason. We need a boot option that will do what you did on systems that are problem. Leave the boot services memory alone until we know what is causing trouble.

Next is lock the boot services memory until after the kernel knows it has passed a safe point in init that should have caught up with rogues normally by reseting everything that can be. To reset a card you don't need its firmware thank you pci spec. Once the card is reset it will forget all dma transfers it had been requested todo.

Remember after resetting everything most likely will lose like the EFI provided file-system driver and other things as well. So when you do the reset you better not need anything EFI is providing.

Reason for this way what comes to my mind is firmware updates so cure one driver get a new problem driver.

So the wifi driver is still on what other dma transfers are still on we don't exactly know. This is why I see the flush everything point is the only way. Really there is no reason why the Linux kernel could not be altered to be directly loaded by EFI.

I/OMMU

Date: 2012-03-18 08:35 am (UTC)
From: (Anonymous)
I'm not really sure the mac you're testing on has Vt-d but it might have helped to see the DMA accesses (though in this case, I/O MMU is programmed when the driver is loaded, so...)

UEFI: was it worth it?

Date: 2012-03-18 08:36 am (UTC)
From: (Anonymous)
Now don't get me wrong. We gotta live with it.

But from a detached engineering perspective: was it worth it pushing the equivalent of a full blown OS (in terms of complexity) down to the BIOS level?

How many nasty bugs await us from there, given that the BIOS makers won't have the chance to afford the QA typically given to OSes (and that is barely sufficient!)?

Re: UEFI: was it worth it?

Date: 2012-03-18 01:43 pm (UTC)
From: (Anonymous)
Of course it isn't worth it. The #1 reason why UEFI is so bloated is because Intel thought it was a good idea to decouple hardware drivers from the OS, or more succinctly, because $unnamed_vendors want control of your hardware no matter what software you're using. Now you need to match that (terribly wrong) idea with the primary benefit of Linux not having a stable kernel ABI: hardware manufacturers do not have the expertise to write fault-free drivers before launch, and do not care afterwards.

What we will end up with is a perennial stream of randomly failing hardware with no chance to fix it. Your only hope of getting your driver fixed is the hardware manufacturer, who cannot be arsed to provide it since once you've bought their product, you are no longer their customer.

Re: UEFI: was it worth it?

Date: 2012-03-19 09:44 am (UTC)
From: (Anonymous)
"UEFI drivers are completely unavailable once the OS starts."

I am not a HW expert but I have read that UEFI drivers could be available to the OS for basic services. Is it possible/required or forbidden?

Re: UEFI: was it worth it?

Date: 2012-03-18 04:30 pm (UTC)
From: (Anonymous)
... who cannot be arsed to provide it since once you've bought their product, you are no longer their customer.

And that seems to be the core point.

Make it their problem.

UEFI scribbling into your memory? Trying to call into freed boot services? Invalid ACPI tables? Missing or inconsistent ASPM settings? => Device is broken => Give it back and demand a refund.

The sad part, that this strategy (while simple and in my eyes even the correct answer to the problem) will not work. Most people will never notice how many workarounds an OS or other software is using to make everything work. And if Linux would print out "your device is broken, get the vendor to repair it", the people would probably blame Linux and not the vendor. And then the vendors will start to sell "Windows only devices" / "MacOS only devices" and any standard violations will be "its supposed to do that". :-(

Re: UEFI: was it worth it?

Date: 2012-03-18 06:58 pm (UTC)
From: (Anonymous)
EFI is a abortion, true, and the more firmware code you inject into a system the more firmware bugs you introduce....

BUT... the hardware vendor can only possibly test the configurations that they expect people to actually use. Once you start installing your own OSes on the hardware then you are using a unsupported configuration and this means that you are on your own for testing these sorts of things. If you find bugs maybe the vendor will care, maybe not.

The ultimate solution, of course, is to buy hardware from vendors that actually expect you to run the configurations you want. Meaning if you want to run Linux on a laptop then you buy a laptop the vendor that has Linux as a _expressly_ supported configuration.

I am not trying to downplay the attempts of above kernel developer struggling to get Linux to work properly on Apple hardware. What he is doing is very valuable as it is finding problems and fixing them and this will benefit more then just Apple hardware users... I am just saying that for the average user that wants their stuff to 'just work' purchasing Apple laptop to run Linux is probably the last thing you'd want to do. There are a lot of Linux-friendly vendors out there.

Re: UEFI: was it worth it?

Date: 2012-03-18 08:13 pm (UTC)
From: (Anonymous)
...the hardware vendor can only possibly test the configurations that they expect people to actually use.
...buy hardware from vendors that actually expect you to run the configurations you want.

That is the problem, not the solution.

Vendors only test "boots Windows" and ship it, because its cheaper to not do any more test. And they do this, because they know, they will get away with it, because "nobody" is complaining about it.

If someone is claiming to implement UEFI, then they better really do implement UEFI. And test against the UEFI spec. And not implement something-not-quite-unlike-UEFI that is just able to boot Windows and/or MacOS.

Nobody is demanding that the vendors try every possible software configuration. That is obvious impossible, as with any somewhat open interface. But what they do need to do (and should be able to do): Test their side of the interface. The problem here (and in all the other cases that count as "the firmware screwing you) is that this implementation is breaking this interface. And not that the combination of this implementation and linux is breaking. If Linux is violating the UEFI (or any other spec), then fix Linux. But if the vendor is violating the spec that he is claiming to support, then it is his duty to fix this. Independent of what this violation is breaking.

And buying from a Linux supporting vendor will not help you, if that vendor only tests "boots Linux". Because the next kernel version can change the order of device initializations or its address space or ... And then it will break.


I think it would be terrible step backwards to switch from standards (even bloated ones like UEFI) to vendor approved OS lists. Because that would be the consequence of your suggestion. Then the vendors can cut even the few remaining later fixes, because the device was always only sold as "works with Windows version XYZ". Want to upgrade to Windows 8? Better buy a new board.

It might still work for Microsoft, because they can exert enough pressure to the vendors because of there market share (-> see secure boot and who will probably be the only firm that gets their keys into any hardware). But even with "Linux-friendly" vendors, suspect you be stuck to a specific kernel soon. Just look how "easy" it is to update the kernel (or system libraries) on an android phone.

Re: UEFI: was it worth it?

Date: 2012-03-20 05:22 pm (UTC)
From: (Anonymous)
2 things:

1. This is a mac. All bets are off. If you want something standards compliant buy something else.

2. UEFI "windows" systems generally behave much better (compared to macs), and from a vendor perspective, why is windows not a good test of UEFI? Windows is by far the most widely used UEFI supporting OS. Much of the spec is open to interpretation. Your best bet it so support the most widely used interpretation.

Re: UEFI: was it worth it?

Date: 2012-03-26 10:17 pm (UTC)
From: (Anonymous)
Well Macs implement pre-UEFI, EFI and at that something very proprietary.

http://support.apple.com/kb/HT1237
http://refit.sourceforge.net/info/apple_efi.html

Macs are using EFI 1.10

Any assumption about boot services behavior in Macs even though it's Intel platform can't be made for Windows PCs.

It's a nice find though on errant behavior of EFI. (Notice not blaming Apple here), it could be the UEFI UNDI driver by that IHV in the Mac itself not properly behaving.

Which firmware is this?

Date: 2012-03-18 02:19 pm (UTC)
From: (Anonymous)
Can you confirm whether this is just one specific machine, or all the recent firmwares?

Because I've got a MacBookPro7,1, and I'm seeing a "EFI Firmware update 2.5" in the MacOSX software update, which I was just about to install when I saw your post.

"This update enables Lion Recovery from an Internet connection on MacBook Pro (13-inc, Mid 2010) models."

Will installing this break my Linux and render it unstable/unusable?

Re: Which firmware is this?

Date: 2012-03-18 05:54 pm (UTC)
From: (Anonymous)
I'm booting with refit -> grub-efi - so yeah, could be an issue. Haven't seen any wonkiness with the current setup. I'd imagine this problem would manifest itself on boot, before the drivers are loaded, right?

Off-topic: Apple is full of weirdness. Just two days ago decided to update to Lion, but it wouldn't let me at first, because MacOS likes to have 128MB unallocated between partitions. Even booting from DVD (Snow Leopard/Lion) was impossible. Had to use target disk mode to shrink a partition. Oh well - at least target disk mode is a pretty cool feature for a firmware to have.

About UEFI

Date: 2012-03-18 08:59 pm (UTC)
From: (Anonymous)
http://www.uefi.org/specs/

It is very annoying and time consuming, but the best way to deal with this totally different system is to actually learn the UEFI and its quirks, and use the tools to see what's going on.

Re: About UEFI

Date: 2012-03-18 09:47 pm (UTC)
From: (Anonymous)
I'm pretty sure that MJG knows the specs forwards, backwards and sideways.

Which is pretty scary, since it is HUGE.

using IOMMU to prevent DMA access

Date: 2012-03-18 10:10 pm (UTC)
From: (Anonymous)
If your platform has IOMMU (Intel VT-D) support, then you can turn that on in bootloader (Ex. grub) which should prevent any memory corruption resulting by DMA.

Date: 2012-03-20 05:57 pm (UTC)
From: (Anonymous)
The famous example is some old Aptio GOP wrapper for legacy VGA BIOS,
some operation will allocate boot-service type memory if the operation is invoked.

Unfortunately, I have seen that some motherboard vendor shipped suspect custom modification which caused more problem than the original Aptio F/W.

I think it is a good reason to explain why it should be self-sustained although which can't prevent buggy F/W from crashing your OS entirely.

Nobody's perfect ...

Date: 2012-03-21 03:54 pm (UTC)
From: [identity profile] https://www.google.com/accounts/o8/id?id=AItOawkO1wz9yESE-0Ku5aphRYcy5k2EmIpa6-A
... but my questions is: do you report such things to Apple, so that are at least aware of the issues? In the case of Apple you seems to have somebody to talk to, since there are some means of reporting issues, as far as I'm aware by the stories that apple users tell me.

Re: Nobody's perfect ...

Date: 2012-03-21 04:21 pm (UTC)
From: [identity profile] https://www.google.com/accounts/o8/id?id=AItOawkO1wz9yESE-0Ku5aphRYcy5k2EmIpa6-A
Did they comment on the issue? Are they planning to fix it? Or refusing to?

Possible solution to UEFI

Date: 2012-04-10 03:32 pm (UTC)
From: (Anonymous)
I have used Linux as my main desktop since 2004 and I dread having to go back to windows.

I had a dream about the UEFI situation last night in which I found a solution to the problem. I am not an engineer or computer scientist, so the dream my have been influenced by the 50mg of benadryl I took before bed.


In my dream I asked someone if the major server and desktop distributions like Fedora, Centos, Debian, Ubuntu, OpenSuse created a properly signed version of their current kernel and a basic, standard userland of utilities -- Perhaps also signed X or Wayland code as well. Could the rest of the userland applications be installed and run without signing if they were written to a universal and signed implementation of the LSB? Sort of a Linux version of Windows Compatibility mode where applications that could run under all supporting distros even after the signed kernel and other components were upgraded to newer versions.

I don't remember how or if the person answered me, but I woke up wishing that such a solution could be found and I would not have to by Windows to run Ubuntu in the future.

ExitBootServices event

Date: 2014-09-11 11:44 am (UTC)
From: [personal profile] lersek
I found this post again, linked from http://www.reddit.com/r/linux/comments/2fddvg/im_matthew_garrett_kernel_developer_firmware/ck85hqc

So, Apple's SNP (Simple Network Protocol) driver is buggy then. The UEFI Driver Writer's Guide explains that network drivers need to register an EVT_SIGNAL_EXIT_BOOT_SERVICES event. When ExitBootServices() is called, the firmware signals this event, and the handler / callback function for the event is invoked. In the function, the driver needs to stop all in-flight DMA, exactly in order to prevent the situation that you describe.

Apple's driver apparently didn't do this at the time you wrote this post. edk2/OvmfPkg/VirtioNetDxe/ does (as an open source example), mapping the "DMA transfer" concept to "virtio transfer".

Profile

Matthew Garrett

About Matthew

Power management, mobile and firmware developer on Linux. Security developer at Nebula. Ex-biologist. @mjg59 on Twitter. Content here should not be interpreted as the opinion of my employer.

Expand Cut Tags

No cut tags