Samsung laptop bug is not Linux specific
Feb. 8th, 2013 10:41 pm![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
mjg59I bricked a Samsung laptop today. Unlike most of the reported cases of Samsung laptops refusing to boot, I never booted Linux on it - all experimentation was performed under Windows. It seems that the bug we've been seeing is simultaneously simpler in some ways and more complicated in others than we'd previously realised.
So, some background. The original belief was that the samsung-laptop driver was doing something that caused the system to stop working. This driver was coded to a Samsung specification in order to support certain laptop features that weren't accessible via any standardised mechanism. It works by searching a specific area of memory for a Samsung-specific signature. If it finds it, it follows a pointer to a table that contains various magic values that need to be written in order to trigger some system management code that actually performs the requested change. This is unusual in this day and age, but not unique. The problem is that the magic signature is still present on UEFI systems, but attempting to use the data contained in the table causes problems.
We're not quite sure what those problems are yet. Originally we assumed that the magic values we wrote were causing the problem, so the samsung-laptop driver was patched to disable it on UEFI systems. Unfortunately, this doesn't actually fix the problem - it just avoids the easiest way of triggering it. It turns out that it wasn't the writes that caused the problem, it was what happened next. Performing the writes triggered a hardware error of some description. The Linux kernel caught and logged this. In the old days, people would often never see these logs - the system would then be frozen and it would be impossible to access the hard drive, so they never got written to disk. There's code in the kernel to make this easier on UEFI systems. Whenever a severe error is encountered, the kernel copies recent messages to the UEFI variable storage space. They're then available to userspace after a reboot, allowing more accurate diagnostics of what caused the crash.
That crash dump takes about 10K of UEFI storage space. Microsoft require that Windows 8 systems have at least 64K of storage space available. We only keep one crash dump - if the system crashes again it'll simply overwrite the existing one rather than creating another. This is all completely compatible with the UEFI specification, and Apple actually do something very similar on their hardware. Unfortunately, it turns out that some Samsung laptops will fail to boot if too much of the variable storage space is used. We don't know what "too much" is yet, but writing a bunch of variables from Windows is enough to trigger it. I put some sample code here - it writes out 36 variables each containing a kilobyte of random data. I ran this as an administrator under Windows and then rebooted the system. It never came back.
This is pretty obviously a firmware bug. Writing UEFI variables is expressly permitted by the specification, and there should never be a situation in which an OS can fill the variable store in such a way that the firmware refuses to boot the system. We've seen similar bugs in Intel's reference code in the past, but they were all fixed early last year. For now the safest thing to do is not to use UEFI on any Samsung laptops. Unfortunately, if you're using Windows, that'll require you to reinstall it from scratch.
So, some background. The original belief was that the samsung-laptop driver was doing something that caused the system to stop working. This driver was coded to a Samsung specification in order to support certain laptop features that weren't accessible via any standardised mechanism. It works by searching a specific area of memory for a Samsung-specific signature. If it finds it, it follows a pointer to a table that contains various magic values that need to be written in order to trigger some system management code that actually performs the requested change. This is unusual in this day and age, but not unique. The problem is that the magic signature is still present on UEFI systems, but attempting to use the data contained in the table causes problems.
We're not quite sure what those problems are yet. Originally we assumed that the magic values we wrote were causing the problem, so the samsung-laptop driver was patched to disable it on UEFI systems. Unfortunately, this doesn't actually fix the problem - it just avoids the easiest way of triggering it. It turns out that it wasn't the writes that caused the problem, it was what happened next. Performing the writes triggered a hardware error of some description. The Linux kernel caught and logged this. In the old days, people would often never see these logs - the system would then be frozen and it would be impossible to access the hard drive, so they never got written to disk. There's code in the kernel to make this easier on UEFI systems. Whenever a severe error is encountered, the kernel copies recent messages to the UEFI variable storage space. They're then available to userspace after a reboot, allowing more accurate diagnostics of what caused the crash.
That crash dump takes about 10K of UEFI storage space. Microsoft require that Windows 8 systems have at least 64K of storage space available. We only keep one crash dump - if the system crashes again it'll simply overwrite the existing one rather than creating another. This is all completely compatible with the UEFI specification, and Apple actually do something very similar on their hardware. Unfortunately, it turns out that some Samsung laptops will fail to boot if too much of the variable storage space is used. We don't know what "too much" is yet, but writing a bunch of variables from Windows is enough to trigger it. I put some sample code here - it writes out 36 variables each containing a kilobyte of random data. I ran this as an administrator under Windows and then rebooted the system. It never came back.
This is pretty obviously a firmware bug. Writing UEFI variables is expressly permitted by the specification, and there should never be a situation in which an OS can fill the variable store in such a way that the firmware refuses to boot the system. We've seen similar bugs in Intel's reference code in the past, but they were all fixed early last year. For now the safest thing to do is not to use UEFI on any Samsung laptops. Unfortunately, if you're using Windows, that'll require you to reinstall it from scratch.
no subject
Date: 2013-02-09 06:00 am (UTC)no subject
Date: 2013-02-09 08:56 am (UTC)UEFI data in NAND Flash on motherboard
Date: 2013-02-09 09:32 am (UTC)CMOS Battery?
Date: 2013-02-09 11:11 am (UTC)Great writeup as always. Can you comment on the rumors going around that removing the CMOS NVRAM battery will make the board bootable again? Obviously that would mean taking apart the laptop to get at the motherboard, which voids the warranty. But for testing the fix, a developer might be willing to give up their warranty to iterate faster.
https://bugs.launchpad.net/ubuntu-cdimage/+bug/1040557/comments/23 on https://bugs.launchpad.net/ubuntu-cdimage/+bug/1040557 is one example of this.
UEFI to BIOS
Date: 2013-02-09 11:56 am (UTC)Seriously, why?
Date: 2013-02-09 01:02 pm (UTC)Seriously. If it's price you are worried about, I'll gladly sell you a brick of wood for a VERY good price if you are willing to believe that it's an equivalent piece of hardware and the only difference is price. You get what you pay for, people.
OK was that predictable enough? Well I'm sorry about that. But come on, when are people going to learn?
Re: Seriously, why?
Date: 2013-02-09 01:44 pm (UTC)Re: Seriously, why?
Date: 2013-02-09 01:50 pm (UTC)no subject
Date: 2013-02-09 02:29 pm (UTC)Re: UEFI to BIOS
Date: 2013-02-09 02:31 pm (UTC)no subject
Date: 2013-02-09 03:16 pm (UTC)Regarding Windows, I recently discovered that you can actually migrate a Windows 8 install from MBR+BIOS to GPT+UEFI. It's not straightforward, but it's possible (using bootrec and bcdboot). I haven't tried the other way around or with Windows 7, but I think that should be doable as well.
Re: Seriously, why?
Date: 2013-02-09 03:24 pm (UTC)no subject
Date: 2013-02-09 03:28 pm (UTC)Re: UEFI to BIOS
Date: 2013-02-09 04:30 pm (UTC)Re: UEFI to BIOS
Date: 2013-02-09 04:31 pm (UTC)Do you know *what* is broken?
Date: 2013-02-09 05:38 pm (UTC)QueryVariableInfo()?
Date: 2013-02-09 09:51 pm (UTC)Re: QueryVariableInfo()?
Date: 2013-02-09 09:53 pm (UTC)Re: Seriously, why?
Date: 2013-02-09 09:57 pm (UTC)Samsung's case, the problem is also the apple of problems.
Re: Seriously, why?
Date: 2013-02-09 10:26 pm (UTC)Re: Seriously, why?
Date: 2013-02-09 10:29 pm (UTC)You apple fanbois are really ridiculous when you try to sell your ignorance for expertise.
Re: Seriously, why?
Date: 2013-02-09 11:59 pm (UTC)Re: Seriously, why?
Date: 2013-02-10 12:12 am (UTC)I haven't come across that many Macbooks with Windows bootcamp, but I've experienced trackpad-related BSODs with a significant number of them (don't remember how many, but I did have to delete applemtp.sys 4 times so far to get the machine to boot at all).
geez what a surprise........
Date: 2013-02-10 04:12 am (UTC)Re: QueryVariableInfo()?
Date: 2013-02-10 07:36 am (UTC)