[personal profile] mjg59
Update: Patches to fix this have been posted

There's a story going round that Lenovo have signed an agreement with Microsoft that prevents installing free operating systems. This is sensationalist, untrue and distracts from a genuine problem.

The background is straightforward. Intel platforms allow the storage to be configured in two different ways - "standard" (normal AHCI on SATA systems, normal NVMe on NVMe systems) or "RAID". "RAID" mode is typically just changing the PCI IDs so that the normal drivers won't bind, ensuring that drivers that support the software RAID mode are used. Intel have not submitted any patches to Linux to support the "RAID" mode.

In this specific case, Lenovo's firmware defaults to "RAID" mode and doesn't allow you to change that. Since Linux has no support for the hardware when configured this way, you can't install Linux (distribution installers will boot, but won't find any storage device to install the OS to).

Why would Lenovo do this? I don't know for sure, but it's potentially related to something I've written about before - recent Intel hardware needs special setup for good power management. The storage driver that Microsoft ship doesn't do that setup. The Intel-provided driver does. "RAID" mode prevents the Microsoft driver from binding and forces the user to use the Intel driver, which means they get the correct power management configuration, battery life is better and the machine doesn't melt.

(Why not offer the option to disable it? A user who does would end up with a machine that doesn't boot, and if they managed to figure that out they'd have worse power management. That increases support costs. For a consumer device, why would you want to? The number of people buying these laptops to run anything other than Windows is miniscule)

Things are somewhat obfuscated due to a statement from a Lenovo rep:This system has a Signature Edition of Windows 10 Home installed. It is locked per our agreement with Microsoft. It's unclear what this is meant to mean. Microsoft could be insisting that Signature Edition systems ship in "RAID" mode in order to ensure that users get a good power management experience. Or it could be a misunderstanding regarding UEFI Secure Boot - Microsoft do require that Secure Boot be enabled on all Windows 10 systems, but (a) the user must be able to manage the key database and (b) there are several free operating systems that support UEFI Secure Boot and have appropriate signatures. Neither interpretation indicates that there's a deliberate attempt to prevent users from installing their choice of operating system.

The real problem here is that Intel do very little to ensure that free operating systems work well on their consumer hardware - we still have no information from Intel on how to configure systems to ensure good power management, we have no support for storage devices in "RAID" mode and we have no indication that this is going to get better in future. If Intel had provided that support, this issue would never have occurred. Rather than be angry at Lenovo, let's put pressure on Intel to provide support for their hardware.
Page 1 of 2 << [1] [2] >>

Shared blame

Date: 2016-09-21 06:09 pm (UTC)
From: (Anonymous)
Intel could do better from a technical point, but one buys a Lenovo product, it is their responsibility to have the right pieces of hardware into place.
So they should be blamed for choosing an hardware with windows-only drivers.
They could also have added an option in the BIOS/UEFI that, once the secure boot has been disabled, allows the user to also disable this RAID mode.

RAID level?

Date: 2016-09-21 06:31 pm (UTC)
From: (Anonymous)
I fought a similar system on a server board once, and actually got a system to install treating it as a RAID 0 with a single disk. It did not boot properly though. Since these are notebook systems with a single disk anyway, is that what they are treating the system as? (i.e. RAID 0 with a single disk) Is there any way to update GRUB to boot off of such a thing?

All that said, I think Intel absolutely should be held responsible, they certainly push themselves as a Linux friendly company, and disk controllers should be a basic device I should always be able to access.

Date: 2016-09-21 06:47 pm (UTC)
From: (Anonymous)
So this is just an educated guess?

But then you insist that it's not Lenovo trying to lock out other operating systems. How do you know that? Did they give you the scoop?

Even if the device would get worse battery life (which we don't know that it would, and nobody who booted up Linux on the ISK model or Live on the ISK2 reported their laptop "melting"), it should be up to the user.

You say that if your guess is right, it's a cheap hack to work around crappy power management in Windows. Do we know Linux has crappy power management like Windows does?

Date: 2016-09-21 06:55 pm (UTC)
From: (Anonymous)

If it was to prevent a clueless user from ending up in that situation where Windows wouldn't boot if they toggled it to AHCI mode, then why did Lenovo write code to make sure that if you used an EFI variable to set it, that it would switch it back to RAID? Is a user that doesn't know what they're doing likely to be in the EFI shell?

Date: 2016-09-21 07:37 pm (UTC)
From: (Anonymous)
"This system has a Signature Edition of Windows 10 Home installed. It is locked per our agreement with Microsoft."

You forgot another interpretation where Lenovo has signed an agreement with Microsoft that prevents installing free operating systems.


Date: 2016-09-21 07:41 pm (UTC)
From: (Anonymous)
"In this specific case, Lenovo's firmware defaults to "RAID" mode and doesn't allow you to change that. Since Linux has no support for the hardware when configured this way, you can't install Linux (distribution installers will boot, but won't find any storage device to install the OS to)."

This is a lie, you're obviously a schill.

Non-RAID settings were intentionally removed from the BIOS, and the RAID format used is non-standard.

Microsoft's Secure Boot requirements

Date: 2016-09-21 10:27 pm (UTC)
From: (Anonymous)
Thanks a lot for doing this write-up, this clarifies things greatly. After arguing with a friend, I was wondering though where the requirement 'the user must be able to manage the key database' originates. Is this part of the Secure Boot specs or is this only required for Microsoft's Windows certification? Thanks!

Dell XPS 15 InfinityEdge

Date: 2016-09-21 10:29 pm (UTC)
From: [identity profile] xnox [launchpad.net]
I have skylake Dell XPS 15 Infinity Edge. It came with NVMe hard drive configured in a RAID setting which was not recognised by the Ubuntu installer. In the BIOS settings I was able to change that to something normal instead (it was listed as AHCI or some such), after that NVMe based installation worked with Ubuntu, but Windows 10 failed to boot. To resolve that, I had to reboot Windows 10 in safety mode, switch from raid to AHCI, boot into safety mode again, which I guess regenerated the "kernel modules included in the Windows boot process" or some such. After that, both Ubuntu and Windows 10 boot happily ever after. This is unfortunately well documented https://en.wikipedia.org/wiki/Advanced_Host_Controller_Interface#Boot_issues I wish windows would always include AHCI driver and/or the OEMs/Vendors did. Are we sure that it's not just the standard Intel Matrix Raid configuration which is supported by MDADM? Ubuntu Desktop installer doesn't support Intel Matrix RAID configuration out of the box at the time.

Date: 2016-09-21 10:53 pm (UTC)
From: (Anonymous)
> The real problem here is that Intel do very little to ensure that free operating systems work well on their consumer hardware

This seems a little extreme. As hardware vendors go, Intel do more to get their hardware supported upstream than a lot of others. Sure, there are areas they could do better, but still.

Lenovo's decision to disable the standard BIOS/UEFI options that allow changing the disk controller mode is the real blocker to having the hardware work. Few people would want to use fakeraid on Linux given the choice anyway.

Date: 2016-09-22 02:47 am (UTC)
From: (Anonymous)
> Why would Lenovo do this? I don't know for sure, [..]
> [..] due to a statement from a Lenovo rep [..] It's unclear what this is meant to mean. [..]

Thanks for the insight in the mechanics of the problem,
but as you state yourself your explanation of Lenovo's intent
is also just an assumption.
Therefore we can't know for sure the real reason for Lenovo's

> [..] That increases support costs. For a consumer device, why would you want to? [..]

Then Lenovo should switch to selling bricks. They have even less support costs.

How far to fake it?

Date: 2016-09-22 04:43 am (UTC)
From: (Anonymous)
Is it the case the kernel doesn't have the necessary IDs in place to "bind" the controller in RAID mode and if so couldn't someone who wanted a life of pain add the ids to their custom kernel such that it did bind to the AHCI driver? While you would never be able to read disks formatted with the true RAID format would it at least allow you to talk to the disks?

Which part is it that is not being understood?

Date: 2016-09-22 06:23 am (UTC)
From: (Anonymous)
RAID: Redundant Array of Inexpensive Drives (Disks)

So which part of RAID is it that is not being understood? RAID by definition REQUIRES 2 or more drives/disks. You can not have a RAID set up with out 2 or more. RAID is ment to be/is a way of securing/backing up data on a computer.

Is this single disk/drive computer set up to use RAID? No! So what other purpose could it serve? Other then to "lock" someone, or something out?

So tell me if it is not a "RAID" setup, which by definition it isn't, just what is it? And why if it is not ment to lock out" is it there at all?

complete BS

Date: 2016-09-22 06:51 am (UTC)
From: (Anonymous)
Sorry but your article is complete BS.

Lenovo are the biggest PC laptop vendor in the world. Laptops don't have RAID.

Intel support for Linux is generally excellent.

Storm in a teacup

Date: 2016-09-22 09:17 am (UTC)
From: [personal profile] cowbutt
"Intel have not submitted any patches to Linux to support the "RAID" mode."

Such patches are unnecessary, as mdadm already supports Intel Rapid Storage Technology (RST - http://www.intel.co.uk/content/www/uk/en/architecture-and-technology/rapid-storage-technology.html ) for simple RAID (e.g. levels 0, 1, 10) arrays, allowing them to be assembled as md or dmraid devices under Linux.

However, it would appear that the version of mdadm in shipping versions of Ubuntu (at least - maybe other distros too) doesn't support the Smart Response Technology (SRT - http://www.intel.com/content/www/us/en/architecture-and-technology/smart-response-technology.html ) feature that's a part of RST and is used by Lenovo to build a hybrid one-stripe RAID0 device from the HDD with a cache on the SSD (I'm sure Lenovo have a good reason for not using a SSHD). Dan Williams of Intel submitted a series of patches to mdadm to support SRT back in April 2014: https://marc.info/?l=linux-raid&r=1&b=201404&w=2 . Perhaps now there's shipping hardware that requires them, there'll be the impetus for distro vendors to get them integrated into mdadm, and their auto-detection in their installers to use the functionality provided sanely.
Edited (some extra words to explain SRT's relationship to RST) Date: 2016-09-22 09:19 am (UTC)

just imagine...

Date: 2016-09-22 12:07 pm (UTC)
From: (Anonymous)
Imagine the HELL ON EARTH that will be if every fuck*ng company decides to do it's own "RAID Mode" controller. Why we have a standard like AHCI then?

The real problem here is that AHCI firmwares are crap, and RAID will not solve problems with overheating, neither make better power consumption counters. RAID is doing the right thing in this specific case, because a lot of companies create AHCI firmware with no optimizations at all, so forcing a "default sane" for power consumption makes the sensation that RAID mode is a better standard.

Until proven wrong, this is still a move to let Linux out of the market, and if Lenovo is not the one to blame, is the one guilty of colluding with Intel to delay Linux support on this equipment.

What does "RAID" actually do?

Date: 2016-09-23 01:50 am (UTC)
From: (Anonymous)
Does anyone have concrete information about what "RAID" does. For example, lspci -vvvnn output from an affected system might be nice.


Date: 2016-09-23 03:00 am (UTC)
From: (Anonymous)
intel, lenovo, nvidia - the wold can safely ignore all of such dumb companies. I do not see any reason wasting time to the problems with their hardware.

Date: 2016-09-23 05:23 am (UTC)
From: (Anonymous)
Someone on Reddit suggested that since Linux doesn't use BIOS calls for anything and addresses the hardware in native mode, and the hardware still supports AHCI native mode, that someone should "teach GRUB read-only RAID support" and then have the kernel set to switch the hardware out of RAID and into AHCI native mode as soon as it is loaded.

Is this an option? Sorry if this is obviously not an answer.

As for the power management thing, Dell lets you turn off the same "RAID" mode and boot Linux. What kind of runtime loss are we talking here? Can't be that bad or someone would have said something. SSDs use like 2 watts? Even if it never goes into power saving mode, it's about like turning your backlight up 5%, right?
From: (Anonymous)
I installed Linux on about 15 laptops, only one failed: a Lenovo. While this looks like an attention seeking headline, it's true.

The first laptop I installed Linux on was about 15 odd years ago - while I had some trouble I got it going eventually. Most follow up laptops where from a variety of suppliers (dell, toshiba, acer etc) and I never had a problem to get any of the hardware working.

The last two laptops are two rather new laptops, both with UEFI/LEGACY, both with AHCI, both with INTEL hardware and all INTEL hardware known to work.

One is a Lenovo Yoga, the other one is a Dell XPS 12.

It took me about 1 hour to get it to go on the Dell, I did that in a number of steps. I first made some space at the end of the drive creating a 30GB space, then tried Fedora 23 without UEFI in Legacy mode - it worked first go.
I then wiped that partition again, changed back to full UEFI, installed Fedora 23 with full UEFI enabled. Grub has taken over and I can easily switch between Linux and Windows 10, all hardware working albeit the touch pad being touchy - but that's the case too in Win10.

Not so the Lenovo - I tried everything for a week. I did RTFM, I read so many articles on the web, I followed every trick of the trade I have learned with myriads of kernel switches, debugging - you name it. I tried.

I have given up, my first failure to ever get a computer to work with Linux.

Luckily there is VmWare and Kali Linux.

Sorry, Lenovo sucks.

From: [identity profile] unixbhaskar.wordpress.com
From last August I don't feel any bigger trouble with it.All the installation went well. Power management ,well, we knew it. And we can live with it.In fact, did so many tweak and so many thing to get better battery time...still..

Anyway I have very little idea how to resolve that...


Date: 2016-09-23 06:49 pm (UTC)
From: (Anonymous)
As far as I can tell? this isn't even an issue. Go out and buy LAST YEAR'S model of the damn machine, which should be able to install whatever it is you want on it. Then when sales of this "most recent" PC go way down?...then maybe they'll pull it from store shelves and online retailers, realizing that now more than ever? the one thing people want most? isn't flash, isn't bling, isn't cute names or matching headphones, the one thing that today's millennial consumer wants? Is choice. and along with choice comes freedom, why do you think it is that Microsoft has hitched itself to Linux with their "Microsoft Loves Linux" campaign? it's to make sure they stay in the public eye and relevant, because more and more people are CHOOSING to buy a Windows PC and install LINUX on it! Remember the old saying: "Keep Your Friends Close, And Your ENEMIES Closer".

Any news regarding the patch?

Date: 2016-09-23 09:47 pm (UTC)
From: (Anonymous)
Hi Matt,

I just became an unhappy new owner of the laptop in question (through no fault of my own, it was batch-bought by my company...), and I'd really like to install Fedora on it.

I know next to nothing about programming, though. If you ever make that patch work, would you also provide step-by-step instructions how to perform an actual install with it? Please please please?

Anyway, thanks for the effort!

Date: 2016-09-24 09:58 pm (UTC)
From: (Anonymous)
A user on Lenovo's forums is running Linux on a SD card on the ISK2 Yoga 900 and said this about the battery life when I quoted your April article about Haswell and Broadwell power management policies:

Matthew's stuff about the PCH power states is interesting. Taking his numbers (which might not be for Skylake) the change in 'idle time' for a 66Wh battery (like in the Yoga 900) between 5W and 8.5W is 13.2 hours and 7.7 hours. Going to go out on a limb here and say that Skylake is probably more efficient.
Any activity on the machine will of course make it use more power.

I've had mine running all week with Ubuntu and off the charger for several days... so sleep works obviously. I'm not noticing any thermal issues which would worry me more. I'm about to start compiling kernels on it so we'll see how that goes... if anything is going to push the thermal management it's that.

I guess what annoys me about all of this is that Lenovo could have just stated this up front; made the patched BIOS available and with the disclaimer it might negatively affect battery performance and cause more thermal throttling. I would be perfectly happy with that... I don't do many eight hour solid sessions on a laptop away from power. YMMV.

What's more annoying is that if you go and read the Intel forums they say go and talk to the Linux kernel developers... and then the kernel developers say Intel doesn't provide the details. Intel have been doing some great work with Linux lately but obviously this isn't one of those areas...

"support" is toxic

Date: 2016-09-26 08:24 pm (UTC)
From: (Anonymous)
When you say, "If Intel had provided that support, this issue would never have occurred.", know this:

Free Operating systems need information. By using the word "support", confusion is introduced. We need to know how to program the hardware we buy. No organization wants to sign up for support. The word is toxic. The business folks will never understand so long as this word is used.

Please don't use it. Please ask, "How do I program the hardware you want me to buy?". "I'm not telling" means I keep shopping. It has nothing to do with support organizations or special intellectual property. If I can't program it, I don't own it, so why would I pay for it?

why isn't it hackable?

Date: 2016-10-15 02:55 pm (UTC)
From: [personal profile] toio
Out of curiosity
So, I'm not a hardware specialist but...
If windows uses intel binary blob driver, why can't linux too?
It's already doesn't recognize the ssd, just like with windows,
so the next step should be binding to the intel driver somehow?

Page 1 of 2 << [1] [2] >>


Matthew Garrett

About Matthew

Power management, mobile and firmware developer on Linux. Security developer at Google. Ex-biologist. @mjg59 on Twitter. Content here should not be interpreted as the opinion of my employer.

Expand Cut Tags

No cut tags