Microsoft aren't forcing Lenovo to block free operating systems

[mjg59]

Update: Patches to fix this have been posted

There's a story going round that Lenovo have signed an agreement with Microsoft that prevents installing free operating systems. This is sensationalist, untrue and distracts from a genuine problem.

The background is straightforward. Intel platforms allow the storage to be configured in two different ways - "standard" (normal AHCI on SATA systems, normal NVMe on NVMe systems) or "RAID". "RAID" mode is typically just changing the PCI IDs so that the normal drivers won't bind, ensuring that drivers that support the software RAID mode are used. Intel have not submitted any patches to Linux to support the "RAID" mode.

In this specific case, Lenovo's firmware defaults to "RAID" mode and doesn't allow you to change that. Since Linux has no support for the hardware when configured this way, you can't install Linux (distribution installers will boot, but won't find any storage device to install the OS to).

Why would Lenovo do this? I don't know for sure, but it's potentially related to something I've written about before - recent Intel hardware needs special setup for good power management. The storage driver that Microsoft ship doesn't do that setup. The Intel-provided driver does. "RAID" mode prevents the Microsoft driver from binding and forces the user to use the Intel driver, which means they get the correct power management configuration, battery life is better and the machine doesn't melt.

(Why not offer the option to disable it? A user who does would end up with a machine that doesn't boot, and if they managed to figure that out they'd have worse power management. That increases support costs. For a consumer device, why would you want to? The number of people buying these laptops to run anything other than Windows is miniscule)

Things are somewhat obfuscated due to a statement from a Lenovo rep:This system has a Signature Edition of Windows 10 Home installed. It is locked per our agreement with Microsoft. It's unclear what this is meant to mean. Microsoft could be insisting that Signature Edition systems ship in "RAID" mode in order to ensure that users get a good power management experience. Or it could be a misunderstanding regarding UEFI Secure Boot - Microsoft do require that Secure Boot be enabled on all Windows 10 systems, but (a) the user must be able to manage the key database and (b) there are several free operating systems that support UEFI Secure Boot and have appropriate signatures. Neither interpretation indicates that there's a deliberate attempt to prevent users from installing their choice of operating system.

The real problem here is that Intel do very little to ensure that free operating systems work well on their consumer hardware - we still have no information from Intel on how to configure systems to ensure good power management, we have no support for storage devices in "RAID" mode and we have no indication that this is going to get better in future. If Intel had provided that support, this issue would never have occurred. Rather than be angry at Lenovo, let's put pressure on Intel to provide support for their hardware.

Comments

Shared blame

[(Anonymous)]

Intel could do better from a technical point, but one buys a Lenovo product, it is their responsibility to have the right pieces of hardware into place.
So they should be blamed for choosing an hardware with windows-only drivers.
They could also have added an option in the BIOS/UEFI that, once the secure boot has been disabled, allows the user to also disable this RAID mode.

Re: Shared blame

[(Anonymous)]

> So they should be blamed for choosing an hardware with windows-only drivers.

And when exactly did you discover you were a pre-cog?

RAID level?

[(Anonymous)]

I fought a similar system on a server board once, and actually got a system to install treating it as a RAID 0 with a single disk. It did not boot properly though. Since these are notebook systems with a single disk anyway, is that what they are treating the system as? (i.e. RAID 0 with a single disk) Is there any way to update GRUB to boot off of such a thing?

All that said, I think Intel absolutely should be held responsible, they certainly push themselves as a Linux friendly company, and disk controllers should be a basic device I should always be able to access.

Re: RAID level?

[(Anonymous)]

GRUB can certainly boot off the thing, but the problem is that the kernel itself cannot detect the drive at all while it's in RAID mode (and it's currently STUCK in RAID mode, even though the Intel hardware fully supports run-of-the-mill vanilla AHCI, because of Lenovo's short-sighted and quite intentional BIOS modification that locks users out from changing their BIOS setting).

[(Anonymous)]

So this is just an educated guess?

But then you insist that it's not Lenovo trying to lock out other operating systems. How do you know that? Did they give you the scoop?

Even if the device would get worse battery life (which we don't know that it would, and nobody who booted up Linux on the ISK model or Live on the ISK2 reported their laptop "melting"), it should be up to the user.

You say that if your guess is right, it's a cheap hack to work around crappy power management in Windows. Do we know Linux has crappy power management like Windows does?

[mjg59]

> So this is just an educated guess?

The change only prevents Linux being installed because Linux doesn't support modern hardware. It's easier to fix that than it is to get a vendor to push a firmware update.

> But then you insist that it's not Lenovo trying to lock out other operating systems. How do you know that? Did they give you the scoop?

Because they're not locking out other operating systems? Linux boots fine.

> Even if the device would get worse battery life (which we don't know that it would, and nobody who booted up Linux on the ISK model or Live on the ISK2 reported their laptop "melting"), it should be up to the user.

Why should it be up to the user? Should the user be able to program every memory timing option, even if by doing so they introduce occasional crashes? Should they be able to set every thermal threshold, even if by doing so they're reducing their hardware life expectancy? All hardware vendors restrict the options available to users.

> You say that if your guess is right, it's a cheap hack to work around crappy power management in Windows. Do we know Linux has crappy power management like Windows does?

It's a hack to work around the fact that Intel won't tell anyone else how to make power management work properly on Intel platforms, including Microsoft. Linux certainly does the wrong thing here.

[(Anonymous)]

Also,

If it was to prevent a clueless user from ending up in that situation where Windows wouldn't boot if they toggled it to AHCI mode, then why did Lenovo write code to make sure that if you used an EFI variable to set it, that it would switch it back to RAID? Is a user that doesn't know what they're doing likely to be in the EFI shell?

[mjg59]

You're attaching intent where there's no evidence of intent. If they only support RAID, the easiest way to implement that is just to force set it on boot

[(Anonymous)]

"This system has a Signature Edition of Windows 10 Home installed. It is locked per our agreement with Microsoft."

You forgot another interpretation where Lenovo has signed an agreement with Microsoft that prevents installing free operating systems.

[mjg59]

A policy that would be entirely ineffective if we actually bothered to support the hardware.

Liar.

[(Anonymous)]

"In this specific case, Lenovo's firmware defaults to "RAID" mode and doesn't allow you to change that. Since Linux has no support for the hardware when configured this way, you can't install Linux (distribution installers will boot, but won't find any storage device to install the OS to)."

This is a lie, you're obviously a schill.

Non-RAID settings were intentionally removed from the BIOS, and the RAID format used is non-standard.

Re: Liar.

[(Anonymous)]

Why would any manufacturer even consider isolating users? Your notion implies Lenovo went out of their way to screw the userbase. Regardless of how you feel, how can you see a decision like that being made?

This does not have a malicious intent. It's probably an oversight by engineering that didn't realistically see this as a plausible scenario on a consumer system, especially an ultrabook. Heck, I even install linux in VMs now rather than overwrite the host OS / dual-boot.

Microsoft's Secure Boot requirements

[(Anonymous)]

Thanks a lot for doing this write-up, this clarifies things greatly. After arguing with a friend, I was wondering though where the requirement 'the user must be able to manage the key database' originates. Is this part of the Secure Boot specs or is this only required for Microsoft's Windows certification? Thanks!

Re: Microsoft's Secure Boot requirements

[(Anonymous)]

It's part of Microsoft's certification requirements, *for x86 systems*. The requirements for ARM were the exact opposite (they say that the user must *not* be able to change the key), though since barely anyone's doing Windows-on-ARM any more that's becoming increasingly less relevant. The UEFI specification (where Secure Boot is actually defined) doesn't prescribe anything about how it should be set up out of the box in any particular firmware implementation (whether any keys should be pre-loaded, whose they should be if so, whether there should be an interface for changing them, etc.)

Dell XPS 15 InfinityEdge

[xnox [launchpad.net]]

I have skylake Dell XPS 15 Infinity Edge. It came with NVMe hard drive configured in a RAID setting which was not recognised by the Ubuntu installer. In the BIOS settings I was able to change that to something normal instead (it was listed as AHCI or some such), after that NVMe based installation worked with Ubuntu, but Windows 10 failed to boot. To resolve that, I had to reboot Windows 10 in safety mode, switch from raid to AHCI, boot into safety mode again, which I guess regenerated the "kernel modules included in the Windows boot process" or some such. After that, both Ubuntu and Windows 10 boot happily ever after. This is unfortunately well documented https://en.wikipedia.org/wiki/Advanced_Host_Controller_Interface#Boot_issues I wish windows would always include AHCI driver and/or the OEMs/Vendors did. Are we sure that it's not just the standard Intel Matrix Raid configuration which is supported by MDADM? Ubuntu Desktop installer doesn't support Intel Matrix RAID configuration out of the box at the time.

Re: Dell XPS 15 InfinityEdge

[(Anonymous)]

Wouldn't that make the device slower? I haven't use the Ubuntu (or any other) installer for a while but maybe it runs an older kernel without the NVMe driver?

[(Anonymous)]

> The real problem here is that Intel do very little to ensure that free operating systems work well on their consumer hardware

This seems a little extreme. As hardware vendors go, Intel do more to get their hardware supported upstream than a lot of others. Sure, there are areas they could do better, but still.

Lenovo's decision to disable the standard BIOS/UEFI options that allow changing the disk controller mode is the real blocker to having the hardware work. Few people would want to use fakeraid on Linux given the choice anyway.

not sure

[(Anonymous)]

Not sure. Haswell power management in 4.6 is still very sucky.

[(Anonymous)]

> Why would Lenovo do this? I don't know for sure, [..]
> [..] due to a statement from a Lenovo rep [..] It's unclear what this is meant to mean. [..]

Thanks for the insight in the mechanics of the problem,
but as you state yourself your explanation of Lenovo's intent
is also just an assumption.
Therefore we can't know for sure the real reason for Lenovo's
actions.

> [..] That increases support costs. For a consumer device, why would you want to? [..]

Then Lenovo should switch to selling bricks. They have even less support costs.

How far to fake it?

[(Anonymous)]

Is it the case the kernel doesn't have the necessary IDs in place to "bind" the controller in RAID mode and if so couldn't someone who wanted a life of pain add the ids to their custom kernel such that it did bind to the AHCI driver? While you would never be able to read disks formatted with the true RAID format would it at least allow you to talk to the disks?

Which part is it that is not being understood?

[(Anonymous)]

RAID: Redundant Array of Inexpensive Drives (Disks)

So which part of RAID is it that is not being understood? RAID by definition REQUIRES 2 or more drives/disks. You can not have a RAID set up with out 2 or more. RAID is ment to be/is a way of securing/backing up data on a computer.

Is this single disk/drive computer set up to use RAID? No! So what other purpose could it serve? Other then to "lock" someone, or something out?

So tell me if it is not a "RAID" setup, which by definition it isn't, just what is it? And why if it is not ment to lock out" is it there at all?

Re: Which part is it that is not being understood?

[teaparty.net]

If I understand mjg59's argument correctly, the issue is not RAID. As you point out, that's not a laptop thing.

The issue is that Intel have not told anybody how to properly-manage power in their hardware; instead, they have released a binary-blob driver for Windows that just does it right. However, for the driver to do things right, Microsoft's driver mustn't bind to the hardware first. The easiest way for Lenovo to achieve this is to put the hardware into RAID mode, and not to let it come out. That means the MS driver examines the hardware, decides it can't deal with it, and ignores it, leaving the Intel driver to come along and claim it.

Presumably the Intel driver is perfectly happy to run in single-disc JBOD mode, but it does it with the right power management. The end result, according to Matthew, is "correct power management configuration, battery life is better and the machine doesn't melt".

I'm not saying it's a good thing, and I don't think Matthew is either. It's just the easiest way for Lenovo to deal with Intel's stupid secret-sauce power management, in a Windows context.

complete BS

[(Anonymous)]

Sorry but your article is complete BS.

Lenovo are the biggest PC laptop vendor in the world. Laptops don't have RAID.

Intel support for Linux is generally excellent.

Re: complete BS

[(Anonymous)]

Laptops don't have RAID.

They don't?

Mine do (Sony VPCZ1, 2 or 4 striped SSD's).

Storm in a teacup

[cowbutt]

"Intel have not submitted any patches to Linux to support the "RAID" mode."

Such patches are unnecessary, as mdadm already supports Intel Rapid Storage Technology (RST - http://www.intel.co.uk/content/www/uk/en/architecture-and-technology/rapid-storage-technology.html ) for simple RAID (e.g. levels 0, 1, 10) arrays, allowing them to be assembled as md or dmraid devices under Linux.

However, it would appear that the version of mdadm in shipping versions of Ubuntu (at least - maybe other distros too) doesn't support the Smart Response Technology (SRT - http://www.intel.com/content/www/us/en/architecture-and-technology/smart-response-technology.html ) feature that's a part of RST and is used by Lenovo to build a hybrid one-stripe RAID0 device from the HDD with a cache on the SSD (I'm sure Lenovo have a good reason for not using a SSHD). Dan Williams of Intel submitted a series of patches to mdadm to support SRT back in April 2014: https://marc.info/?l=linux-raid&r=1&b=201404&w=2 . Perhaps now there's shipping hardware that requires them, there'll be the impetus for distro vendors to get them integrated into mdadm, and their auto-detection in their installers to use the functionality provided sanely.

Re: Storm in a teacup

[(Anonymous)]

I should add that mdadm is not present in Ubuntu live images by default - one has to pull it in by issuing "sudo apt[-get] install mdadm". BTW, I don't know if mdadm would detect the RAID controller/disk immediately upon installation, or it would require a reboot. In the latter case you may wish to use a USB key with enough spare room to save the system status and reboot. I'd use UNetBootin to prepare such a USB key.

The main issue here is, a user who doesn't even see a disk, probably wouldn't know to go as far as installing mdadm. IMHO, given the broadening diffusion of NVMe and RAID devices, Debian, Canonical, REDHAT, Fedora etc. might wish to make mdadm part of their live images by default (and eventually strip it from the installed system if it's unnecessary).

just imagine...

[(Anonymous)]

Imagine the HELL ON EARTH that will be if every fuck*ng company decides to do it's own "RAID Mode" controller. Why we have a standard like AHCI then?

The real problem here is that AHCI firmwares are crap, and RAID will not solve problems with overheating, neither make better power consumption counters. RAID is doing the right thing in this specific case, because a lot of companies create AHCI firmware with no optimizations at all, so forcing a "default sane" for power consumption makes the sensation that RAID mode is a better standard.

Until proven wrong, this is still a move to let Linux out of the market, and if Lenovo is not the one to blame, is the one guilty of colluding with Intel to delay Linux support on this equipment.

What does "RAID" actually do?

[(Anonymous)]

Does anyone have concrete information about what "RAID" does. For example, lspci -vvvnn output from an affected system might be nice.

--luto

Re: What does "RAID" actually do?

[mjg59]

It should be exactly the same as you get when you enable "RAID" on the Skylake XPS 13s.

[(Anonymous)]

intel, lenovo, nvidia - the wold can safely ignore all of such dumb companies. I do not see any reason wasting time to the problems with their hardware.

[(Anonymous)]

Someone on Reddit suggested that since Linux doesn't use BIOS calls for anything and addresses the hardware in native mode, and the hardware still supports AHCI native mode, that someone should "teach GRUB read-only RAID support" and then have the kernel set to switch the hardware out of RAID and into AHCI native mode as soon as it is loaded.

Is this an option? Sorry if this is obviously not an answer.

As for the power management thing, Dell lets you turn off the same "RAID" mode and boot Linux. What kind of runtime loss are we talking here? Can't be that bad or someone would have said something. SSDs use like 2 watts? Even if it never goes into power saving mode, it's about like turning your backlight up 5%, right?

[mjg59]

If you wanted to try that, the best place to do it would be the UEFI setup code in the kernel under arch/x86/boot/compressed/eboot.c - that way you can do it independent of bootloader. Grub's able to read files anyway since it's using the firmware calls to do that.

The power management thing isn't about the SSD itself, it's about ensuring that the controller is programmed correctly so that the entire CPU package can enter deep sleep states. Modern systems should idle at around 4W at most, so small amounts make abig difference here.

installed Linux on about 15 laptops, only one failed: a Lenovo

[(Anonymous)]

I installed Linux on about 15 laptops, only one failed: a Lenovo. While this looks like an attention seeking headline, it's true.

The first laptop I installed Linux on was about 15 odd years ago - while I had some trouble I got it going eventually. Most follow up laptops where from a variety of suppliers (dell, toshiba, acer etc) and I never had a problem to get any of the hardware working.

The last two laptops are two rather new laptops, both with UEFI/LEGACY, both with AHCI, both with INTEL hardware and all INTEL hardware known to work.

One is a Lenovo Yoga, the other one is a Dell XPS 12.

It took me about 1 hour to get it to go on the Dell, I did that in a number of steps. I first made some space at the end of the drive creating a 30GB space, then tried Fedora 23 without UEFI in Legacy mode - it worked first go.
I then wiped that partition again, changed back to full UEFI, installed Fedora 23 with full UEFI enabled. Grub has taken over and I can easily switch between Linux and Windows 10, all hardware working albeit the touch pad being touchy - but that's the case too in Win10.

Not so the Lenovo - I tried everything for a week. I did RTFM, I read so many articles on the web, I followed every trick of the trade I have learned with myriads of kernel switches, debugging - you name it. I tried.

I have given up, my first failure to ever get a computer to work with Linux.

Luckily there is VmWare and Kali Linux.

Sorry, Lenovo sucks.

Re: installed Linux on about 15 laptops, only one failed: a Lenovo

[(Anonymous)]

Lenovo X1 Carbon (Windows 8, 2014) UEFI Secure boot only, alternate boot device prohibited,
i.e. the UEFI interface is not even accessible. Lenovo has been configuring this way for some time. It does not seem to be a faithful production of the EFI standard, IMHO.

Lenovo Thnikpad x250 boots fine with at least 8 distro and power is fine

[unixbhaskar.wordpress.com]

From last August I don't feel any bigger trouble with it.All the installation went well. Power management ,well, we knew it. And we can live with it.In fact, did so many tweak and so many thing to get better battery time...still..

Anyway I have very little idea how to resolve that...

Lenovo.....

[(Anonymous)]

As far as I can tell? this isn't even an issue. Go out and buy LAST YEAR'S model of the damn machine, which should be able to install whatever it is you want on it. Then when sales of this "most recent" PC go way down?...then maybe they'll pull it from store shelves and online retailers, realizing that now more than ever? the one thing people want most? isn't flash, isn't bling, isn't cute names or matching headphones, the one thing that today's millennial consumer wants? Is choice. and along with choice comes freedom, why do you think it is that Microsoft has hitched itself to Linux with their "Microsoft Loves Linux" campaign? it's to make sure they stay in the public eye and relevant, because more and more people are CHOOSING to buy a Windows PC and install LINUX on it! Remember the old saying: "Keep Your Friends Close, And Your ENEMIES Closer".

Re: Lenovo.....

[(Anonymous)]

There's some problems with that.

Last year's machine had a significantly slower graphics chip. The HD 520 isn't really good for much more than displaying a spreadsheet. Especially not at the high resolutions the screen is capable of. The Iris 540 graphics chip...I've seen benchmarks that suggest it's about 60% faster.

So you could run the older model on Linux, if you never wanted to run anything more complicated than a web browser.

Any news regarding the patch?

[(Anonymous)]

Hi Matt,

I just became an unhappy new owner of the laptop in question (through no fault of my own, it was batch-bought by my company...), and I'd really like to install Fedora on it.

I know next to nothing about programming, though. If you ever make that patch work, would you also provide step-by-step instructions how to perform an actual install with it? Please please please?

Anyway, thanks for the effort!

Re: Any news regarding the patch?

[(Anonymous)]

Realistically, it would need to be upstreamed, and in the meantime, distributions need to be lobbied to use it.

[(Anonymous)]

A user on Lenovo's forums is running Linux on a SD card on the ISK2 Yoga 900 and said this about the battery life when I quoted your April article about Haswell and Broadwell power management policies:

Matthew's stuff about the PCH power states is interesting. Taking his numbers (which might not be for Skylake) the change in 'idle time' for a 66Wh battery (like in the Yoga 900) between 5W and 8.5W is 13.2 hours and 7.7 hours. Going to go out on a limb here and say that Skylake is probably more efficient.
Any activity on the machine will of course make it use more power.

I've had mine running all week with Ubuntu and off the charger for several days... so sleep works obviously. I'm not noticing any thermal issues which would worry me more. I'm about to start compiling kernels on it so we'll see how that goes... if anything is going to push the thermal management it's that.

I guess what annoys me about all of this is that Lenovo could have just stated this up front; made the patched BIOS available and with the disclaimer it might negatively affect battery performance and cause more thermal throttling. I would be perfectly happy with that... I don't do many eight hour solid sessions on a laptop away from power. YMMV.

What's more annoying is that if you go and read the Intel forums they say go and talk to the Linux kernel developers... and then the kernel developers say Intel doesn't provide the details. Intel have been doing some great work with Linux lately but obviously this isn't one of those areas...

"support" is toxic

[(Anonymous)]

When you say, "If Intel had provided that support, this issue would never have occurred.", know this:

Free Operating systems need information. By using the word "support", confusion is introduced. We need to know how to program the hardware we buy. No organization wants to sign up for support. The word is toxic. The business folks will never understand so long as this word is used.

Please don't use it. Please ask, "How do I program the hardware you want me to buy?". "I'm not telling" means I keep shopping. It has nothing to do with support organizations or special intellectual property. If I can't program it, I don't own it, so why would I pay for it?

why isn't it hackable?

[toio]

Out of curiosity
So, I'm not a hardware specialist but...
If windows uses intel binary blob driver, why can't linux too?
It's already doesn't recognize the ssd, just like with windows,
so the next step should be binding to the intel driver somehow?

Re: why isn't it hackable?

[(Anonymous)]

> If windows uses intel binary blob driver, why can't linux too?

Because if it works wrong, you screwed? Because you can't hack into it? Because blob updates to kernel releases would lag behind? And, for that matter, because you can't rely at all on a company to update a binary blob for every kernel release?

> It's already doesn't recognize the ssd, just like with windows

Patches been posted, as the update mentioned. Though they seem stuck a bit — peoples in the mailing list saying like it needs a review by commity, because such behavior might be undefined in some cases.