A new set of vulnerabilities were disclosed recently. As part of mitigating “meltdown”, the kernel introduced a new feature called Kernel Page Table Isolation (KPTI). This was a big change to come in late in the typical kernel development cycle but it provides important protection with some performance penalty. Updated kernels for supported versions of Fedora contain the KPTI patches. This article a high level overview of how KPTI works.
Modern processors for desktop computers offer different security levels to run code. The kernel runs at the most privileged level (“kernel space”) since it needs to access all parts of the hardware. Almost all other applications run at a lower privilege level (“user space”) and make system calls into the kernel to access privileged data.
Modern processors for desktop computers also support virtual memory. The kernel sets up page tables to control the mapping between a virtual address and physical address. Access between kernel and userspace is also controlled by page tables. A page that is mapped for the kernel is not accessible by userspace although the kernel can typically access user space.
Translating these mappings can be expensive so the hardware has a translation lookaside buffer (TLB) to store mappings. Sometimes it’s necessary to remove the old mappings (“flush the TLB”) but doing so is costly and code is written to minimize such calls. One trick here is to always have the kernel page tables mapped when user processes are running. The page table permissions prevent userspace from accessing the kernel mappings but the kernel can access the mappings immediately when a system call is made.
Meltdown exploit and how KPTI mitigates it
The meltdown exploit demonstrated that having the kernel mapping available in userspace can be risky. Modern processors prefetch data from all mappings to run as fast as possible. What data gets prefetched depends on the CPU implementation. When a running userspace program accesses a kernel mapping, it will take a fault and typically crash the program. The CPU however, may prefetch kernel data without causing any change to the running program. Prefetching is not usually a security risk because there are still permission checks on the addresses so userpace programs cannot access kernel data. What the meltdown researchers discovered was it was possible to measure how long data accesses took on prefetched data to gain information about the system. This is what’s referred to as a side-channel attack. The KPTI patches reworked how page tables are set up so that the kernel is no longer mapped in userspace. This means that userspace cannot prefetch any kernel data and thus the exploit is mitigated.
Actually writing an attack to collect useful data from this exploit can take weeks or months to develop for a single system. Still, in the interests of security the KPTI patches were merged to close this hole. The side effect of not having the kernel page tables always mapped means that the TLBs must be flushed more often, causing a performance penalty. By default, Fedora has KPTI enabled to provide security for users. KPTI can be turned off by passing “nopti” on the command line if users don’t want the performance hit.
gronki
Can the patch be disabled to avoid slowdown?
Ryan Lerch
The last paragraph of this article has some details on this.
Steven Miano
The last paragraph says use ‘nopti’ from the command line, and the Red Hat documentation says to use pti=no on the kernel parameter(s)…..
….this might need a bit more clarification.
Source: https://access.redhat.com/articles/3311301#page-table-isolation-pti-6
Content:
“””
Customers and vendors can disable the PTI feature by passing “nopti” to the kernel command line at boot, or dynamically with the runtime debugfs control below:
Raw
# echo 0 > /sys/kernel/debug/x86/pti_enabled
“””
Stephen
nopto and pti=no are the same option, many (all?) boolean kernel options have this variant.
Kurt B
These look to be options to disable kpti on a RHEL kernel. These files don’t appear to exist in a more modern kernel like what is in Fedora. What is the runtime control that is available in the Fedora kernel?
SuperGirl
Question, but the performance isn’t affected with the KPTI? can it make slow our machines?
Joohn
The last paragraph of this article has some details on this.
lemc
I wonder if this “Meltdown” vulnerability will eventually be fixed at the CPU (hardware) level by Intel/AMD. If this happens, (1) would this hardware fix come in the next generation of CPUs (e.g., Intel’s “Cannon Lake”); (2) would the KPTI kernel fix then be disabled/removed ? (2) would normal CPU performance be restored ?
srakitnican
I would not expect hardware to be fixed any time soon even for the current CPU architectures. Meltdown vulnerability seems to affect only Intel CPUs.
https://lkml.org/lkml/2017/12/27/2
Cody
No. I’m pretty sure they’ve said no. But how would they do it anyway? Even if they could do it reasonably it’d be risky. It’d also be financially inadvisable. Think about it: do you really think they’re going to go through that effort when they could sell new chips?
A new design sure but not the old. Same from other chip makers.
Cody
Btw: It’s also ARM CPUs…
Robby O'Connor
I am not entirely sure you’d want to do that. This mitigates a hardware vulnerability. I don’t like the slowdown, but it needs to be there until the hardware fix happens…
Cody
I’m glad to see someone who is being sane on the matter. And by ‘sane’ I mean wise. What is actually going on in anybody’s head – or perhaps ‘not going on’ is more correct – to want to disable a fix to a security flaw in hardware… is beyond me. Let’s hope that they’re not in charge of another person’s security. Truthfully it’s entirely shameful to disabling a security fix for a hardware design flaw. Remember too now that it’s definitely in the open exploits will start being seen ITW. And once that happens these people who haven’t patched or – and I’m sorry but this is an accurate word and perhaps more kind than true – foolishly disabled them are those who will have issues whether definitely or potentially is besides the point.
Matheus Alves
Interesting post. Thanks for the explanation.
Mike
What’s still unclear is how much PCID helps and if what’s needed is PCID alone (like Sandybridge) or INVPCID + PCID in the cpu (such as Haswell and beyond). Can you clarify this?
J. S. Clarke
As far as know (ie the news) this is only an issue with Intel chips. AMD issued a statement to the effect that this exploit is not applicable to their products.
Justin W. Flory
That’s true for the Meltdown vulnerability, but not Spectre. Spectre affects virtually all modern CPU architectures. This article from Data Center Knowledge was insightful for me: http://www.datacenterknowledge.com/manage/how-red-hat-dealing-spectre-cpu-meltdown
Ron Bog
I have an AMD processor. Will the update affect me at all?
danny
if you have amd, then no, the patch for meltdown doesn’t affect you. only intel and a few select arm processors.
Florian Engel
Have you some kind of source for this information? AFAIK, KPTI should contain the former KAISER patch, which fixes a possible attack vector against KASLR . Since AMD CPUs are vulnerable against this attack vector, I do not see why they should not be affected by the KPTI patch. Check the Wikipedia page for more on this attack: https://en.wikipedia.org/wiki/Kernel_page-table_isolation
IMHO, KPTI will affect your system security as well as your performance regardless of your CPU and should not be disabled unless you know exactly what you do.
Florian Engel
To further clarify that: the current kernel version (4.14.11) will affect AMD and Intel CPUs. There is a patch for 4.14.12 which will deactivate PTI by default on AMD CPUs (https://lkml.org/lkml/2017/12/27/2).
AMD states on CNBC that their chips are not susceptible to all three variants of the attack. As a consequenct, they see “near zero risk” for their CPUs (https://www.cnbc.com/2018/01/03/amd-rebukes-intel-says-flaw-poses-near-zero-risk-to-its-chips.html).
I’m assuming that they assess the risk of overcoming KASLR and the consequences of that as “near zero”.
Brem
There are some words used that bring some confusion about the issue.
You start 2 paragraphs with modern processors for desktops…
You are this way telling that the issue is limited to desktops which is not true.
And don’t forget that Fedora is also used on server platforms.
And also, modern processors since 1982 offer different security levels.
Bonos
Will I need to update both hardware firmware AND normal OS updates for the OS fixes to work?
I hope not as last time I tried a bios upgrade I had to recover from a unresponsive black screen, I then learned to stay away from firmware upgrades unless it’s really required….
Justin W. Flory
Hi Bonos – my understanding is that a hardware firmware upgrade is not necessary. Maybe someday there will be patches at the hardware level, but everything I am reading online recommends operating system upgrades, not firmware upgrades. I could be wrong, though.
Justin W. Flory
I stand corrected. A “true” fix seems to be at the hardware level, but the unpatched hardware-level attacks are very difficult to pull off. The recent software patches seem to effective enough to cover most cases. I read this article from Data Center Knowledge that explained it, courtesy of Red Hat’s chief ARM architect, Jon Masters.
Tuna
Jon Masters is unfazed but his statements seem to not be in line with what the papers from mletdownattack.com say (both extremely readable, I hope unis pick these up next week to give students some homework), which also caution that there may be more tricks in the pipeline yet:
MELTDOWN:
Seems to be easy to weaponize (and maybe this is already happening), they are running in a lot of scenarios:
Meltdown breaks all security assumptions given by the CPU’s memory isolation capabilities. We evaluated the attack on modern desktop machines and laptops, as well as servers in the cloud. Meltdown allows an unprivileged process to read data mapped in the kernel address space, including the entire physical memory on Linux and OS X, and a large fraction of the physical memory on Windows. This may include physical memory of other processes, the kernel, and in case of kernel-sharing sand-box solutions (e.g., Docker, LXC) or Xen in paravirtualization mode, memory of the kernel (or hypervisor), and other co-located instances. While the performance heavily depends on the specific machine, e.g., processor speed, TLB and cache sizes, and DRAM speed, we can dump kernel and physical memory with up to 503 KB/s. Hence, an enormous number of systems are affected.
…
We also tried to reproduce the Meltdown bug on several ARM and AMD CPUs. However, we did not manage to successfully leak kernel memory with the attack described in Section 5, neither on ARM nor on AMD. The reasons for this can be manifold. First of all, our implementation might simply be too slow and a more optimized version might succeed. For instance, a more shallow out-of-order execution pipeline could tip the race condition towards against the data leakage. Similarly, if the processor lacks certain features, e.g., no re-order buffer, our current implementation might not be able to leak data. However, for both ARM and AMD, the toy example as described in Section 3 works reliably, indicating that out-of-order execution generally occurs and instructions past illegal memory accesses are also performed.
SPECTRE:
The feasibility of exploitation depends on a number of factors, including aspects of the victim CPU and software and the adversary’s ability to interact with the victim. While network-based attacks are conceivable, situations where an attacker can run code on the same CPU as the victim pose the primary risk. In these cases, exploitation may be straightforward, while other attacks may depend on minutiae such as choices made by the victim’s compiler in allocating registers and memory. Fuzzing
tools can likely be adapted by adversaries to find vulnerabilities in current software. As the attack involves currently-undocumented hardware effects, exploitability of a given software program may vary among processors. For example, some indirect branch redirection tests worked on Skylake but not on Haswell. AMD states that its Ryzen processors have “an artificial intelligence neural network that learns to predict what future pathway an application will take based on past runs”, implying even more complex speculative behavior. As a result, while the stop-gap countermeasures described in the previous section may help limit practical exploits in the short term, there is currently no way to know whether a particular code construction is, or is not, safe across today’s processors – much less
future designs.”
Mehdi
Can someone tell me ow to use nopti? I really hate the performance hit that I am now experiencing.
Tuna
I haven’t tested this but I think one would:
1) edit
and append the ”
” argument to the
string
2) Run
, see https://fedoraproject.org/wiki/GRUB_2
3) Reboot
Loye Young
The press is reporting a 5 to 20% reduction in performance. From an operating system developer’s standpoint, that is a big number because the developer is always looking for ways to scoop up every bit of speed out of the system. The cumulative effect of all the developers’ efforts is huge.
However, from a everyday user standpoint (especially desktop/laptop/tablet user), 5 to 20% is likely not noticeable. The overwhelming majority of the time, the CPU is simply waiting for the user to do something.
For power users, the difference means buying a new rig a few months earlier than you were going to anyway. The speed of hardware doubles (these days) about every two years, so the effect of even a 20% reduction in speed is on the order of 5 to 6 months at the most.
The big hit will be in “the cloud”, on shared hosting, and at corporate data centers. In those situations, the owners of the hardware will need to adjust how many servers they need to keep up with incoming traffic. If CPU speed is the bottleneck, then the incoming traffic will force the managers to have fewer sites/VMs on the same physical server. The additional capital cost will have a real bottom-line effect on profitability.
Happy Trails
Cody
Honestly my i7 4790k I notice no difference whatever. And even my first generation i5 low end has no problem whatsoever (though that’s CentOS and only text it’s my mail server etc.).
So they can say that all they want but I find 5% to 20% extremely high. Other servers I have access to don’t show any slowdown either. Not that it should matter anyway – anyone who disables security fix for performance is foolish at best (esp this type of vulnerability).
Three systems in particular (though I actually have others I can access this should be enough for my point):
model name : Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz
model name : Intel(R) Core(TM) i5 CPU 750 @ 2.67GHz
model name : Intel(R) Xeon(R) CPU E3-1246 v3 @ 3.50GHz
No noticeable performance problems. But personally I would say it’s too soon to be able to verify if any service is going to be affected adversely. And even if it is are you 100% sure of that? You have another system you say? Okay fine but how can you be sure there isn’t anything else going on in the system? And meanwhile while you’re figuring that out you better be recording the other things (which might or might not affect performance).
But in the end it’s stupid to disable security fix for performance.
Tuna
Unless one is running a large number of processes and heavy database/network load, one should not worry about the “slowdown”. If the machine is well isolated from running untrusted code (and this includes JavaScript coming from the Internet), the KPTI can be switched off.
Youtube has a range of explanatory videos right now, e.g. https://www.youtube.com/watch?v=8FFSQwrLsfE
Tuna
The presenter in the video I linked to also says that the exploit was independently discovered by a number of researchers, is easy to hack up (“a few hours”) and, as the embargo on this was about 6 months it would not be surprising that it is currently being exploited by nation states (exploitation does leave not indicators though, so one cannot really know): https://youtu.be/8FFSQwrLsfE?t=2960
So “writing an attack to collect useful data from this exploit can take weeks or months to develop for a single system” … was true in the past.
lsx2007106
the kernel is so big
eert
how updating firmware for my motherboard from linux?
badtux
For those wondering why I would disable a security fix:
1) I have 100% control over my server hardware in my data center, which is used by only by employees of my company that I have 100% monitoring over. No insecure users use that hardware, and only a trusted group of computer programs run on my hardware (not any random software off the Internet ). Thus I gain no real security from the kpti patches.
2) The kpti patches cause slowdowns of up to 30% on i/o intensive workloads like, say, pretty much everything we do, but especially on my database servers.
Thus
3) I can either pass pti=no on the kernel command line, or buy 30% more hardware. The latter is not going to happen. It just won’t.
For vendors running in insecure environments, this is a bigger issue. I imagine Amazon is pulling their hair out right now. But inside a typical corporate data center where only corporate users are allowed and only corporate applications (no untrusted 3rd party applications) are running and said corporate users are heavily monitored? DIsabling the patch is not a big deal, while buying 30% more hardware most certainly is.
And of course for average home computers, you won’t notice a slowdown at all. Most home computers’ cpus are virtually unloaded the vast majority of the time, unlike my virtualization servers, which are typically running at 80% or higher CPU utilization because I want to make use of every one of those expensive megahertz that I have in that data center before I go shopping for more hardware. And most home computers do very little I/O, and when they do, it’s sequential I/O like loading a program or copying large video files from point A to point B that requires very little flipping between user and kernel space. Not like my database server, which is generating thousands of IO’s per second and where a 30% slowdown means my workload can’t process in time.