On a virtual machine monitor like VMware ESXi Server, how are shadow page tables implemented?
I understand that VMMs such as VMware ESXi Server support shadow page tables for mapping guest operating system virtual page addresses directly to machine (hardware) addresses. I was told that the shadow page tables are then used directly by the processor's pager engine to allow memory accesses in the virtual machine without overhead.
I would like to know a little more about how the shadow page table engine works in VMM. Is my high-level understanding above correct? If yes,
-
What data structures are used when implementing shadow page tables?
-
What is the flow of control from the guest operating system to the hardware?
-
Finishing off reading directly from the open source VMM source code, what resources can I explore to learn more about hardware virtualization?
a source to share
Here's what I can say. Please correct me if I am wrong. The dark page table is created and maintained by Hypervisor / VMM. It is a table that contains the guest virtual addresses and the physical addresses of the machine. Imagine that without the page table table, to get to the physical address of the machine, we must first get the virtual address and then go through the OS page table (guest) to get the guest physical address, then we need to convert the guest physical address to the physical address cars. So here's what's going on, see how a single guest virtual address translates to the physical machine address under the senario shadow page table:
-
The first physical processor will see the viral address, and its goal will be to get the physical address of the machine. First, it tries to look at the TLB (translation look towards the buffer), if the entry is in the TLB, we get the address of the machine. This is the simplest case, which we call accidental TLB deletion. No performance problem at all. It will run at what is called native speed.
What happens if there are no TLB gaps in the TLB?
-
If there is no entry in the TLB, the processor will process the page table in the shadow page table. Assuming there is a corresponding mapping (Guest VA to Machine Physical address), the processor will add the value to the TLB, then restart execution and we will be happy about this. This is another good case. It can take about 10 cycles to scan the shadow page table, so we don't have to worry a lot.
What happens if there is no entry in the shadow page table?
-
The processor is scanning the shadow page table and cannot find an entry. Well, in this case, since the search is a privilege, an error will occur. VMM (Virtual Machine Monitor) will look in the guest page table to resolve the issue. This case gets a little more complicated. In any case, when the VMM goes through the guest page table, there will be two possibilities.
-
In the case of a search, an entry is found: When a search has found an entry, we can only walk in the guest page table to finally get the guest physical address. Hey, our goal is to get the address of the physical machine. How do we get there. The monitor will accept a guest physical address and will appear in its PMap table (or structure). If it finds an entry, it will enter the value (mostly the guest virtual address, the physical address of the machine) into the shadow page table. Now we have an entry in the shadow page table, we are good to go like when the processor restart the instruction to get the mapping from the shadow page table. , Ah! forget to mention this case when the monitor does a latent page fault to resolve using PMap or PhysMap to get the corresponding physical machine address.
-
If no entry is found, the monitor (VMM) will introduce a virtual guest page error. Now visiting he sees that there is a page error. OS will come and fix the problem. This can take thousands to one hundred thousand cycles or more in case the page was replaced to disk by a guest. Let's assume the OS (guest OS) solves the problem. We can restart 3.1 steps.
-
Well, the whole flow gets a little complicated. Hope you can understand the process. , Note: The shadow page table is implemented in software such as: VMware, Microsof. It is only used in Binary Translation (BT) mode. In the nested page table, we don't need a shadow page table at all.
There are some problems with the shadow table.
-
We rely on the guest to void the TLB. The point is, we want to maintain consistency between the guest page table and the shadow page table. Imagine what happens if the guest refreshes the page table, what happens if the guest switches the process. It should switch the page table. In this case, I must tell the hardware that I am updating the entry in the page table and I will invalidate it.
-
Requires aggressive caching of hidden page table: We need to freeze the shadow page table. See what happens if a guest makes a context switch and we have a lot of guest processes. He has to report the hardware he has to change this shadow page table pointer. Each switch will flash TLB. Traditionally we have a shadow page table for each running process, but we don't have as much as a shadow page table compared to processes it has a table.
-
Write protection to the guest page table (another word is tracked), to see what happens, for example, the page was blocked by the operating system for some reason, we need to get information.
a source to share
more than 50 views and nothing?
I spoke to Vidar Holen via IRC on ## linux at freenode.net. He suggested that I take a look at this AMD technical report . This turned out to be a great resource. Anyone else have any other suggestions?
a source to share
Basically, the guest OS will try to translate the virtual address to a physical address, but this seemingly physical address is not actually a real physical address as they come from the VMM / hypervisor and hence these addresses are not contiguous addresses like this is the case with a conventional OS without a VM. Thus, another translation is required to transfer this physical guest address to the real machine address. For this, the VMM / hypervisor maintains shadow page tables to map these physical addresses to physical addresses.
Also, the hardware provides a mechanism to avoid moving the page table by exposing the TLB, but if you can imagine, these TLBs inside the guest don't have to be real hardware TLBs, and the VMM / hypervisor has to mimic them in some way. At the same time, the shadow page tables can be used as a TLB for the guest.
So this is the basic idea behind the shadow page table, but it is probably the most complex technology in hardware virtualization technology. I forgot a lot of details and hacks that I also don't quite understand.
Below is a link that talks about some of these problems with simplified shadow page tables and how kvm tries to avoid them.
http://lwn.net/Articles/216794/
Another thing is that there is also hardware support for this mechanism, and they are called EPT and NPT, supported by both intel and amd.
NTN.
a source to share