In part 1, I explained the virtual machine memory settings that are available to us. This caused quite some controversy on what would happen when a limit is hit that’s configured lower than the actual allocated memory.
Scott Herold did an excellent follow-up on this. Be sure to read his article too, because it contains valuable information.
This time I’ll explain how the ESX kernel assigns memory to a VM. To explain how ESX memory management works, we first need to clarify some definitions, which are often unclear and used interchangeably in many writings.
- Machine memory: this is the physical memory installed in the ESX host and is managed by the ESX kernel.
- Physical memory: this is the allocated memory to the guest, which the guest OS sees as its physical memory (since the guest OS is unaware that it’s running virtual!). This memory is managed by the guest OS.
- Virtual memory: this is the memory as seen from the applications running inside the guest OS.
Virtual Machine memory usage
Now lets take a look at how memory is assigned to the guest. Whenever an application needs a memory page for storing data, it requests it through a system call to the operating system. The operating system keeps tracks of which memory pages are in use and which are free. Imagine this as two simple lists, one for the free memory pages and one for the allocated pages. So when a system call is made by the application, the operating system will look on its list of free memory pages and assign a page to the application. The operating system then moves this page from the “free” list to the “allocated” list.
On the hypervisor layer, memory is allocated on demand. When the vm accesses its physical memory for the first time, the hypervisor will assign a machine memory page to the vm. The hypervisor keeps a record for this assignment in a Physical Memory Page to Machine Memory Page (PMP-MMP) mapping table for every vm, so the hypervisor knows which page is in use and what its location in the machine memory is. This process is illustrated in Figure1.
So we now know how memory is assigned, but what happens when the application no longer needs the memory page. In this case the application will free the memory page again through a system call to the operating system. The operating system will then move this memory page from the “allocated” list to the “free” list. Because the operating system is unaware that it’s actually virtualized there is no interaction with the hypervisor. The underlying hypervisor is not aware that the memory page is now actually free. So while memory pages are constantly allocated and freed by the guest operating system inside the vm, the hypervisor can only allocate memory to the vm. The hypervisor can’t reclaim memory through guest frees.
When we take a quick look at the summary tab on our vm in vCenter, we see host memory usage and guest memory usage. Host memory usage is the amount of machine memory in MB allocated to the guest as previously described. Remember however that this value also includes virtualization overhead (e.g. the PMP-MMP mapping table). On a non-memory overcommitted host this represents a “high water mark” on the guest’s memory usage, but host memory usage is based on shares when the host is overcommitted.
At the same page we also see guest memory usage. This is the amount of memory in MB that is actively used by the guest’s OS and its applications. But didn’t I just mention before that the hypervisor is unaware of the actual state of a guest’s physical memory page?
Well that’s correct, and this value is actually calculated through statistical sampling. This means that the hypervisor selects a random subset of the guest’s memory and monitors how many pages are accessed in a certain interval. This value is than extrapolated to the full size of the guest’s memory and shown as guest memory usage.
Why is there a difference between this guest memory usage value from ESX and from inside the guest? This is because the guest’s OS has a much better visibility of what memory is actively used. The ESX hypervisor can only use sampling to guess the “active” memory size.
Figure2 schematically represents guest memory states. From the total allocated memory to the guest (guest’s physical memory), memory can be either allocated to the OS and applications (and subsequently allocated from machine memory) or unallocated. The allocated memory can be either active, or inactive.
For now this concludes part2. After knowing how memory is assigned we’re going to look how memory can be reclaimed by the ESX kernel next time.
Continue reading Part3
- ESX Memory Management – Part 1 Tweet I receive a lot of questions lately about ESX memory management. Things that are very obvious to me seem to be not so obvious at all for some other...
- VMware ESX(i) 3.5 Update4 released Tweet VMware has released ESX(i) 3.5 update4. Do not forget to read the release notes here or you can go to the download page here While going through the release notes...
- ESX console password aging Tweet Yesterday I did a post on how to change your ESX root password using a Powershell script and told you that I, as a good administrator, didn’t change my...
- Unable to login to your ESX server Tweet Ivo Beerens posted this article last week on the defunct cimservera processes that render an ESX Host unmanageable. See also this VMWare KB Article. Symptoms include: Unable to log...
- Unattended upgrade of HP management agents Tweet After upgrading to ESX 3.5 to update3, I found out that the HP management agents needed to be upgraded to version 8.1.1, since this version supports ESX3.5 update3. So...