ESX Memory Management – Part 2

In part 1, I explained the virtual machine memory settings that are available to us. This caused quite some controversy on what would happen when a limit is hit that’s configured lower than the actual allocated memory.
Scott Herold did an excellent follow-up on this. Be sure to read his article too, because it contains valuable information.

This time I’ll explain how the ESX kernel assigns memory to a VM. To explain how ESX memory management works, we first need to clarify some definitions, which are often unclear and used interchangeably in many writings.

  • Machine memory: this is the physical memory installed in the ESX host and is managed by the ESX kernel.
  • Physical memory: this is the allocated memory to the guest, which the guest OS sees as its physical memory (since the guest OS is unaware that it’s running virtual!). This memory is managed by the guest OS.
  • Virtual memory: this is the memory as seen from the applications running inside the guest OS.


Virtual Machine memory usage

Now lets take a look at how memory is assigned to the guest. Whenever an application needs a memory page for storing data, it requests it through a system call to the operating system. The operating system keeps tracks of which memory  pages are in use and which are free. Imagine this as two simple lists, one for the free memory pages and one for the allocated pages. So when a system call is made by the application, the operating system will look on its list of free memory pages and assign a page to the application. The operating system then moves this page from the “free” list to the “allocated” list.

On the hypervisor layer, memory is allocated on demand. When the vm accesses its physical memory for the first time, the hypervisor will assign a machine memory page to the vm. The hypervisor keeps a record for this assignment in a Physical Memory Page to Machine Memory Page (PMP-MMP) mapping table for every vm, so the hypervisor knows which page is in use and what its location in the machine memory is. This process is illustrated in Figure1.

So we now know how memory is assigned, but what happens when the application no longer needs the memory page. In this case the application will free the memory page again through a system call to the operating system. The operating system will then move this memory page from the “allocated” list to the “free” list. Because the operating system is unaware that it’s actually virtualized there is no interaction with the hypervisor. The underlying hypervisor is not aware that the memory page is now actually free. So while memory pages are constantly allocated and freed by the guest operating system inside the vm, the hypervisor can only allocate memory to the vm. The hypervisor can’t reclaim memory through guest frees.

When we take a quick look at the summary tab on our vm in vCenter, we see host memory usage and guest memory usage. Host memory usage is the amount of machine memory in MB allocated to the guest as previously described. Remember however that this value also includes virtualization overhead (e.g. the PMP-MMP mapping table). On a non-memory overcommitted host this represents a “high water mark” on the guest’s memory usage, but host memory usage is based on shares when the host is overcommitted.

At the same page we also see guest memory usage. This is the amount of memory in MB that is actively used by the guest’s OS and its applications. But didn’t I just mention before that the hypervisor is unaware of the actual state of a guest’s physical memory page?
Well that’s correct, and this value is actually calculated through statistical sampling. This means that the hypervisor selects a random subset of the guest’s memory and monitors how many pages are accessed in a certain interval. This value is than extrapolated to the full size of the guest’s memory and shown as guest memory usage.
Why is there a difference between this guest memory usage value from ESX and from inside the guest? This is because the guest’s OS has a much better visibility of what memory is actively used. The ESX hypervisor can only use sampling to guess the “active” memory size.



Figure2 schematically represents guest memory states. From the total allocated memory to the guest (guest’s physical memory), memory can be either allocated to the OS and applications (and subsequently allocated from machine memory) or unallocated. The allocated memory can be either active, or inactive.

For now this concludes part2. After knowing how memory is assigned we’re going to look how memory can be reclaimed by the ESX kernel next time.

Continue reading Part3

Related posts:

  1. ESX Memory Management – Part 1 Tweet I receive a lot of questions lately about ESX memory management. Things that are very obvious to me seem to be not so obvious at all for some other...
  2. VMware ESX(i) 3.5 Update4 released Tweet VMware has released ESX(i) 3.5 update4. Do not forget to read the release notes here or you can go to the download page here While going through the release notes...
  3. ESX console password aging Tweet Yesterday I did a post on how to change your ESX root password using a Powershell script and told you that I, as a good administrator, didn’t change my...
  4. Unable to login to your ESX server Tweet Ivo Beerens posted this article last week on the defunct cimservera processes that render an ESX Host unmanageable. See also this VMWare KB Article. Symptoms include: Unable to log...
  5. Unattended upgrade of HP management agents Tweet After upgrading to ESX 3.5 to update3, I found out that the HP management agents needed to be upgraded to version 8.1.1, since this version supports ESX3.5 update3. So...

15 Comments on “ESX Memory Management – Part 2”

  1. #1 Twitted by jasonboche
    on May 4th, 2009 at 12:40 pm

    [...] This post was Twitted by jasonboche – [...]

  2. #2 Joel
    on May 5th, 2009 at 2:11 pm

    I wonder if the agent that’s installed on guests hooks into system calls to notify ESX of when memory is freed.

  3. #3 Arnim van Lieshout
    on May 5th, 2009 at 8:43 pm

    I suppose you mean VMware Tools.
    The answer is no. VMware tools do not hook into system calls.
    Hooking into system calls can be difficult and maybe needs changing the OS. Also it depends on the guest type and even things can be different from release to release. So VMware chose a safer and simpler approach by introducing the balloon driver.

  4. #4 VMware ESX minnehåndtering | Lars Jostein Silihagen
    on May 31st, 2009 at 9:48 pm

    [...] Part2: Hvordan den enkelte virtuelle maskin bruker minne [...]

  5. #5 The Return of Virtualization Short Takes - - The weblog of an IT pro specializing in virtualization, storage, and servers
    on Jun 5th, 2009 at 6:44 pm

    [...] by a series of blog posts by Arnim van Lieshout on VMware ESX memory management (Part 1, Part 2, and Part 3), Scott Herold decided to join the fray with this blog post. Both Scott’s post [...]

  6. #6 ESX Memory Management – Part 3 | Arnim van Lieshout
    on Aug 10th, 2009 at 10:37 am

    [...] part 1, I explained the virtual machine settings that are available to us regarding to memory, in part 2, I explained how the ESX kernel assigns memory to a VM and in this part I will dive into ESX memory [...]

  7. #7 VMware: Terminal server performance tuning on VMware ESX |
    on Sep 17th, 2009 at 8:38 pm

    [...] ESX Memory Management – Part 2 Categories: Citrix, Microsoft, VMware Tags: Citrix, ESX, Performance, Policy, VMware, Windows 2003 Comments (0) Trackbacks (0) Leave a comment Trackback [...]

  8. #8 Trent
    on Nov 13th, 2009 at 7:48 pm

    Arnim, I have a question which I believe may be memory related. I have an ESX server 3.5u4 with 15 VM’s. When a VM sits for a while, two days, with little to no use/activity the restart time for a jboss java application is very slow, about 4 times longer. However, after the restart I can initiate another restart which takes what I would call the typical amount of time to restart. Is this because memory has been swapped out or it is some other ESX issue. Any guidance would be appreciated.

  9. #9 Arnim van Lieshout
    on Nov 19th, 2009 at 1:02 am

    Hi Trent,

    It could be memory related. See if memory resources are scarce on your host (check for ballooning/swapping on running vms on that host). When there’s memory contention on your host, and that particular vm is idle, than the share-system in combition with the idle memory tax can reclaim memory from that vm. When you restart java, the java process reclaims the memory it needs, but when the host doesn’t have the memory available, it first has to free memory from other vms before it can be assigned to that vm and the java process. This will impact startup performance for java, since java claims a preconfigured amount of memory on startup. After the first startup, the memory is seen as active and not swapped out by the hypervisor directly, so any consecutive startups will have the memory available right away.
    Try setting a reservation for that vm, to prevent ESX from swapping that memory out to disk.
    As I said before, to be sure if the problem is memory related, you need to investigate what happens on your ESX host when you first restart that java app. Monitor ESX behaviour with esxtop.


  10. #10 Trent
    on Nov 19th, 2009 at 1:07 am

    Thank you for the response. I will dig deeper into the issue and will also try using memory reservations.

    On the surface there doesn’t seem to be a lack of physical memory, the memory usage on the ESX host is usually 60 – 70%.

    Again thank you.

  11. #11 Support your favourite blog. Vote Now! | Arnim van Lieshout
    on Jan 6th, 2010 at 11:39 am

    [...] Memory Management – Part 1, Part 2 & Part [...]

  12. #12 Diagram: ESX Memory Management and Monitoring v1.0 | HyperViZor
    on Jan 22nd, 2010 at 10:02 pm

    [...] Management in VMware ESX Server – by Carl A. Waldspurger – Arnim van Lieshout Blog (Part-1 , Part-2, [...]

  13. #13 VCAP-DCA Study notes – 3.1 Tune and Optimize vSphere Performance |
    on Apr 16th, 2011 at 12:34 pm

    [...] this great series of blog posts from Arnim Van Lieshout on memory management – part one, two and three. And as always the Frank Denneman [...]

  14. #14 ESX Memory Management – Part 3 | Digital Beermat
    on Jun 15th, 2012 at 12:23 pm

    [...] In the background, the ESX kernel frees up the machine memory page that corresponds to the physical machine memory page allocated to the balloon driver. When there is enough memory reclaimed, the balloon driver will deflate after some time returning physical memory pages to the guest OS again. This process will also decrease the Host Memory Usage parameter (discussed inpart 2) [...]

  15. #15 VMware ESX内存管理 | 甜瓜地—IT技术博客
    on Sep 13th, 2012 at 4:15 pm

    [...] 原文:在了解ESX内存管理如何工作前, 需弄清以下三个定义.  [...]

Leave a Comment