Virtual Machine Optimization

This afternoon I'm working on creating some core virtual machine hard drives with my colleague Pete at Improving.  I found this excellent article on creating and optimizing VHDs using differencing.  But, it doesn't answer a question I have about sharing base VHDs.  My understanding of the NT kernel (the core OS for Windows Server/XP/Vista) is that memory-mapped file IO gives the benefit of only one in-memory copy of any read-only file regardless of how many processes load that file into their working set.  I get this idea from Helen Custer's old book on with Windows NT operating system, which I read when it first came out.   

So, I have this theory for which I can't find any others confirming or denying.  It goes like this:

* Create a base VHD with the core operating system for use in multiple machines (a la the article).

* Mark the base VHD read-only.

* Create a new VPC/Virtual Server/VM using a differencing file on the base VHD.

* Create a second VM using a second differencing file on the base VHD.

* Start both VMs at once.  The base VHD should only physically be loaded into physical RAM once!  The physical memory of the file will be logically mapped into each of the VM processes, but won't consume any more physical or paged memory on the host machine.

I guess I have to actually run the experiment to prove this as no one I can find seems to be thinking about this memory-mapping concept enough to make a statement one way or another.  Google has failed me. :(

Comments

[Daniel Fisher]

Any ideas if this will work if i run sysprep on the differencing file to have different sys-IDs? Best regards, Daniel

[Todd Girvin]

We proved my hypothesis! Pete and I tested this with Hyper-V on Server 2008. We have a base VHD that's about 8GB and two separate diff drives for two separate machines. I think the diff drives are about 4KB in size. On a host machine with 8GB of RAM, we loaded on VM and had 3600MB free. After loading the second VM, we had 3300MB free. So, it only took about 300MB to load the second VM. At this rate, I should be able to run 8 to 10 different VMs with a shared base drive on this 8GB machine. Of course, mileage may vary as the difference between machines increase. Once we install Exchange on one VM and MOSS on another, I expect them to start consuming more memory, but the core OS will at be shared. Daniel, yes this will work regardless of the changes you make at the diff drive level. You can sysprep, change the machine IDs and names, whatever you want. Each area of the system that you change compared to the original system will create a changed page in the difference drive. If all the machines change those pages, then the base page won't even be loaded from the base VHD. Only the un-difference pages from the base VHD will be mapped into physical RAM.

[Steve Campbell]

Isn't the VHD just hard-disk, i.e. not loaded into RAM? My understanding is that memory used by each machine is determined by the amount of RAM assigned to the machine, not the VHD size.

[Todd Girvin]

Regarding a VHD being just a hard-disk, the answer is yes - and no. The VHD is a file that acts as a hard-disk to the Virtual Machine process. Inside that file are pages that contain files opened and loaded by the processes in the VM. Each of those pages that get loaded consume RAM in the VM and therefore in the host machine. It's true that the RAM assigned to the VM affects the RAM allocated from the host. But performance of the host and consequently the guest VMs is also affected by paging and efficient use of RAM. If the non-allocated RAM pages are filled with different read-only pages from different VHDs, eventually you'll run out of spare pages and have to drop files to load others. But, if all VMs share the same pages from the same VHD, once those files are loaded into physical RAM, they can be shared by all the VMs and therefore there's less competition for pages, less contention, fewer files getting dropped, less paging and reloading, and everything runs faster. Sharing VHDs between VMs should greatly increase performance in a single host.

[www.google.com/accounts/o8/id?id=AItOawkBlbLKpgNKCEQciNzIB340Gf3FfoIoOk0]

I like your idea, and glad to hear that it worked. I am curious if after almost 2 years, has this model worked out in practical application? For example; how did you manage system updates? My concern would be, did the subset linked VHD's grow to substantially large amounts, negating any practical use? Or does even a large subset VM still have better overall performance?