Virtualization, Cloud, Infrastructure and all that stuff in-between

My ramblings on the stuff that holds it all together

Category Archives: VCP

8 Node ESXi Cluster running 60 Virtual Machines – all Running from a Single 500GBP Physical Server


I am currently presenting a follow-up to my previous vTARDIS session for the London VMware Users Group where I demonstrated a 2-node ESX cluster on cheap PC-grade hardware (ML-115g5).

The goal of this build is to create a system you can use for VCP and VCDX type study without spending thousands on normal production type hardware (see the slides at the end of this page for more info on why this is useful..) – Techhead and I have a series of joint postings in the pipeline about how to configure the environment and the best hardware to use.

As a bit of a tangent I have been seeing how complex an environment I can get out of a single server (which I have dubbed v.T.A.R.D.I.S: Nano Edition) using virtualized ESXi hosts, the goals were;

  • Distributed vSwitch and/or Cisco NX100V
  • Cluster with HA/DRS enabled
  • Large number of virtual machines
  • Single cheap server solution
  • No External hardware networking (all internal v/dvSwitch traffic)

The main stumbling block I ran into with the previous build was the performance of the SATA hard disks I was using, SCSI was out of my budget and SATA soon gets bogged down with concurrent requests which makes it slow; so I started to investigate solid state storage (previous posts here).

By keeping the virtual machine configurations light and using thin-provisioning I hoped to squeeze a lot of virtual machines into a single disk, previous findings seem to prove that cheap-er consumer grade SSD’s can support massive amount of IOps when compared to SATA (Eric Sloof has a similar post on this here)

So, I voted with my credit card and purchased one of these from Amazon – it wasn’t “cheap” at c.£200 but it will let me scale my environment bigger than I could previously manage which means less power, cost, CO2 and all the other usual arguments you try to convince yourself that a gadget is REQUIRED.

So the configuration I ended up with is as follows;

1 x HP ML115G5, 8Gb RAM, 144Gb SATA HDD c.£300 (see here) but with more RAM
1 x 128Gb Kingston 2.5” SSDNow V-Series SSD c£205

I installed ESX4U1 classic on the physical hardware then installed 8 x ESXi 4U1 instances as virtual machines inside that ESX installation


This diagram shows the physical server’s network configuration


In order for virtualized ESXi instances to talk to each other you need to update the security setting on the physical host’s vSwitch only as shown below;


This diagram shows the virtual network configuration within each virtualized ESXi VM with vSwitch and dvSwitch config side-side.


I then built a Windows 2008R2 Virtual Machine with vCenter 4 Update 1 as a virtual machine and added all the hosts to it to manage

I clustered all the virtual ESXi instances into a single DRS/HA cluster (turning off admission control as we will be heavily oversubscribing the resources of the cluster and this is just a lab/PoC setup


Cluster Summary – 8 x virtualized ESXi instances – note the heavy RAM oversubscription, this server only has 8Gb of physical RAM – the cluster thinks it has nearly 64Gb



I then built an OpenFiler Virtual Machine and hooked it up to the internal vSwitch so that the virtualized ESXi VMs can access it via iSCSI, it has a virtual disk installed on the SSD presenting a 30Gb VMFS volume over iSCSI to the virtual cluster nodes (and all the iSCSI traffic is essentially in-memory as there is no physical networking for it to traverse.


Each virtualized ESXi node then runs a number of nested virtual machines (VM’s running inside VMs)

In order to get Nested virtual machines to work; you need to enable this setting on each virtualized ESXi host (the nested VM’s themselves don’t need any special configuration)


Once this was done and all my ESXi nodes were running and settled down, I have a script to build out a whole bunch of nested virtual machines to execute on my 8-node cluster. the VM’s aren’t anything special – each has 512Mb allocated to it and won’t actually boot past the BIOS because my goal here is just to simulate a large number of virtual machines and their configuration within vCenter, rather than meet an actual workload – remember this is a single server configuration and you can’t override the laws of physics, there is only really 8Gb or RAM and 4 CPU cores available.

Each of the virtual machines was connected to a dvSwitch for VM traffic – which you can see here in action (the dvUplink is actually a virtual NIC on the ESXi host).



I power up the virtual machines in batches of 10 to avoid swamping the host, but the SSD is holding up very well against the I/O

With all 60 of the nested VMs and virtualized ESXi instances loaded these are the load stats










I left it to idle overnight and these are the performance charts for the physical host; the big spike @15:00 was the scripts running to deploy the 60 virtual machines


Disk Latency


Physical memory consumption – still a way to go to get it to 8Gb – who says oversubscription has no use? 🙂

image image

So, in conclusion – this shows that you can host a large number of virtual machines for a lab setup, this obviously isn’t of much use in a production environment because as soon as those 60VM’s actually start doing something they will consume real memory and CPU and you will run out of raw resources.

The key to making this usable is the solid state disk – in my previous experiments I found SATA disks just got soaked under load and caused things like access to the VMFS to fail (see this post for more details)

Whilst not a production solution, this sort of setup is ideal for VCP/VCDX study as it allows you to play with all the enterprise level features like dvSwitch and DRS/HA that really need more than just a couple of hosts and VMs to understand how they really work. for example; you can power-off one of the virtual ESXi nodes to simulate a host failure and invoke the HA response, similarly you can disconnect the virtual NIC from the ESXi VM to simulate the host isolation response.

Whilst this post has focused on non-production/lab scenarios it could be used to test VMware patch releases for production services if you are short on hardware and you can quite happily run Update manager in this solution.

If you run this lab at home it’s also very power-efficient and quiet, there are no external cables or switches other than a cross-over cable to a laptop to run the VI Client and administer it; you could comfortably have it in your house without it bothering anyone – and with an SSD there is no hard disk noise under load either 🙂

Thin-provisioning also makes good use of an SSD in this situation as this screenshot from a 30Gb virtual VMFS volume shows.


The only thing  you won’t be able to play around with seriously in this environment is the new VMware FT feature – it is possible to enable it using the information in this post and learn how to enable/disable but it won’t remain stable and the secondary VM will loose sync with the primary after a while as it doesn’t seem to work very well as a nested VM. If you need to use FT for now you’ll need at least 2 physical FT servers (as shown in the original vTARDIS demo)

If you are wondering how noisy it it at power-up/down TechHead has this video on YouTube showing the scary sounding start-up noise but how quiet it gets once the fan control kicks-in.

ML115 G5 Start-up Noise

Having completed my VCP4 and 3 I’m on the path to my VCDX and next up is the enterprise exam so this lab is going to be key to my study when the vSphere exams are released.

Get a move on and do your VCP4 upgrade


If you are a VCP3 you’ll need to get a move on and upgrade your certification to VCP4 unless you have time to sit (and pay for) some classroom training next year – you need to have passed the exam before December 31st 2009 (i.e in 43 days time!)

Also bear in mind there might be a bit of a rush – anyone else remember the NT4 MCSE –> one-shot Windows 2000 upgrade exam? there will be a lot of people in the same boat as you (and I!) and time is running out, this is especially a problem if you only have access to a limited number of testing centres where you live as they will be getting booked up.

As some insurance VMware are also offering a free re-take at present; but there is a catch – you have to wait for a voucher to be emailed to you before you can book your exam with free re-take option – and it says the Friday following your registration – so bear this delay in mind if you want to take this option.

If you are a VCP3 you should have received an email from VMware with a link to register for

For the official word on what you need to do – go here – you’ll also need your VMware myLearn username and password (which is recoverable from the site if you’ve forgotten it)

You can register for the 2nd shot option here

It seems a bit odd, but you need to register for this “virtual class” to be issued the voucher (screen cap of successful registration below..)


I am now waiting for my voucher via email* so I can register for my VCP exam with free re-take option – check this post for links to my study materials and another plug for Simon Long’s excellent resources here, oh and I need some time to take the exam as well 🙂


*You may want to check the email address registered with your myLearn account is valid/correct



I had my 2nd attempt at the VMWare FastTrack course last week (the 1st attempt was cut short due to power problems, and I’ve only just managed to find the time to reschedule).

The course went ok, the 1st 2 days were a bit dull as I’d already sat through them once; the course has been restructured since and is now contiguous through the 2 books, in the 1st attempt we were jumping back and forth as the Fast Track course is the Install & Configure and DSA course mashed together.

I found the pace ok, infact I could probably have been pushed a lot harder, as such I didn’t find it as “intensive or extended hours” – we finished by at least 5.30 most days and earlier some other days – the last day did feel a bit like treading water as it was quite spread out.

Wasn’t overly impressed by the facilities at QA-IQ’s Roseberry Ave – could do with a lick of paint, some better lighting and A/C that works properly. In all fairness it did look like they were in the middle of refitting it. More importantly the instructor was good and the kit/resources worked as required – no free lunch though 😉

No VCP exam voucher is included with QA-IQ as you get at DNS arrow – considering the QA and DNS courses are virtually the same price I think that’s a bit cheap – so you may want to check with your vendor before booking.

There is a lot of team working in building up DRS clusters and doing HA etc. and you have to have sat the course in order to be officially certified as a VCP.

I sat my exam yesterday morning and passed – not by as much as I’d have liked, but I was a bit lazy and didn’t do much (any) specific exam prep – there were a whole load of questions on a particular subject that I had not revisited since the course* and I fell foul of the “mark for review” option where you can go back at the end and can review/change your answers before submitting the exam – several of them from correct to incorrect as I later worked out – d’oh I learnt (and subsequently) forgot that from my MCSE exam days – if you don’t know 100% your 1st instinct was probably right.

Ah well, another one down – must get round to updating my MCSE too.. I quite like the certification exams, it’s just finding the time to learn the MS/VMWare answers and I’m lucky that English is my 1st language as I think a lot of the cert questions (not just VMWare – MS, Cisco etc.) are more about English comprehension and understanding what they are actually asking in order to answer correctly.


*As usual I had to sign an NDA, so no I can’t say what they were – sorry

VMWare Fast Track VCP Course – Lights Out Edition


I have been on a VMWare fast track course this week to complete my VCP cert; annoyingly the VCP is one of those vendor certifications where you need to prove you’ve sat the course as well as doing the exam to qualify as a “proper” VCP.

I’ve been doing it at QA-IQ in central London, and whilst I know most of the content its good for filling in the gaps and detail for the areas where my knowledge is high-level.

Unfortunately the building lost all power 1/2 way through yesterday and we had to finish the remaining theory for the day without PC’s or projectors etc. – full credit to the instructor who didn’t break a sweat doing so, I’d probably have sworn a lot more.

When I arrived this morning they still haven’t got power back on so the day has been cancelled, waiting for a call from our account manager to work out what to do next. it’s already a compressed version of 2 training courses and my schedule is pretty tight so wonder what’s going to happen.

EDF the electricity supplier are bringing in generators – so someone must have really broken it!