Virtualization, Cloud, Infrastructure and all that stuff in-between

My ramblings on the stuff that holds it all together

Category Archives: VCDX

Peer to Peer VCDX Defense Workshop at UK VMUG

I have an idea for a side-workshop/session at the upcoming UK VMUG in November for people that are actively following the VCDX4/5 track thus I am polling for interest via this post…

The format would be that participants get a 15-20min slot to present their design defence to a “panel” comprising of other VMUG members. including some of whom have taken (and failed) the actual defense process with a view to providing feedback and some questioning.

We don’t have any VCDX secret-sauce to share or special VMware info that you can’t find on the web, but the goal is to allow you to dry-run your presentation in-front of a friendly audience of like-minded community members before you brave it for real.

You would need to commit to come prepared with a Powerpoint version of your defence presentation, the panel won’t have seen your full design documents, and won’t be reviewing them.

I have created a poll, so if you’re interesting in participating, and will be attending the UK VMUG in November (details here) then please let me know via the poll below so I can estimate interest and seek panel members from the community.

Because we will need to arrange logistics I would ask that you only express an interest if you are planning to attend the UK VMUG and will have something to present, a significant number of no-shows will make it difficult to justify putting this on.

Please vote below (Poll is open until the end of September)


Bad News My VCDX Design Defense Experience

With apologies to all fellow Brits for the US spelling (Google, you know Winking smile).

Following on from email with my good news… there was a further email with some less positive news..

As some of you know I made a stealth appearance at Partner Exchange in Orlando a week or so ago, I was there to do my VCDX defense – I didn’t announce it in a public sense, not from fear/risk of failure – but more because I didn’t really want people reminding me how tough it was, how low the pass rate is, how much I’ll be grilled until I bleed etc. as we in the Internet-space feel compelled to do Smile personally I like to just get on with difficult things, rather than navel gaze and consider the worst.

Not many people have posted on their experiences of not passing the VCDX, and that does seem a bit unbalanced considering the pass/fail ratio; that’s understandable as we don’t always like discussing things less positive about ourselves – but here it is, for the benefit of anyone else considering it, whilst my initial attempt at VCDX was unsuccessful I quite enjoyed the process and I think it’s a good learning experience for anyone.

I had been on the VCDX3 track for a while and already had my design well underway before I joined VMware, mine was a real design, done for a customer – anonymised as required and I considered this quite an important fact – whilst fictional designs are acceptable I think it’s better to use a real one as you have a much more realistic set of requirements and some real-world challenges to overcome.

I asked a lot of questions when I joined VMware in September, and am pleased to report that whilst I work with several well-known VCDX panelists, the process for VMware employees is exactly the same as it is for customers, partners and the public, there are no internal shortcuts or special favours for VMware staff –which was important to me as I didn’t want to feel being part of VMware “entitled” me in anyway, many VMware staff have passed the VCDX, but likewise many have failed too. My colleagues excused themselves from participating on my defence and this is normal practice across VMware to avoid any kind of personal involvement or accusation of favours.

With the PEX sessions being announced as the last chance for the VCDX3 track I ploughed on and submitted my documents. I had my pack returned twice for further materials – I was missing a BC/DR and operation procedures section in my package, this was because BC/DR was out of scope in this phase of my design project (although it was discussed at a high-level) and operational processes were established already, my project merely added some extra parts, I explained this fact in my original design submission.

To meet the requirements for the re-submission I had to build some documentation to embellish these parts of my design, adding detail where it was missing for the re-submission, yes I essentially made it up – this was a bit odd for me but is acceptable within the scope of the VCDX; many people are discussing the VCDX design submission in terms of it being a portfolio/show-case of your work which is a good analogy – build upon what was actually done but add to it to show the quality of what you could do (given ability to do so).

My design wasn’t exclusively vSphere, it involved a whole bunch of hardware, Windows, Firewalling and other items but I was lucky that the standard approach I use already broke out requirements, assumptions, risks etc. rather than focusing on the purely technical aspects as some people do – and that whole requirements vs. technical choice vs. justification is a key part of “architecture” – delivering a solution to business requirements rather than some clever technical-wizzery.

I found it quite a slog doing this final part of the process, I would estimate that I spent the following on my design;

  • 30-40 hours on initial design – which was for a customer project and something I had to do during the course of my normal day-job.
  • 5 hours on VCDX application form; cross-referencing, calling out requirements etc.
  • 8-10 hours on tweaking/clarifying document for VCDX submission (this was post project completion so I could address some issues that were encountered during the implementation)
  • 10 hours to provide further details on operational processes, installation procedures, clarification of a few technical points & re-doing cross-referencing with VCDX application form (resubmission 1, 1 week to resubmit)
  • 15 hours to provide detailed operational processes, essentially a run-book for the platform, re-doing cross-referencing with VCDX application form (resubmission 2, 1 week to resubmit)

This point got my documents accepted and I was granted a date for a defence at PEX and I started my defence preparations.

I felt pretty well prepared and I’ve been a consultant/architect for a number of years so presenting, discussing and reviewing designs in front of customers is something I’m comfortable doing, I spent my time reviewing my design and building my presentation.

I hadn’t specifically addressed BC/DR in my design other than noting the risk and a manual rebuild process for phase 1; I spent some time building some slides that addressed the high-level BC/DR plan that was anticipated for later phases of my (real-world) project.

I estimate that my defence preparations were about the following

  • 5-8 hours building Executive Summary Presentation, calling out key decision points and copies of the relevant diagrams in-case I needed to refer back to them during the defence session, making copious notes, references etc.
  • 10-15 hours reviewing design document, checking against published materials – white papers and books etc
  • I didn’t practice or dry-run my presentation, that doesn’t work for me but I made sure I knew the key points and what was where in my slide-deck.

I got to the session on-time and with plenty of time to spare, made sure I had lunch and was ready to go; there was a pretty lengthy delay before I got started due to some logistical issue and whilst I didn’t feel nervous this didn’t really help – I wanted to get going, and done.

I didn’t find my panel intimidating, they were all courteous and professional with a moderator from the certification team to handle timings, logistics etc. I gave my exec summary presentation and answered questions from the panel as they came – unfortunately for me this meant I didn’t get to parts of my presentation calling out key decision points and BC/DR etc. but I don’t know if this caused an issue in terms of overall scoring.

They dived into sections of my design covering all the typical functional elements of a vSphere design – asking me to whiteboard, explain and justify some decisions that were made – I dropped several klangers here, whilst I can’t go into details, I knew my design but I had some errors around storage and misunderstood some parts of my multi-pathing configuration, I had reviewed this a number of times before my defence and submission and had changed and muddled it in my head, which didn’t do me any favours when I was asked to detail it and explain.

The panelists weren’t personal or condescending about this, if anything they were more poker faced and gave me an opportunity to explain it as I saw it.

I completed the design defence part feeling that it hadn’t gone too badly, but expecting that my unclear thinking and explanation would hurt my overall performance.

After a short comfort break there was a design workshop scenario type session, given a number of requirements and some background information – I think I was a bit muddled on my dealing with this section although it’s something I’m used to doing in a professional sense – white-boarding and thinking on my feet is ok but I found it hard to come up with and clearly express a solution to the presented problem.

The final session was a troubleshooting exercise, again I was presented with a situation which was articulated via some prepared text, a presentation and some pre-prepared screen-grabs, whilst I didn’t get to the root cause I think I followed a reasonable troubleshooting path to get there, and eliminated some red-herrings.

For both the design and troubleshooting scenarios I understand it’s more about the journey than arriving at a conclusion – articulating your thought processes, rationale and goals – you have access to a whiteboard and are encouraged to use it – I found it particularly useful to make notes/”park” bits of information when given and to say what I was thinking rather than just going silent (even when my brain was actually empty Smile).

So, with that I had finished – I was relieved but didn’t have that “I’ve passed” gut-feeling so I parked it and got on with the other stuff I was doing, my results came through 7 working days after my defence but it does say to allow up to 10, unfortunately I hadn’t made it this time.

You are given some basic feedback on your performance via email, areas where they think you needed to do better on, I do however think this could be improved upon;

I understand the requirement not to give too much away about the marking schema but I would encourage the certification team to break it down into a score out of 10 for the key areas, it’s almost obvious (and logical) from the VCDX application form that marking is performed on areas like; Compute, Storage, Networking, Operational Procedures, security, BC/DR etc. I personally think it would be more constructive to give a rolled-up mark out of 10 for each of those areas and disclose an overall pass/fail ratio so you can gauge if it’s something you missed by a little or were way off and need to totally reconsider your approach – just like you get with the main VCP & VCAP certifications.

So, no VCDX 3 for me – luckily I’ve already done the required VCAP exams to allow me to submit a new vSphere4 design for a future defence on the VCDX4 track; I’m currently undecided on modifying and re-submitting or using a different project for my 2nd attempt.

There are some minor changes for the VCDX4 track, and I think Duncan has already mentioned that defence sessions won’t be co-located exclusively with events like VMworld anymore and the application form is a bit more streamlined.

So, in summary I quite enjoyed the process, shame I didn’t pass but I think I know where I went wrong and I know how to address it when I retry, my advice; don’t be too scared to give it a go, but be prepared for a lot of work!

VCE310 and VCD310 and the path to VCDX Exam Experiences


A couple of weeks ago I took and passed both my VMware Enterprise Administrator and Design exams, so I thought I would post up my thoughts.

Firstly, I was totally unprepared for the Enterprise Admin exam (VCE310) and this was reflected in my score. A bit of a scheduling SNAFU on my part meant that I found I had just 1hr5min to get to the testing centre – and I was at least 1hr away on a good day.

Because there was limited availability for the exam and I had a pretty busy work schedule I had to book a couple of months in advance and for whatever reason it went into my calendar as being on 18th June, when in fact it was actually booked for 18th May. I probably entered it on my BlackBerry so guess I fat-fingered it, memories of specific dates then disappeared into the great brain cloud so I was working to my calendar which is usually de-facto when it comes to my schedule, or not in this case! – A great argument for the PearsonVue site to send a calendar entry attachment like so many other event scheduling sites.

So, it was purely by chance that whilst I was looking up reference material at what I thought was a month ahead of the exam that I came across the booking email – and noticed that it was booked for today!

It was too late to re-arrange/book the exam (I tried) but I figured that being as I had paid for it, I may as well have a go even if I was expecting to fail – I have several years experience of tinkering and designing with VI3 although I thought my CLI skills would let me down as the exam curriculum seemed to focus heavily on that.

I also thought there would be a similar set of config minimum/maximum questions like there are for the VCP track – I historically have a bit of a problem with these as I’m not great at memorising this sort of information – and to be honest they often seem to change between updates so I just look them up as and when I have to, rather than learn them parrot fashion.

I had just 1hr5 min to dash to the test! and dash I did indeed! luckily London’s public transport was kind to me that day.

1st thing to notice was that the exams were held in a PearsonVue Professional testing centre, which are more advanced than the usual testing facilities for previous Microsoft and VMware exams – they also take signatures and photos for authentication and ID verification, 2 forms of photo ID etc.

Whilst I can’t disclose contents or details of the exam due to the understandable NDA I can say that it’s a mix of formal multi-choice questions (similar to the VCP) and a live-lab where you get VI client access to a real VI3 environment.

In the lab portion you are given a number of scenarios to configure or problems to resolve, there were some screen glitches/oddities – I assume because it’s being hosted remotely but you have access to everything in the UI and if you can’t remember the exact syntax you can find your way to it just as you would in the real-world.

You don’t get a score at the end as the lab and questions need to be scored offline – I suspect this is because they can record what you do within a session – so if you don’t complete one of the lab scenarios properly or missed a step they can review what you did, or at least the steps you took to figure out how to do something, it doesn’t say it specifically but I guess if you found the solution in the online help it would decrease your score :).

It was a rather painful 10-12 business days to get the result via email – but I did pass, not by a huge margin, but considering how unprepared I was I was pretty pleased – I think this owes a lot to the lab portion, I don’t know what ratio of marks are allocated to the lab, but I’d like to think my hands-on skills made up for any questions I didn’t answer correctly on the day due to my unpreparedness.

As soon as you get the VCE310 results you have to wait a further couple of days until you are able to register for the VCD310 design exam – you do this online with PearsonVue, rather than via a VMware certification agent like I did for the VCE310 exam, there must be a background process to update the PerasonVue profile as it took about 5 days before I could see the VCD310 exam as available for scheduling.

Whilst I was in Chicago last week I was cleared to take the VCD310 exam and I had a free morning before I got my flight back to the UK, so I booked an 8am slot, my job mainly involves architecting solutions, rather than hands-on/operations these days so I felt I was well prepared for the test.

Again, it was at a PearsonVue professional testing centre, if you are going to do these exams I would suggest getting there earlier than a normal testing centre as in my experience there are long queues to get registered and authenticated for early morning slots.

These centres do testing for all kinds of industries, for example at the US testing centre I attended for my VCD310 exam there were about 20 people in front of me to take emergency medicine / paramedic type certifications, and they had to be fingerprinted etc. so it’s not just geeks queuing up for their MCPs and CCNAs. As a side-note I feel for the non-IT test takers, they had to write full-on essays within their tests – which is why all participants in the room are given ear-plugs or noise cancelling headphones at that particular centre.

Again, I can’t disclose specific content but the VCD310 exam was split into two sections

the 1st section was a pretty straightforward multi-choice questions, with more of a design focus, capacities, HA, best solution for problem X within a particular constraint, important thing is to read the question and understand what is being asked, maybe not everything provided is relevant.

Whilst the questions required more than average brain-power to answer they were fairly straightforward and I finished within time.

The 2nd section was hard, not due to what is being asked, but because of the short amount of time allocated to read a customer requirement and produce a diagram to outline a solution that will meet the customer requirements.

It took me almost 50% of the time allotted to read the requirement and note down anything I thought was relevant but getting it drawn with the frankly rather annoying diagramming tool was very difficult and I ran out of time before I had completed the solution.

I liked that this section was very much real-world like (if only all customers could express their requirements so clearly! :)), but the tool took a bit of getting used to, the low screen-resolution also meant I spent a lot of time scrolling up and down the requirement doc and switching between that, my notes and the diagram that I was building.

With that I finished, leaving a few comments with the above remarks.

Unlike the VCE310 exam you get an instant result, and I’m pleased to say I passed this one too, but I suspect I did well on the questions but dropped a lot of marks on the 2nd part because I didn’t complete the diagram in time.

People think I’m weird, but I do quite enjoy taking these tests, and these two were no exception, a lot of it is about understanding the question and what the intended goals are, they are not cheap – from memory about £250-£300 GBP, compared to typical MS/VCP exams which are about £100 GBP.

So, that’s the exam requirement out of the way for VCDX3 so I am working on my application form with the hope of doing the defence sometime soon.

Just after I booked my VCE310 exam the information on the newer VCAP/VCDX4 track was released.

by this point I had already booked VCE310 and the availability of the new vSphere Enterprise Admin & Design exams were a way off and didn’t fit with my with my goal to complete this in Q3/4 2010, so based on this diagram from this page my plan is to do my VCDX3 certification then upgrade this to VCDX4 with the VCAP4-DCD exam, I already have my VCP4 exam so this is the quickest route for me and would result in gaining both VCDX 3 & 4 certification without having to do two defences, or wait until 2011 to get it done.


If you are planning to go down this track, it’s worthwhile bearing in mind that you need to have the VCE310, VCD310 exams completed before you can submit your application, which must then be accepted before you can be scheduled for your defence session, and the VXDX application looks to be a pretty onerous document.

Also worth noting that it seems that for the newer VCDX4 track they have removed the requirement to pass the Enterprise Admin exam before you can take the design exam as it was with the VCDX3 track I have been following – so that should help speed up the process for some people as you can do them in a different order.

8 Node ESXi Cluster running 60 Virtual Machines – all Running from a Single 500GBP Physical Server


I am currently presenting a follow-up to my previous vTARDIS session for the London VMware Users Group where I demonstrated a 2-node ESX cluster on cheap PC-grade hardware (ML-115g5).

The goal of this build is to create a system you can use for VCP and VCDX type study without spending thousands on normal production type hardware (see the slides at the end of this page for more info on why this is useful..) – Techhead and I have a series of joint postings in the pipeline about how to configure the environment and the best hardware to use.

As a bit of a tangent I have been seeing how complex an environment I can get out of a single server (which I have dubbed v.T.A.R.D.I.S: Nano Edition) using virtualized ESXi hosts, the goals were;

  • Distributed vSwitch and/or Cisco NX100V
  • Cluster with HA/DRS enabled
  • Large number of virtual machines
  • Single cheap server solution
  • No External hardware networking (all internal v/dvSwitch traffic)

The main stumbling block I ran into with the previous build was the performance of the SATA hard disks I was using, SCSI was out of my budget and SATA soon gets bogged down with concurrent requests which makes it slow; so I started to investigate solid state storage (previous posts here).

By keeping the virtual machine configurations light and using thin-provisioning I hoped to squeeze a lot of virtual machines into a single disk, previous findings seem to prove that cheap-er consumer grade SSD’s can support massive amount of IOps when compared to SATA (Eric Sloof has a similar post on this here)

So, I voted with my credit card and purchased one of these from Amazon – it wasn’t “cheap” at c.£200 but it will let me scale my environment bigger than I could previously manage which means less power, cost, CO2 and all the other usual arguments you try to convince yourself that a gadget is REQUIRED.

So the configuration I ended up with is as follows;

1 x HP ML115G5, 8Gb RAM, 144Gb SATA HDD c.£300 (see here) but with more RAM
1 x 128Gb Kingston 2.5” SSDNow V-Series SSD c£205

I installed ESX4U1 classic on the physical hardware then installed 8 x ESXi 4U1 instances as virtual machines inside that ESX installation


This diagram shows the physical server’s network configuration


In order for virtualized ESXi instances to talk to each other you need to update the security setting on the physical host’s vSwitch only as shown below;


This diagram shows the virtual network configuration within each virtualized ESXi VM with vSwitch and dvSwitch config side-side.


I then built a Windows 2008R2 Virtual Machine with vCenter 4 Update 1 as a virtual machine and added all the hosts to it to manage

I clustered all the virtual ESXi instances into a single DRS/HA cluster (turning off admission control as we will be heavily oversubscribing the resources of the cluster and this is just a lab/PoC setup


Cluster Summary – 8 x virtualized ESXi instances – note the heavy RAM oversubscription, this server only has 8Gb of physical RAM – the cluster thinks it has nearly 64Gb



I then built an OpenFiler Virtual Machine and hooked it up to the internal vSwitch so that the virtualized ESXi VMs can access it via iSCSI, it has a virtual disk installed on the SSD presenting a 30Gb VMFS volume over iSCSI to the virtual cluster nodes (and all the iSCSI traffic is essentially in-memory as there is no physical networking for it to traverse.


Each virtualized ESXi node then runs a number of nested virtual machines (VM’s running inside VMs)

In order to get Nested virtual machines to work; you need to enable this setting on each virtualized ESXi host (the nested VM’s themselves don’t need any special configuration)


Once this was done and all my ESXi nodes were running and settled down, I have a script to build out a whole bunch of nested virtual machines to execute on my 8-node cluster. the VM’s aren’t anything special – each has 512Mb allocated to it and won’t actually boot past the BIOS because my goal here is just to simulate a large number of virtual machines and their configuration within vCenter, rather than meet an actual workload – remember this is a single server configuration and you can’t override the laws of physics, there is only really 8Gb or RAM and 4 CPU cores available.

Each of the virtual machines was connected to a dvSwitch for VM traffic – which you can see here in action (the dvUplink is actually a virtual NIC on the ESXi host).



I power up the virtual machines in batches of 10 to avoid swamping the host, but the SSD is holding up very well against the I/O

With all 60 of the nested VMs and virtualized ESXi instances loaded these are the load stats










I left it to idle overnight and these are the performance charts for the physical host; the big spike @15:00 was the scripts running to deploy the 60 virtual machines


Disk Latency


Physical memory consumption – still a way to go to get it to 8Gb – who says oversubscription has no use? 🙂

image image

So, in conclusion – this shows that you can host a large number of virtual machines for a lab setup, this obviously isn’t of much use in a production environment because as soon as those 60VM’s actually start doing something they will consume real memory and CPU and you will run out of raw resources.

The key to making this usable is the solid state disk – in my previous experiments I found SATA disks just got soaked under load and caused things like access to the VMFS to fail (see this post for more details)

Whilst not a production solution, this sort of setup is ideal for VCP/VCDX study as it allows you to play with all the enterprise level features like dvSwitch and DRS/HA that really need more than just a couple of hosts and VMs to understand how they really work. for example; you can power-off one of the virtual ESXi nodes to simulate a host failure and invoke the HA response, similarly you can disconnect the virtual NIC from the ESXi VM to simulate the host isolation response.

Whilst this post has focused on non-production/lab scenarios it could be used to test VMware patch releases for production services if you are short on hardware and you can quite happily run Update manager in this solution.

If you run this lab at home it’s also very power-efficient and quiet, there are no external cables or switches other than a cross-over cable to a laptop to run the VI Client and administer it; you could comfortably have it in your house without it bothering anyone – and with an SSD there is no hard disk noise under load either 🙂

Thin-provisioning also makes good use of an SSD in this situation as this screenshot from a 30Gb virtual VMFS volume shows.


The only thing  you won’t be able to play around with seriously in this environment is the new VMware FT feature – it is possible to enable it using the information in this post and learn how to enable/disable but it won’t remain stable and the secondary VM will loose sync with the primary after a while as it doesn’t seem to work very well as a nested VM. If you need to use FT for now you’ll need at least 2 physical FT servers (as shown in the original vTARDIS demo)

If you are wondering how noisy it it at power-up/down TechHead has this video on YouTube showing the scary sounding start-up noise but how quiet it gets once the fan control kicks-in.

ML115 G5 Start-up Noise

Having completed my VCP4 and 3 I’m on the path to my VCDX and next up is the enterprise exam so this lab is going to be key to my study when the vSphere exams are released.