Virtualization, Cloud, Infrastructure and all that stuff in-between

My ramblings on the stuff that holds it all together

How VMware IT use VMware Internally


This was a very interesting session, it wasn’t on my printed programme so I assume it was re-organised from somewhere else but as I was in the area I went to it rather than my planned session. The presentation was given by VMware’s CIO and was set to cover how VMware use virtualization internally, to deliver normal business services to it’s internal users.

after some scene setting of what VMware technology can do for consolidation and workload management (would think most people attending VMworld on the 2nd day would know this already.. but) he moved into describing what VMware use internally and how they have been through the same evolution as their customers, their stated goal is to move to cloud services to make their own operations more efficient and flexible (well they would, wouldn’t they :)).

VMware have expanded very rapidly over the last 10 years taking on lots of staff and opening up global offices, data centres and labs; he confessed that often solutions had been put in place in haste and this had led to a growing pain in management and stability.

VMWare run a large ERP system based around Oracle ERP and RAC to run their core business systems as well as a Microsoft Exchange based messaging.

VMware are starting to make heavy use of the VDI scenario with 550 users at the moment, they don’t silo VDI to particular job roles and users are a mix of engineers, sales and administrative staff.

The standard client is a Wyse thin-client with a 24” LCD monitor and this is the CEO’s primary machine 🙂 on the back end the standard hardware configuration is he HP c7000 blade chassis with Cisco 3020 blade switches uplinked into Cisco 3750 L3 switches, storage is provided by an EMC CX3-80.

VMware say this VDI configuration saves them c.$900USD over a typical notebook setup per user.

he confessed that VMware haven’t virtualized 100% of their internal IT (yet) 2 application services still remain on physical servers;

  • VMware Capacity Planner – which is due to be virtualized in Q1’09
  • Oracle RAC – which is due to be virtualized in Q2’09

I thought it was quite ironic that capacity planner still lives on a physical box but is responsible for the demise of so many physical servers on customer sites 🙂

There were some interesting diagrams on the Blade layout for the Oracle RAC and Exchange systems; which I will try to download and post if allowed, but in the meantime it runs on 2 x HP c7000 blade chassis with 4Gb RAM allocated per CPU core.

VMware also had a physical Exchange 2003 server until last year; as part of a migration to Exchange 2007 they implemented 14 virtualized Exchange 2007 servers; 11 mailbox nodes – mostly in a CCR configuration and the remainder in HT and CAS roles; split across 4 ESX blades (pretty sure he said 4).

Typical mailbox sizes are under 2Gb but it was refreshing to hear that like anywhere else they had some challenging mailbox sizes, particularly the execs 🙂

One government customer has over 750k mailboxes running under virtualized instances of Exchange, which is good to hear.

As part of a general consolidation programme VMware introduced some standardised hardware configurations; HP c7000 series blades and a mix of EMC CX3-80 and EMC DMX4 SAN’s (the latter being for the more demanding enterprise applications like Oracle ERP) – I would expect VMware IT get the luxury of a very healthy staff discount from EMC when designing such solutions:)

They consolidated down from 6 main data centres to 2, a tier 4 primary and a tier 2 DR site; they are implementing SRM for DC failover and relocation.

They have just opened a new 1500 rack “green” data centre in WA (wherever that is..) to host their vast R&D facility which as they need to test builds against lots of vendors kit as well as two internal cloud facilities.

The new DC takes full advantage of hot/cold aisle and passive air cooling and recycled building materials. in-fact due to the climate and cooling tech they only need to run the chillers for 3 months out of the year which vastly reduces cooling and thus power costs. In addition, power comes from a pair of redundant (yes, really!) hydro-electric power stations, I believe he said they were paying c.$0.02 KW/Hr for this “green” energy and are working on certifying it for LEED Platinum – which I assume is some EPA type programme.

In terms of supporting this environment they have achieved a level of 145 virtual machines per system administrator; which is pretty high; in general terms that have realised an overall 10:1 server consolidation ratio and have (honestly) experienced only 1 server crash in the last year which was attributed to VMware ESX.

Nothing like eating your own dogfood I guess, interesting to hear that VMware have been through the same challenges as most other businesses in terms of growth and consolidation – it certainly adds some credibility to their message over and above what they have done with customers. it would have been interesting to have some coverage of how their development and lab and R&D systems work but I guess that could be considered more sensitive in such a competitive market.

13 responses to “How VMware IT use VMware Internally

  1. Josue February 25, 2009 at 9:26 pm

    Very interesting. Thank you for sharing. As advice to a writer: Your article was rather full of grammatical errors and mis-spellings. You may want to have a literate friend proof-read/edit your posts. Writing style is unfortunately less forgiving that speaking style.

    • vinf February 25, 2009 at 10:37 pm

      Thanks for reading, all feedback is good feedback 🙂

      it was a rather rushed post due to other commitments, I shall of course defer to my significantly more literate online friends in future – 🙂

  2. Scott February 25, 2009 at 10:36 pm

    10:1 server consolidatin ratio doesn’t seem very high to me. Did they give any details of the blades they were using ie processors, ram?

    • vinf February 25, 2009 at 10:43 pm

      I think this was on the slides, I’m waiting for a copy; in terms of the ratio I think that was averaged across the business. Oracle instances had a lot of server containment going on for example and he did say they recycled a fair bit of hardware and there were no dates attached to that 10:1 ratio, I think he said 4-way dual-core blades (so 8 cores/blade).

      There was a bullet point about when sometimes it’s more power efficient (and thus cost-efficient) to purchase new, higher density CPU blades.

  3. Dean February 25, 2009 at 11:27 pm

    WA = Washington state 🙂

  4. Pingback: Cisco - Welcome to the Human Network » How VMware IT use VMware Internally

  5. Pingback: VMWorld Europe Day 2: Wrap-up, a good day despite the curious lack of forks.. « Virtualization, Windows, Infrastructure and all that “stuff” in-between

  6. Datto February 26, 2009 at 9:06 pm

    Site plan and other basic information about VMware’s new datacenter in Wenatchee, WA:


  7. Pingback: VMworld 2009 Europe Linkage » Yellow Bricks

  8. Pingback: VMware Europe 2009 linkage « DeinosCloud’s Blog

  9. Pete June 19, 2009 at 2:39 am

    That’s a wonderful look inside of the VMware machine! One glaring question I would have though is:

    How do they back it all up?

    I’ve found many great products to backup just VMware VMs, and I’ve also found others that do a great job of picking up applications, like Exchange. I haven’t really found one that does both.

    I’d love to hear what they use internally.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: