Building a Better Test Lab
October 22, 2007This is the outline of a number of posts on building a {relatively} low-cost accurate test lab of your production systems using P2V, VMWare, ESX, custom scripted HP voodoo, HP MSA1500 SAN, Virtual Switch Tagging (VST), Checkpoint on Sun Firewalls and Cisco switches. in order to clone a complicated multi-tier Windows based production platform with lots of DMZ segments into a VMWare farm for use as a test/dev & development environment (and possibly a DR one too in future)
This is all based on some of my recent work with customers* and I hope will help someone else to navigate the pitfalls (both business and technological) I & my team encountered in delivering this idea.
The following is a list of titles or sections and will hopefully serve as an index, but please, don’t expect them all at once I do have a day job to do! 
Why do this?
The traditional approach to test labs & why I think it’s wrong
Pro’s
Con’s
Isn’t this all a bit too complicated/mad-scientist/far out?
Reload lab from production process - how often?
is change control important?
What do you want a test lab to do?
Scoping/Expectation Setting
Load Testing - is VMWare right for this
Dynamic/Grid based approach to load testing
Break/Fix analysis
Release Testing
Options for disaster recovery/production failover
What won’t it do?
Storage Design
“Big” SAN’s are always better if you have them, but what if you don’t?
HP MSA 1500 - it’s not big, but it’s clever
Disk/SAN bandwidth - my practical experiences
Server Design
ESX Node specification
The RAM per VM debate
Networking Design
VLAN tagging
VST vs. Guest Tagging etc.
Firewalls
Clone to test lab Process
P2V Tools - VMWare Convertor vs. the rest
Changing IP addresses
HP uninstall Scripts
Build-Out Steps
Build ESX environment
Scripted VMWare installations - automatically create custom Virtual NIC’s/LANs
Adjust install paths for SAN storage
Set administrator password/create accounts
Install Networking
Configure VLAN’ing
IP Load Balancing
Install Firewall(s)
Test Communications between virtual DMZ segments and across hosts
Import Production machines
VMWare Convertor
General issues found
P2V Windows 2003 Domain Controllers - Special Notes
P2V’ing entire Windows Cluster’s - not that easy but do-able
P2V Process over a WAN - issues found & workaround.
Fresh VM 1st boot, changing IP address etc.
HP tools removal
Some further problems caused by changing IP addressing.
Into the Future
Can you use this for disaster recovery?
VMWare Lab Manager
Total Automation - Platespin products?
*This article has been deliberately made anonymous & I’m afraid I can’t disclose the name of the customer or provide any further reference materials without a commercial engagement via my employer, you can contact me for more details on this via this blog.
This article & information contained within is provided entirely without warranty.
Technorati Tags: VMWare , ESX , HP , MSA1500 , Test Lab , P2V , production , clone , Sun , Checkpoint firewall , VLAN tagging , VST
[...] like this idea of this. it sounds like a product for automating a lot of what I blogged about in my Build a better test lab series (much delayed in publishing… must find some time to post [...]
What if we are using MSVS? I’m assuming VMware Lab Manager will not support that for test lab. Who would be the next vendor to provide MSVS support?
As far as I know Lab Manager is VMWare only.
That said Microsoft have a new product that will be capable of managing VMWare and MS Virtual server + others - you should be able to use this to “manage” the VMs - but not sure how deep this management goes or how it lines up to VMWare’s offering - most of these MS products spend so much time trying to do get coverage of the market/tech sometimes they don’t go deep enough in terms of functionality, but I have not had a chance to try it out yet.
The principals within this article are mostly the same regardless of the underlying virtualization product - as long as it supports P2V, VLAN tagging - if required.
The MS product is called MSVMM
some links here http://blogs.technet.com/virtualization/archive/2007/09/06/system-center-virtual-machine-manager-huge-announcements.aspx
and http://www.microsoft.com/systemcenter/scvmm/default.mspx
If the actual Hypervisor becomes reasonably irrelevant http://blog.scottlowe.org/?p=436 I would expect to see VMWare’s management products being capable of managing other Hypervisors - they’ll have to do so to stay alive, having the best technology only carries you so far in the market..
[...] spent a lot of time at the start of 2007 building this type of system from scratch (see the build a better test lab posts). hopefully this will go a long way to making it easier to [...]
Hello,
I’m on the process to Build a Virtual Environment with ESX between Primary Site and DR Site.
I’m worried about the IP Address of the Virtual Machines when the moved to the DR Site, how the Client will be able to reach to those VMs in Production which are located in the Primary Site?
The Primary Site is in different subnet other than the DR Site.
Branches are connected to the Primary Site.
At the DR Site, we have the Same ESX Servers Setup with SAN Storage and the same Netwoking and VLAN will be used. “It’s like a Replica of Setup”.
How the Branches will be able to reach the VMs when it’s booted from the DR Site?
Can someone elaborate for me how this can work?
Hi Catlin,
You should take a look at VMLogix LabManager (http://www.vmlogix.com). It is a virtual lab automation solution that is hypervisor agnostic and supports leading hypervisors from Citrix, Microsoft and VMware. We do support Microsoft Virtual Server, in response to your question.
You can request for a free 30 day evaluation on our website.
- Srihari Palangala, VMLogix
http://blog.vmlogix.com