As noted here and here, VMWare have had ESX 3.5u2 certified under Microsoft’s SVVP programme, this is excellent news and will knock down one of the long standing barriers to greater adoption of virtualisation as I wrote about here - support.
Most notably for me this means blessed support of Exchange 2007sp1 running under ESX!
Excellent work to get this done so quickly - MS only announced the SVVP programme a short while ago.
Official list of MS products supported under VMWare is here.
Just incase you ever wondered what it looks like here is a screendump..
this is the VMWare equivalent of Microsoft’s BSOD (Blue Screen of Death)
I got this whilst running ESX 3.5 under VMWare Workstation 6.5 build 99530, it happened because I was trying to boot my ESX installation from a SCSI hard disk - which it didn’t like - I assume because of driver support, swapped for an IDE one and it worked fine…
update - actually the VM had 384Mb of RAM allocated and that’s what actually stopped it from booting.. upped to 1024Mb and it runs fine.
Its the first time I’ve seen one - all the production ESX boxes I’ve worked with have always been rock-solid (touch wood)
I’m preparing a blog post about unattended installations of ESX when I hit this, in case you were wondering.
I’ve noticed this a couple of times, if you P2V a VM from VM Workstation to ESX using VM Converter - it brings across a virtual USB device which isn’t supported by ESX.. if you look at the properties you get the following amusing entries..
Now, I wish I really did have a funky USB dongle, or doohickey… sounds useful!
As noted here there is a doc that has been jointly produced between VMWare and Cisco which has all the details required for integrating VI virtual switches with physical switching.
Especially handy if you need to work with networking teams to make sure things are configured correctly to allow failover properly between redundant switches/fabrics etc. - it’s not as simple as it looks, and people often forget the switch-side configurations that are required.
Doc available here (c.3Mb PDF)
Crazy, yeah - but hey you’ve got to try it, prompted by a question from Prasad - can you run ESX in a VM under ESX?
In the interest of science I just tried this, I used VM Convertor to convert my working ESX under workstation image as-is to my ESX box (hoping it would keep the custom settings intact, and saving me from having to rebuild it)
good news, the VM converter did it’s thing and it does start up on the ESX box.
..bad news, it doesn’t get past this screen as far as I can tell…it’s sat there for a good 20mins so I don’t think its going to get much further.
Also tried to import my ESX 3i image to see if that would work, but VM Convertor wouldn’t import it for some reason, so will have to try a clean install on that.
Looks like some kind of error when it’s trying to determine what version it is..
[2008-06-13 00:23:29.164 'P2V' 5748 error] [task,295] Task failed: P2VError UNKNOWN_METHOD_FAULT(sysimage.fault.OsVersionNotFound)
[2008-06-13 00:23:29.164 'P2V' 5748 verbose] [task,339] Transition from InProgress to Failure requested
[2008-06-13 00:23:29.164 'P2V' 5748 verbose] [task,388] Transition succeeded
Ah well, anyone know how to get this going/if it’s possible?
Note to remember, don’t forget to check the duplex settings on NICs handling your vMotion traffic.
My updated clustered ESX test lab is progressing (more posts on that in the next week or so)… and I’m kind of limited in that I only have an old 24-port 100Mb Cisco hub for the networking at the moment.
vMotion warns about the switch speed as a possible issue.
I had my Service Console/ vMotion NICit forced to 100/full and when I 1st tried it vMotion took 2hrs to get to 10%, I changed it to auto-negotiate whilst the task was running and it completed without breaking the vMotion task ain a couple of seconds, dropped only 1 ping to the VM I moved.
Cool, it’s not production or doing a lot of workload but useful to know despite the warning it will work even if you’ve only got an old hub for your networking, and worth remembering that Duplex mis-matches can literally add hours and days onto network transfers.
Following on from my earlier post I upgraded my installation to the new build of 6.5. it un-installed the old build and re-installed the latest without a problem, took about 30mins and required a reboot of the host OS.
All my previously suspended XP/2003 VM’s resumed ok without a restart but needed an upgrade to the VMTools which did require a restart of the guest OS - all completed with no problems.
Now, onto installing ESX….
I used the settings from Eric’s post here to edit my .vmx file
ethernet0.virtualDev = “e1000″
monitor.virtual_exec = “hardware”
monitor_control.restrict_backdoor = “true”
Note - you need to select an x64 Linux version from the VM type drop down, if you have to go back and change it via the GUI after you’ve edited the .vmx file it overwrites the Ethernet card “e1000″ setting to “vlance” so you need to edit again otherwise the ESX installer won’t find a compatible NIC and won’t install.
it was initially very slow to boot; 5mins on my dual core laptop with only one error - which was expected..
To improve the performance I changed my installation to run the non-debug version of the Workstation binaries (rename the vmware-vmx.exe to vmware-vmx-debug.exe)
note: this isn’t recommended unless you know what you are doing, VMWare will rely on the output from the debug version of the code if you need to report any issues)
It also seems to work for the installable version of ESX 3i… (although I’ve not quite figured out the point of that version yet :)).
Install prompt
it did fail with an error the 1st time round..
this was because I had specified an IDE disk as per the ESX instructions, I changed it to a SCSI one and it worked ok.
Finished..
The ESX 3i install has a footprint of about 200Mb on disk, and ESX 3.5 uses 1.5Gb.
I’m going to keep the 3.5 install on my laptop and will try to use linked clones to maintain a couple of different versions/configs to save disk space.. I’m sure I could knock up a quick script to change the hostname/IP of each clone - if I do I’ll post it here.
Why would you want to do this? well because you can, of course 
and its handy for testing patch updates and scripts for ESX management etc.
I will also try to get a ESX DRS cluster running under workstation with a couple of ESX hosts and shared storage over iSCSI using something like OpenFiler as shown here. won’t exactly be production performance, but useful for testing and demo’ing.
If you deploy your VM’s from a master image using Virtual Center’s Deploy from template functionality (below).
When you try and delete a virtual machine you’ve created from disk
You get the following prompt
Are you sure you want to delete this VM and it’s associated base disk?
Please note if other VMs are sharing this base disk, they will no longer have access to this disk.
This does not refer to the master VM image you deployed from; in other words if you delete the VM it does not break all other VMs deployed from the initial template.
One other point to note, when you perform “Deploy virtual machine from template” operation, the target field (below) is actually the name of the base image you are cloning, rather than the name of the eventual VM you are creating from it - odd, but that’s how it is (below)
You’ve been able to buy solid state SAN technology like the Tera-RAMSAN from TMS which gives you up to 1Tb of storage, presented over 4Gb/s fibre channel or Infiniband @10Gb/s… with the cost of flash storage dropping its going to soon fall in to the realms of affordability (from memory a year ago 1Tb SSD SAN was about £250k, so would assume that’s maybe £150k now - would be happy to see current pricing if anyone has it though).
If you were able to combine this with a set of ESX hosts dual-connected to the RAMSAN and traditional equipment (like an HP EVA or EMC Clariion) over a FC or iSCSI fabric then you could possibly leverage the new Storage vMotion features that are included in ESX 3.5 to achieve a 2nd level of performance and load levelling for a VM farm.
It’s pretty common knowledge that you can use vMotion and the DRS features to effectively load level or average VM CPU and memory load across a number of VMWare nodes within a cluster.
Using the infrastructure discussed above could add a second tier of load balancing without downtime to a DRS cluster. If a VM needs more disk throughput or is suffering from latency then you could move them to/from the more expensive solid-state storage tiers to FC-SCSI or even FATA disks, this ensures you are making the best use of fast, expensive storage vs. cheap, slow commodity storage.
Even if Virtual Center doesn’t have a native API for exposing this type of functionality or criteria for the DRS configuration you could leverage the plug-in or scripting architecture to use a manager of managers (or here) to map this across an enterprise and across multiple hypervisors (Sun, Xen, Hyper V)
I also see EMC integrating flash storage into the array itself, would be even better if you could transparently migrate LUNS to/from different arrays and disk storage without having to touch ESX at all.
Note: This is just a theory I’ve not actually tried this - but am hoping to get some eval kit and do a proof on concept…
I’ve been really busy the last couple of weeks and I’ve had to trim down my incoming RSS feeds, as there was too much noise and I was missing important things like the following;
Linkage here.
VMWare are shaping up to have a really good set of management tools - lab and site recovery manager are of particular interest to me for several projects I’m working on.
The following screen dump is from an HP DL380G5 server that runs all the core infrastructure under VMWare Server (the free one) for a friend’s company which I admin sometimes.
It is housed in some co-lo space and runs the average range of Windows servers used by a small but global business, Exchange SQL, Windows 2003 Terminal Services.
As a result of some planned (but not very well communicated!) power maintenance the whole building lost power earlier today, when it was restored I grabbed the following screenshot as the 15 or so Virtual Machines automatically booted.
interesting to note that all the VM’s had been configured to auto-start with the guest OS, meaning there wasn’t any manual intervention required, even though it was a totally dirty shutdown for both the host and guest OS’es (No UPS, as the building and suite is supposed to have redundant power feeds to each rack - in this instance the planned maintenance was on the building wiring so required taking down all power feeds for a 5 yearly inspection..)
There are no startup delay settings in the free version of VMWare Server so they all start at the same time, interesting to note the following points..
The blue line that makes a rapid drop is the pages/second counter, and the 2nd big drop (green) is the disk queue length. the hilighted (white) line is the overall %CPU time, note the sample frequency was 15 seconds on this perfmon.
After it had settled down, I took the following screenshot, it hardly breaks a sweat during its working day. there are usually 10-15 concurrent users on this system from around the world (access provisioned via an SSL VPN device) and a pretty heavily used Exchange mail system.
The box is an HP DL380 G5 with 2 x quad core CPUs (8 cores in total) and 16Gb of RAM, it has 8 x 146Gb 15k HDDs in a single RAID 5 set + hot-spare, it was purchased in early 2007 and cost c.£8,000 (UK Prices)
It runs Windows 2003 Enterprise Edition x64 edition with VMWare Server 1.0.2 (yes, its an old build.. but if it ain’t broke..) and they have purchased multiple w2k3 ent-edition licences to take advantage of the virtualisation use-rights to cover the installed virtual OS’es.
It’s been in-place for a year and hardly ever has to be touched, its rock-solidly available and the company have noticed several marked improvements since they P2V’d their old servers onto this platform, as follows;
Hopefully this goes to show the free version of VMWare’s server products can work almost as well if budget is a big concern, ESX would definitely give some better features and make backup easier, they are considering upgrading and combining with something like Veeam Backup to handle failover/backup.
The followign screens show a working configuration from the RDP 3.80 PXE Configuration Manager
Have had lots of problems with this deploying Windows OS’es and VMWare ESX 3.5 onto an HP c7000 Blade chassis, still not resolved all the problems, but this definitely works for deploying Windows!
The documentation reads like you should always use the Linux PE configuration and it handles switching between WinPE/LinuxPE depending on which OS job you drop on a target. in my experience this doesn’t work and you need to manually change the PXE configuration to default to LinuxPE or WinPE depending on the OS you want to target.
And
Still a work in progress as I have a c7000 to which I want to deploy a mix of Windows and ESX/Redhat OS’es….
I did get a previous installation to install ESX 3.5 by hacking the default ESX 3.02 job, but its since been re-installed and I can’t do it now
RDP 6.90 seems to list Windows 2008 and ESX 3.5 in the quickspecs, but I’ll be damned if I can find where to download it, going to have to call HP methinks!
As I’ve posted before installing via iLo is just a non-starter if you really do want a flexible and fast deployment configuration - so it has to be RDP.
More later…
As the Hoff posts here and on VMTN here. the proposed vulnerability that you can manipulate and possibly compromise a VM during a VMotion process isn’t exactly major, it’s clever.. but - like anything if you don’t follow the best-practice recommendations then you expose yourself to these risks… same reason they recommend you lock your server room or don’t have blank passwords - this attack is akin to gaining physical access to the hardware or being able to sniff a physical switch port - in this instance, it’s “virtual” hardware.
VMWare have always recommended keeping the VMotion traffic on a separate VLAN or network.
the other vulnerability where VMTools can be compromised on is different, but again preventable.. and not enabled on server instances of VMWare.
Interesting article here on some stress testing VMWare have done running Exchange 2007 under virtualization on VI3.5.
It’s working.. .and working well, now - official support?
Martin’s post here prompted me to blog something I’ve been meaning to do for a while.
Virtualization projects and services are cool; we all understand the advantages in power/cooling and the flexibility it can bring to our infrastructures.
But what about support, if you are a service provider (internal or outsourcing) you normally need to be able to offer an end-end SLA on your services. typically this would be backed off against a vendor like Microsoft or Oracle via one of their premium support arrangements.
From what I see in the industry, with most software vendors especially Microsoft there is almost no way a service provider can underwrite an SLA as application/OS vendors give themselves significant scope to say “unsupported configuration” if you are running it under a hypervisor or other VM technology… Microsoft use the term commercially reasonable in their official policy - who decides what this is?
I would totally accept that a vendor would not guarantee performance under a hypervisor - that’s understandable and we have tools to analyse, monitor and improve (Virtual Centre, MOM, DRS, increase resources etc.). but too many vendors seem to use it as a universal “get out of jail free card”.
Issues of applications with dependency on physical hardware aside (fax cards, realtime CPU, DSP, PCI cards etc.) In my entire career working with VM technology I’ve only ever seen one issue that could be directly attributed to being caused by virtualization - and to be fair that was really a VMTools issue; rather than VMWare itself.
Microsoft have an official list of their applications that are not supported here - why is this? speech server I could maybe understand as it would probably be timer/DSP sensitive - but the rest? Sharepoint? I know for a fact ISA does work under VMWare as I use it all the time.
Microsoft Virtual Server support policy http://support.microsoft.com/kb/897613
Support policy for Microsoft software running in non-Microsoft hardware virtualization software http://support.microsoft.com/kb/897615/
Exchange is specifically excluded (depending on how you read the articles)
· On the Exchange Server 2007 System requirements page it only mentioned Unified messaging as being unsupportable in a virtual environment http://technet.microsoft.com/en-us/library/aa996719.aspx
· Yet on TechNet it is clear stated that “Neither Exchange 2007 nor Exchange 2007 SP1 is supported in production in a virtual environment” http://technet.microsoft.com/en-us/library/bb232170(EXCHG.80).aspx
Credit due to a colleague for pulling together the relevant Microsoft linkage
But I know it….
a) works fully - I do it all the time.
b) Lots of people are doing this in production with lots of users (many people at VMWorld US last year)
c) VMWare have a fully-supportable x64 hypervisor - It’s just MS that don’t
What is the industry going to do about this?, I asked this question of peers a lot at VMWorld and at BriForum; and to be honest everyone has the same concern but have a few different approaches;
Dont’ tell/ask - 99% of the time a tech support rep won’t know its running under VMWare/a.n.other hypervisor so why complicate matters by telling them - could of course back-fire on you!
Threaten - “If you won’t support under VMWare we’ll use one of your competitors applications”; however this only really works if you are the US govt. or Globocorp Inc. or operate in a very niche application market.
Mitigate - reflect this uncertainty in an SLA, best-endeavours etc. this would kill most virtualization efforts in their tracks for an enterprise customer.
The same support issue has been around for a long time; Citrix/Terminal Services, application packaging, automated installations, etc. are treated as “get out of jail free cards” by support organisations…
But whilst there are some technical constraints (usually only affecting badly written apps) with terminal services and packaging, virtualization changes the game and should make it simpler for a vendor to support as there is no complex runtime integration with a host OS + bolt-ons/hacks it’s just an emulated CPU/disk/RAM you can do whatever you like within it.
So - the open debate; what do you do? and how do you manage it?
Please comment…
There is a new site here (disclaimer: it does seem to be promoting a commercial service, but has some useful information that has been put into the public domain); describing some methods to roll your own P2V backup approach; I’ve not read in detail yet; but looks like Frane Borozan has solved some of the challenges I’ve encountered in the past automating the Free VMWare Convertor tool.
When I get some time I will revisit my build a better test lab series (and update it!) I hope to be able to integrate some of Frane’s ideas.
Thanks to Techhead for passing on the link; we worked together on the platform underlying the Build a better test lab series and he did a lot of work on the P2V and post-P2V automation tasks - he’s got a lot of handy scripts for doing this on an HP platform
Virtualized DR is going to be big this year; I have a long line of customers with this high on their list of priorities… Both for cross site 100% VMWare implementations and for the ability to backup/restore physical platforms to VMWare grid in a DR situation.
It just makes so much sense; no delay whilst racking & stacking recovery kit or problems restoring to different hardware etc. your admin’s can even do it from home - which can have some significant advantages in the event of a natural disaster like Katrina or floods like we had over the last couple of years in the UK
PlateSpin Forge is something we are seriously looking at as well as Symantec Backup Exec System Recovery Server Edition (who win a prize for extending the longest, most annoying product name! despite acquiring it from Veritas).
Will be an interesting year; I’m sure Sungard and all those recovery centre facilities will be moving to a grid/resource rental model rather than pure rack/floor space and retained hardware on-contract.
There’s an interesting post over on Forrester research blog by James Staten. he’s talking some more about data centres in a container; making the data centre the FRU rather than a server or server components (Disk, PSU etc.).
This isn’t a new idea but it I’m sure the economics of scale currently mean this is currently suitable for the computing super-powers (Google, Microsoft - MS are buying them now!) - variances in local power/comms cost could soon force companies to adopt this approach rather than be tied to a local/national utility company and their power/comms pricing.
But just think if you are a large out-sourcing type company you typically reserve, build and populate data centres based on customer load, now this load can be variable; customers come and go (as much as you would like to keep them long-term this is becoming a commodity market and customer’s demand you are able to react quickly to changes in THEIR business model - which is typically why they outsource - they make it YOUR problem to service their needs).
It would make sense if you could dynamically grow and shrink your compute/hosting facility based on customer demand in this space - thats not so easy to do with a physical location as you are tied to it in terms of power availability/cost and lease period.
New suite build out at a typical co-lo company can take 1-2 months to establish networking, racks, power distribution, cabling, operational procedures etc. (and that’s not including physical construction if it’s a new building) - adopting the blackbox approach could significantly reduce the start-up time and increase your operational flexibility
Rather than invest in in-suite structured cabling, rack and reusable (or dedicated) server/blade infrastructures why not just have terminated power, comms and cooling connections and plug them in as required within a secured warehouse like space.
Photos from Sun Project Blackbox
You could even lease datacentre containers from a service provider/supplier to ensure there is no cap-ex investment required to host customers.
If your shiny new data centre is runs out of power then you could relocate it a lot easier (and cheaply) as it’s already transportable rather than tied to the physical building infrastructure; you are able to follow the cheapest power and comms - nationally or even globally.
As I’ve said before the more you virtualize the contents of your datacentre the less you care about what physical kit it runs on… you essentially reserve power from a flexible compute/storage/network “grid” - and that could be anything/anywhere.
I’ve not done anything with my home ESX server this week as I’ve been busy with work; so this will be interesting - it’s been powered up all the time with all the VM’s spinning; but not doing very much.
CPU Utilisation
Memory Utilisation
Disk Utilisation
Whist running this set of VMs.. (the CPU stats for VMEX01 and VMEX02 are a bit skewed as I added this bit after the original post and they are both running seti@home (hence increased CPU)
So, nothing interesting to see here - but might be worth bearing in mind for some kind of sizing estimate; this is a single core CPU (HT enabled) PC with 4Gb RAM and a single 500Gb SATA disk
Hopefully I will get some time this week to load up SETI@Home or Folding@Home and see what that does it should be a good test to see how well the hypervisor manages CPU timesharing between hosts.
You are currently browsing the archives for the ESX category.
My TechEd EMEA 2008 Twitter Feed The Latest from VMWare..
