Virtualization, Cloud, Infrastructure and all that stuff in-between

My ramblings on the stuff that holds it all together

Category Archives: HP

VMWare Server Performance – A Practical Example

 

The following screen dump is from an HP DL380G5 server that runs all the core infrastructure under VMWare Server (the free one) for a friend’s company which I admin sometimes.

It is housed in some co-lo space and runs the average range of Windows servers used by a small but global business, Exchange SQL, Windows 2003 Terminal Services.

As a result of some planned (but not very well communicated!) power maintenance the whole building lost power earlier today, when it was restored I grabbed the following screenshot as the 15 or so Virtual Machines automatically booted.

interesting to note that all the VM’s had been configured to auto-start with the guest OS, meaning there wasn’t any manual intervention required, even though it was a totally dirty shutdown for both the host and guest OS’es (No UPS, as the building and suite is supposed to have redundant power feeds to each rack – in this instance the planned maintenance was on the building wiring so required taking down all power feeds for a 5 yearly inspection..)

There are no startup delay settings  in the free version of VMWare Server so they all start at the same time, interesting to note the following points..

The blue line that makes a rapid drop is the pages/second counter, and the 2nd big drop (green) is the disk queue length. the hilighted (white) line is the overall %CPU time, note the sample frequency was 15 seconds on this perfmon.

image 

After it had settled down, I took the following screenshot, it hardly breaks a sweat during its working day. there are usually 10-15 concurrent users on this system from around the world (access provisioned via an SSL VPN device) and a pretty heavily used Exchange mail system.

image

The box is an HP DL380 G5 with 2 x quad core CPUs (8 cores in total) and 16Gb of RAM, it has 8 x 146Gb 15k HDDs in a single RAID 5 set + hot-spare, it was purchased in early 2007 and cost c.£8,000 (UK Prices)

It runs Windows 2003 Enterprise Edition x64 edition with VMWare Server 1.0.2 (yes, its an old build.. but if it ain’t broke..) and they have purchased multiple w2k3 ent-edition licences to take advantage of the virtualisation use-rights to cover the installed virtual OS’es.

It’s been in-place for a year and hardly ever has to be touched, its rock-solidly available and the company have noticed several marked improvements since they P2V’d their old servers onto this platform, as follows;

  • No hardware failures – moving from lots of low-end servers (Dell) and desktops to a single box (10:1 consolidation)
  • The DL380 has good redundancy built in, but it’s also backed up with a h/w maintenence contract, and they also have a spare cold-standby server to resume service from backups if data is lost.
  • Less noise, the old servers were dotted around their old offices in corners, racks etc – this is the main thing they liked!
  • Simple access anywhere – using a Juniper SA2000 SSL VPN,  its easy to get secure access from anywhere
  • Less reliance on physical offices and cheap DSL-grade data communications, now the servers are hosted on the end of a reliable, data centre class network link with an SLA to back it up. if an individual office looses its ADSL connection, no real issue – people pick up their laptop(s) and work from home/starbucks etc.
  • Good comms are cheaper in data centres than in your branch offices (usually)

Hopefully this goes to show the free version of VMWare’s server products can work almost as well if budget is a big concern, ESX would definitely give some better features and make backup easier, they are considering upgrading and combining with something like Veeam Backup to handle failover/backup.

HP Rapid Deployment Pack – PXE Settings for Deploying Windows OS

 

The followign screens show a working configuration from the RDP 3.80 PXE Configuration Manager

image

Have had lots of problems with this deploying Windows OS’es and VMWare ESX 3.5 onto an HP c7000 Blade chassis, still not resolved all the problems, but this definitely works for deploying Windows!

The documentation reads like you should always use the Linux PE configuration and it handles switching between WinPE/LinuxPE depending on which OS job you drop on a target. in my experience this doesn’t work and you need to manually change the PXE configuration to default to LinuxPE or WinPE depending on the OS you want to target.

image

And

image

 

Still a work in progress as I have a c7000 to which I want to deploy a mix of Windows and ESX/Redhat OS’es….

I did get a previous installation to install ESX 3.5 by hacking the default ESX 3.02 job, but its since been re-installed and I can’t do it now

RDP 6.90 seems to list Windows 2008 and ESX 3.5 in the quickspecs, but I’ll be damned if I can find where to download it, going to have to call HP methinks!

As I’ve posted before installing via iLo is just a non-starter if you really do want a flexible and fast deployment configuration – so it has to be RDP.

More later…

How does an HP Fibre Channel Virtual Connect Module Work?

 

Techhead and I have spent a lot of time recently scratching our heads over how and where fibre channel SAN connections go in a c7000 blade chassis.

If you don’t know, a FC-VC module looks like this, and you install them in redundant pairs in adjacent interconnect bays at the rear of the chassis.

image 

You then patch each of the FC Ports into a FC switch.

The supported configuration is one FC-VC Module to 1 FC switch (below)

clip_image002[6]image 

Connecting one VC module to more than one FC switch is unsupported (below)

clip_image002[8]image

So, in essence you treat each VC module as terminating all HBA Port 1’s and the other FC-VC module as terminating all HBA Port 2’s.

The setup we had:

  • A number of BL460c blades with dual-port Qlogic Mezzanine card HBAs.
  • HP c7000 Blade chassis with 2 x FC-VC modules plugged into interconnect bay 3 & 4 (shown below)

image image

The important point to note is that whilst you have 4 uplinks on each FC-VC module that does not mean you have 2 x 16Gb/s connection “pool or trunk” that you just connect into.

Put differently if you unplug one, the overall bandwidth does not drop to 12Gb/s etc. it will disconnect a single HBA port on a number of servers and force them to failover to the other path and FC-VC module.

It does not do any dynamic load balancing or anything like that – it is literally a physical port concentrator which is why it needs NPIV to pass through the WWN’s from the physical blade HBAs.

There is a concept of over-subscription, in the Virtual Connect GUI that’s managed by setting the number of uplink ports used.

Most people will probably choose 4 uplink ports per VC module, this is 4:1 oversubscription, meaning each FC-VC port (and there are 4 per module) has 4 individual HBA ports connected to it, if you reduce the numeber of uplinks you increase the oversubscription (2 uplinks = 8:1 oversubscription,  1 uplink = 16:1 oversubscription)

image

Which FC-VC Port does my blade’s HBA map to?

The front bay you insert your blade into determines which individual 4Gb/s port it maps to and shares with other blades) on the FC-VC module, its not just a virtual “pool” of connections, this is important when you plan your deployment as it can affect the way failover works.

the following table is what we found from experimentation and a quick glance at the “HP Virtual Connect Cookbook” (more on this later)

FC-VC Port Maps to HBA in Blade Chassis Bay, and these ports are also shared by..
Bay3-Port 1, Bay-4-Port 1 1,5,11,15
Bay3-Port 2, Bay-4-Port 2 2,6,12,16
Bay3-Port 3, Bay-4-Port 3 3,7,9,13
Bay3-Port 4, Bay-4-Port 4 4,8,10,14

image

Each individual blade has a dual port HBA, so for example the HBA within the blade in bay 12 maps out as follows

HBA Port 1 –> Interconnect Bay 3, Port 2

HBA Port 2 –> Interconnect Bay 4, Port 2

 

Looking at it from a point of a single SAN attached Blade – the following diagram is how it all should hook together

image 

 Path Failover

Unplugging an FC cable from bay 3, port 4 will disconnect one of the HBA imageconnections to all of the blades in bays 4,8,10 and 14 and force the blade’s host OS to handle a failover to its secondary path via the FC-VC module in bay 4.

 

A key take away from this is that your blade hosts still need to run some kind of multi-pathing software, like MPIO or EMC PowerPath to handle the failover between paths – the FC-VC modules don’t handle this for you.

 

image 

FC Loading/Distribution

A further point to take away from this is that if you plan to fill your blade chassis with SAN attached blades, each with an HBA connected to a pair of FC-VC modules then you need to plan your bay assignment carefully based on your server load.

Imagine if you were to put heavily used SAN-attached VMWare ESX Servers in bays 1,5,11 and 15 and lightly used servers in the rest of the bays then you will have a bottleneck as your ESX blades will all be contending with each other for a single pair of 4Gb/s ports  (one on each of the FC-VC modules) whereas if you distributed them into (for example) bays 1,2,3,4 then you’ll spread the load across individual 4Gb/s FC ports.

Your approach of course may vary depending on your requirements, but I hope this post has been of use.

image

There is a very, very useful document from HP called the HP Virtual Connect Fibre Channel Cookbook that covers all this in great detail, it doesn’t seem to be available on the web and the manual and online documentation don’t seem to have any of this information, if you want a copy you’ll need to contact your HP representative and ask for it.

HP iLo Very Slow for Installing an OS

 

A bit of a disappointment; we’re trying to do a WinPE 2.0 CD/DVD based installation for our Windows 2003/2008 standard blade servers in an HP c7000 enclosure.

Installing from a .ISO image presented to the iLo via the virtual media applet is dog-slow (5-10 times slower than from a physical CD/DVD- why is this? – surely its technically possible to make this access run faster and GigE chipsets are cheap-as these days. I’ve been through every combination of switching/duplex/port config and even via a cable directly into the Blade OA.

The same issue seems to manifest itself on traditional rack mount HP servers – the iLo just isn’t fast enough to make this a workable solution, unless you are really patient.

I know we could use the RDP and do it as a PXE type installation over the network to each blade, but this doesn’t really achieve what I want…

Most customers maintain an OoB (Out of Band) network to which all of the management interfaces  (iLo, DRAC, etc.) are connected to. the reasoning for this is obvious; if you loose your main core switching network you can get access via a totally different physical network and path to assist in troubleshooting.

For this same reasoning I would like to use this method to build servers from a master boot CD/DVD image, you can present a .ISO image to a server via the virtual media applet on the iLo. We have a fully end-end build process that sets up the HP array controllers, flashes BIOS and installs the OS and drivers etc. from a CD/DVD.

We just update the boot CD .ISO file as required and its flexible and it doesn’t rely on any deployment infrastructure (PXE server, RDP server etc.) so we can port it between customers and data centres, VM’s and physical machines and do a bare-metal builds without requiring any build/network infrastructure in place.

This isn’t just limited to a Windows OS – I tried the same with an ESX installation; took over an hour (compared to 5-10 mins from a local CD)

Performance Update on Cheap ESX PC

 

I’ve not done anything with my home ESX server this week as I’ve been busy with work; so this will be interesting – it’s been powered up all the time with all the VM’s spinning; but not doing very much.

CPU Utilisation

image

Memory Utilisation

image

Disk Utilisation

image

Whist running this set of VMs.. (the CPU stats for VMEX01 and VMEX02 are a bit skewed as I added this bit after the original post and they are both running seti@home (hence increased CPU)

image

 

So, nothing interesting to see here – but might be worth bearing in mind for some kind of sizing estimate; this is a single core CPU (HT enabled) PC with 4Gb RAM and a single 500Gb SATA disk

Hopefully I will get some time this week to load up SETI@Home or Folding@Home and see what that does 🙂 it should be a good test to see how well the hypervisor manages CPU timesharing between hosts.

Interesting Article on how DreamWorks are Speeding up Access for Animators

 

I have a geeky secret; I used to be really into ray-tracing and 3D graphics not so much from an “art” point of view – although I do have an interest in that and computer modelling/visualisation checks a lot of boxes for me as I always wanted to be a civil engineer or architect (well, I kind of am… but with computers..!)

it was one of the only applications I found in the early/mid 90’s that could really tax a machine and I spent a lot of time playing with large render jobs using PovRay and progressed to 3D studio for DOS and then a bit of a dabble with building render farms using 3DS Max before I had to go and get a “proper” job with less spare time.

I would love the time to get back into it, with the power available today you could produce some awesome images, although maybe I am somewhat hampered through lack of talent… maybe that will be downloadable now?

….So anyway, here’s an interesting article on how DreamWorks Animation have sped up access to their render farm using Ibrix Parallel file server software… they shift a lot of data!

I’ve worked on a project where we’ve tried to implement similar high-performance grid-based storage systems for large media files; but they were somewhat less successful/undeveloped; this one looks promising.

I wonder if these kind of vendors will start moving into the virtualization space; it’s essentially the same principal.

Deliver large flat files (.VMDK), over cheap/scalable commodity media (GigE) as quick a possible

This would reduce the depende.ncy on expensive back-end fibre channel SANs, and you could invest more in flexible Ethernet – or maybe Infiniband to deliver networking and storage within a “virtual fabric”

If it’s “virtual” and “grid” based the quality/features of individual hardware devices (DL380, NAS device etc.) that make it up the overall grid are less important and a 100% software approach gives you the flexibility to pick & choose building blocks from the most appropriate/affordable manufacturer rather than be locked into a costly single vendor solution (HP EVA, EMC Clariion, DMX etc.)

Thanks to Martin at Bladewatch for the link.

VMWare ESX on IBM Blades

 

I’ve worked with HP c-class stuff recently, but this is a good article on the IBM equivalent and the post looks a lot simpler to read than all the official IBM docs!

In my book if you are virtualizing your infrastructure there is less of a religious argument on the underlying hardware – it’s a lot more flexible so do you care as much?

Thanks to Martins post on Bladewatch for the handy link

New Blade Blog

 

As Scott points out, Aaron has made a promising start with his new Blade focused blog*

Welcome, Aaron – I’ve already found something useful on your site and may need to rethink one of my blade deployment plans slightly!

*Edited 10/2/08 to update link for Aaron’s new URL