Advertisements

Virtualization, Cloud, Infrastructure and all that stuff in-between

My ramblings on the stuff that holds it all together

Category Archives: HP Blade

Confused by HP Flex10 Design for vSphere

HP have a very clever 10Gb Ethernet technology called Flex10, which is an evolution of the previous Virtual Connect modules, it allows you to carve a single 10Gb NIC into multiple FlexNICs of varying capacities; this is especially useful if you are looking to deploy vSphere on HP blades.

it can be tricky to visualise Flex10 without some pictures and simple explanations of how it’s put into practice, and I know I struggled with this when I first implemented it as well as various firmware/driver update issues -  luckily Julian Wood (also in the UK) has an excellent set of posts on just this very subject – check out his blog here

Julian has some specific posts around Flex10 as below;

http://www.wooditwork.com/2011/02/18/scripting-flex-10-esx-design-with-powercli/

http://www.wooditwork.com/2011/02/17/flex-10-esx-design-with-simplicity-and-scalability-part-1/

http://www.wooditwork.com/2011/02/17/flex-10-esx-design-with-simplicity-and-scalability-part-2/

http://www.wooditwork.com/2011/01/17/hp-flex-10-esx-and-broadcom-fun-continues/

my HP, EVA, c-series blade resources can be found here and my original How does Virtual Connect Work? guide (now very out of date) is here

As an aside just before the Christmas break I did some work with a customer implementing the next-generation of Flex10 blade hardware which is called FlexFabric, essentially using the on-board 10Gb NICs in G7 HP blades as a converged network adapter (CNA) to do FCoE and Ethernet in a single device, it’s clever but I’ve seen a significant number of driver and firmware issues so some of the kinks are still being worked out in ESX/HP integration but it looks like they are getting there now.

Advertisements

Matching an HP FlexNIC in ESXi to Chassis Flex10 Bay and Port Number at Install Time.

 

When you are setting up an HP blade with ESXi you’ll want to ensure the administration interface is setup on the correct FlexNICs so you can add it to vCenter and continue the configuration – this can be confusing as there can be a large number of NICs presented. Up to 4 per Flex10 module, and in this case we have 2 Flex10 modules in the chassis, giving a maximum of 8 FlexNICs per blade.

At this stage I found that the Easiest way to match up which FlexNICs you have mapped on the Virtual Connect administrator to how they show up in the ESXi (as vmnicX)  is as follows;

iLo to the ESXi installation, Hit F2/Configure Management Network/Network Adapters

Compare this with the Port mapping feature via the OA (shown below) and you can map the MAC addresses for each vmnic against the FlexNIC LOM:x-y entry so it quickly becomes obvious which is which.

image

You can also look in the server profile assigned to the blade if you need more information about which VLAN a FlexNIC maps to.

image

HP Blade iLo – it’s 2010 and disappointingly still using a 100Mb NIC?

I wrote this original post over 2 years ago when I first had to build a set of HP blades in a c7000 chassis, having just taken delivery of another c7000 I had hoped that technology in the onboard administrator (OA) modules had progressed a bit further than it has.

For some reason they still only have 100Mb/s on board NICs, despite all other parts of this chassis using the all-singing and dancing Flex10 10Gb technology.

iLo/OA NIC Settings

image

Blade Chassis Interconnect Bays – high speed goodness

 image

Whilst I realise PXE and Altiris etc. are the preferred method for building out large numbers of blades, it doesn’t work for everyone and particularly when it’s the 1st chassis in a Greenfield environment you need to get an OS on a blade before you can even start to setup a PXE server – Windows 2003 over iLo takes about 4 hours to install, 2008 is even worse – or have someone use a USB attached CD to install the 1st blade an OS physically in the DC.

I accept that the iLo is for out of band, emergency type access where the blade OS isn’t accessible – but in my experience there are a large number of scenarios where you need to use tools that boot from DVD/CD .ISO images in a hurry like data recovery, forensic analysis, bare-metal OS restore tools, P2V tools etc. and 100Mb/s just doesn’t cut it these days.

Also if you work in a secured environment then getting clearance to run great and useful tools like DHCP/PXE on networks that could be routed to/from outside the DC is hard, sometimes virtual .ISO media over an isolated and secured OoB network or a person on-site in the DC with physical media is the only acceptable method for moving data.

Gigabit networking is such a ubiquitous technology these days, and with the OA modules for a blade chassis costing around £1k GBP each (2 generally required for redundancy) you would think there is enough budget to go for Gigabit on-board and make the excellent virtual media features really usable in all scenarios

image

On the plus-side, they have a fancy new BIOS splash screen 🙂

image

…or is it just me?

Resources for HP c-class blade and EVA Design for vSphere 4

 

I am currently working on a design for a vSphere 4  platform on HP’s EVA SAN and c-class blade chassis. In order to provide flexible network connectivity we are leveraging the new Flex 10 Virtual Connect Modules as well as VC Fibre Channel modules to simplify administration

Because finding things on the HP site can sometimes be a bit hit & miss, this post serves as a bookmark to the more useful resources I found.

Hardware Configurator – Generate Bill of Materials (BoM)

HP eConfigurator online tool to configure and cost blades and chassis options and produce a validated bill of materials – be sure to select your country to ensure you get the correct power options and list prices

image

 

Virtual Connect

HP Virtual Connect cookbook – updated for Flex 10 (Feb 2010)

Virtual Connect Webinar series

How does a Virtual Connect FC Module work? (warning – old and outdated with current firmware)

 

Flex10 Links

Virtualised Reality (Barry Coombs)

http://www.wooditwork.com/2011/02/18/scripting-flex-10-esx-design-with-powercli/

http://www.wooditwork.com/2011/02/17/flex-10-esx-design-with-simplicity-and-scalability-part-1/

http://www.wooditwork.com/2011/02/17/flex-10-esx-design-with-simplicity-and-scalability-part-2/

http://www.wooditwork.com/2011/01/17/hp-flex-10-esx-and-broadcom-fun-continues/

 

EVA Storage

HP EVA User Guide

Best Practices for HP Storageworks EVA with vSphere [Whitepaper]

Best Practices for HP StorageWorks Enterprise Virtual Array with VMware vSphere 4 [WEBINAR]

 

vSphere Installation

HP-Specific ESXi Installable Download (HP Passport Account Required)

 

Power

HP Power Calculator Spreadsheets (BL, DL, PL, EVA) in .xls format (Office 2010 users need to “Enable Editing” to take it out of protected mode in order for the links to work

image image

HP Blade Power Sizing Utility (can be a bit buggy and slow – but works) – and can export in a number of different formats including Word (Example Doc)

image

 

Firmware Maintenance CD Download

https://vinf.net/2010/04/15/latest-hp-firmware-maintenance-cd-v9/

Links Updated & Section reorganised 23rd Feb 2011

Virtualization – the key to delivering "cloud based architecture" NOW.

 

There is a lot of talk about delivering cloud or elastic computing platforms, a lot of CxO’s are taking this all in and nodding enthusiastically, they can see the benefits.. so make it happen!….yesterday.

Moving your services to the cloud, isn’t always about giving your apps and data to Google, Amazon or Microsoft.

You can build your own cloud, and be choosy about what you give to others. building your own cloud makes a lot of sense, it’s not always cheap but its the kind of thing you can scale up (or down..) with a bit of up-front investment, in this article I’ll look at some of the practical; and more infrastructure focused ways in which you can do so.

image

Your “cloud platform” is essentially an internal shared services system where you can actually and practically implement a “platform” team that operates and capacity plans for the cloud platform; they manage it’s availability and maintenance day-day and expansion/contraction.

You then have a number of “service/application” teams that subscribe to services provided by your cloud platform team… they are essentially developers/support teams that manage individual applications or services (for example payroll or SAP, web sites etc.), business units and stakeholders etc.

Using the technology we discuss here you can delegate control to them over most aspects of the service they maintian – full access to app servers etc. and an interface (human or automated) to raise issues with the platform team or log change requests.

I’ve seen many attempts to implement this in the physical/old world and it just ends in tears as it builds a high level of expectation that the server/infrastructure team must be able to respond very quickly to the end-“customer” the customer/supplier relationship is very different… regardless of what OLA/SLA you put in place.

However the reality of traditional infrastructure is that the platform team can’t usually react as quick as the service/application teams need/want/expect because they need to have an engineer on-site, wait for an order and a delivery, a network provisioning order etc. etc (although banks do seems to have this down quite well, it’s still a delay.. and time is money, etc.)

Virtualization and some of the technology we discuss here enable the platform team to keep one step ahead of the service/application teams by allowing them to do proper capacity planning and maintain a pragmatic headroom of capacity and make their lives easier by consolidating the physical estate they manage. This extra headroom capacity can be quickly back-filled when it’s taken up by adopting a modular hardware architecture to keep ahead of the next requirement.

Traditional infrastructure = OS/App Installations

  • 1 server per ‘workload’
  • Silo’d servers for support
  • Individually underused on average = overall wastage
  • No easy way to move workload about
  • Change = slow, person in DC, unplug, uninstall, move reinstall etc.
  • HP/Dell/Sun Rack Mount Servers
  • Cat 6 Cables, Racks and structured cabling

The ideal is to have an OS/app stack that can have workloads moved from host A to host B; this is a nice idea but there are a whole heap of dependencies with the typlical applications of today (IIS/apache + scripts, RoR, SQL DB, custom .net applications). Most big/important line of business apps are monolithic and today make this hard. Ever tried to move a SQL installation from OLD-SERVER-A to SHINY-NEW-SERVER-B? exactly. *NIX better at this, but not that much better.. downtime required or complicated fail over.

This can all be done today, virtualization is the key to doing it – makes it easy to move a workload from a to b we don’t care about the OS/hardware integration – we standardise/abstract/virtualize it and that allows us to quickly move it – it’s just a file and a bunch of configuration information in a text file… no obscure array controller firmware to extract data from or outdated NIC/video drivers to worry about.

Combine this with server (blade) hardware, modern VLAN/L3 switches with trunked connections, and virtualised firewalls then you have a very compelling solution that is not only quick to change, but makes more efficient use of the hardware you’ve purchased… so each KW/hr you consume brings more return, not less as you expand.

Now, move this forward and change the hardware for something much more commodity/standardised

Requirement: Fast, Scalable shared storage, filexible allocation of disk space and ability to de-duplicate data, reduce overhead etc, thin provisioning.

Solution: SAN Storage, EMC Clariion, HP-EVA, Sun StorageTek, iSCSI for lower requirements, or storage over single Ethernet fabric – NetApp/Equalogic

Requirement: Requirement Common chassis and server modules for quick, easy rip and replace and efficient power/cooling.

Solution: HP/Sun/Dell Blades

Requirement: quick change of network configurations, cross connects, increase & decrease bandwidth

Solution: Cisco switching, trunked interconnects, 10Gb/bonded 1GbE, VLAN isolation, quick change enabled as beyond initial installation there are fewer requirements to send an engineer to plug something in or move it, Checkpoint VSX firewalls to allow delegated firewall configurations or to allow multiple autonomous business units (or customers) to operate from a shared, high bandwidth platform.

Requirement: Ability to load balance and consolidate individual server workloads

Solution: VMWare Infrastructure 3 + management toolset (SCOM, Virtual Centre, Custom you-specific integrations using API/SDK etc.)

Requirement: Delegated control of systems to allow autonomy to teams, but within a controlled/auditable framework

Solution: Normal OS/app security delegation, Active Directory, NIS etc. Virtual Center, Checkpoint VSX, custom change request workflow and automation systems which are plugged into platform API/SDK’s etc.

the following diagram is my reference architecture for how I see these cloud platforms hanging together

image 

As ever more services move into the “cloud” or the “mesh” then integrating them becomes simpler, you have less of a focus on the platform that runs it – and just build what you need to operate your business etc.

In future maybe you’ll be able to use the public cloud services like Amazon AWS to integrate with your own internal cloud, allowing you to retain the important internal company data but take advantage of external, utility computing as required, on demand etc.

I don’t think we’ll ever get to.. (or want) to be 100% in a public cloud, but this private/internal cloud allows an organisation to retain it’s own internal agility and data ownership.

I hope this post has demonstrated that whilst, architecturally “cloud” computing sounds a bit out-there, you can practically implement it now by adopting this approach for the underlying infrastructure for your current application landscape.

HP Rapid Deployment Pack – PXE Settings for Deploying Windows OS

 

The followign screens show a working configuration from the RDP 3.80 PXE Configuration Manager

image

Have had lots of problems with this deploying Windows OS’es and VMWare ESX 3.5 onto an HP c7000 Blade chassis, still not resolved all the problems, but this definitely works for deploying Windows!

The documentation reads like you should always use the Linux PE configuration and it handles switching between WinPE/LinuxPE depending on which OS job you drop on a target. in my experience this doesn’t work and you need to manually change the PXE configuration to default to LinuxPE or WinPE depending on the OS you want to target.

image

And

image

 

Still a work in progress as I have a c7000 to which I want to deploy a mix of Windows and ESX/Redhat OS’es….

I did get a previous installation to install ESX 3.5 by hacking the default ESX 3.02 job, but its since been re-installed and I can’t do it now

RDP 6.90 seems to list Windows 2008 and ESX 3.5 in the quickspecs, but I’ll be damned if I can find where to download it, going to have to call HP methinks!

As I’ve posted before installing via iLo is just a non-starter if you really do want a flexible and fast deployment configuration – so it has to be RDP.

More later…

How does an HP Fibre Channel Virtual Connect Module Work?

 

Techhead and I have spent a lot of time recently scratching our heads over how and where fibre channel SAN connections go in a c7000 blade chassis.

If you don’t know, a FC-VC module looks like this, and you install them in redundant pairs in adjacent interconnect bays at the rear of the chassis.

image 

You then patch each of the FC Ports into a FC switch.

The supported configuration is one FC-VC Module to 1 FC switch (below)

clip_image002[6]image 

Connecting one VC module to more than one FC switch is unsupported (below)

clip_image002[8]image

So, in essence you treat each VC module as terminating all HBA Port 1’s and the other FC-VC module as terminating all HBA Port 2’s.

The setup we had:

  • A number of BL460c blades with dual-port Qlogic Mezzanine card HBAs.
  • HP c7000 Blade chassis with 2 x FC-VC modules plugged into interconnect bay 3 & 4 (shown below)

image image

The important point to note is that whilst you have 4 uplinks on each FC-VC module that does not mean you have 2 x 16Gb/s connection “pool or trunk” that you just connect into.

Put differently if you unplug one, the overall bandwidth does not drop to 12Gb/s etc. it will disconnect a single HBA port on a number of servers and force them to failover to the other path and FC-VC module.

It does not do any dynamic load balancing or anything like that – it is literally a physical port concentrator which is why it needs NPIV to pass through the WWN’s from the physical blade HBAs.

There is a concept of over-subscription, in the Virtual Connect GUI that’s managed by setting the number of uplink ports used.

Most people will probably choose 4 uplink ports per VC module, this is 4:1 oversubscription, meaning each FC-VC port (and there are 4 per module) has 4 individual HBA ports connected to it, if you reduce the numeber of uplinks you increase the oversubscription (2 uplinks = 8:1 oversubscription,  1 uplink = 16:1 oversubscription)

image

Which FC-VC Port does my blade’s HBA map to?

The front bay you insert your blade into determines which individual 4Gb/s port it maps to and shares with other blades) on the FC-VC module, its not just a virtual “pool” of connections, this is important when you plan your deployment as it can affect the way failover works.

the following table is what we found from experimentation and a quick glance at the “HP Virtual Connect Cookbook” (more on this later)

FC-VC Port Maps to HBA in Blade Chassis Bay, and these ports are also shared by..
Bay3-Port 1, Bay-4-Port 1 1,5,11,15
Bay3-Port 2, Bay-4-Port 2 2,6,12,16
Bay3-Port 3, Bay-4-Port 3 3,7,9,13
Bay3-Port 4, Bay-4-Port 4 4,8,10,14

image

Each individual blade has a dual port HBA, so for example the HBA within the blade in bay 12 maps out as follows

HBA Port 1 –> Interconnect Bay 3, Port 2

HBA Port 2 –> Interconnect Bay 4, Port 2

 

Looking at it from a point of a single SAN attached Blade – the following diagram is how it all should hook together

image 

 Path Failover

Unplugging an FC cable from bay 3, port 4 will disconnect one of the HBA imageconnections to all of the blades in bays 4,8,10 and 14 and force the blade’s host OS to handle a failover to its secondary path via the FC-VC module in bay 4.

 

A key take away from this is that your blade hosts still need to run some kind of multi-pathing software, like MPIO or EMC PowerPath to handle the failover between paths – the FC-VC modules don’t handle this for you.

 

image 

FC Loading/Distribution

A further point to take away from this is that if you plan to fill your blade chassis with SAN attached blades, each with an HBA connected to a pair of FC-VC modules then you need to plan your bay assignment carefully based on your server load.

Imagine if you were to put heavily used SAN-attached VMWare ESX Servers in bays 1,5,11 and 15 and lightly used servers in the rest of the bays then you will have a bottleneck as your ESX blades will all be contending with each other for a single pair of 4Gb/s ports  (one on each of the FC-VC modules) whereas if you distributed them into (for example) bays 1,2,3,4 then you’ll spread the load across individual 4Gb/s FC ports.

Your approach of course may vary depending on your requirements, but I hope this post has been of use.

image

There is a very, very useful document from HP called the HP Virtual Connect Fibre Channel Cookbook that covers all this in great detail, it doesn’t seem to be available on the web and the manual and online documentation don’t seem to have any of this information, if you want a copy you’ll need to contact your HP representative and ask for it.

HP iLo Very Slow for Installing an OS

 

A bit of a disappointment; we’re trying to do a WinPE 2.0 CD/DVD based installation for our Windows 2003/2008 standard blade servers in an HP c7000 enclosure.

Installing from a .ISO image presented to the iLo via the virtual media applet is dog-slow (5-10 times slower than from a physical CD/DVD- why is this? – surely its technically possible to make this access run faster and GigE chipsets are cheap-as these days. I’ve been through every combination of switching/duplex/port config and even via a cable directly into the Blade OA.

The same issue seems to manifest itself on traditional rack mount HP servers – the iLo just isn’t fast enough to make this a workable solution, unless you are really patient.

I know we could use the RDP and do it as a PXE type installation over the network to each blade, but this doesn’t really achieve what I want…

Most customers maintain an OoB (Out of Band) network to which all of the management interfaces  (iLo, DRAC, etc.) are connected to. the reasoning for this is obvious; if you loose your main core switching network you can get access via a totally different physical network and path to assist in troubleshooting.

For this same reasoning I would like to use this method to build servers from a master boot CD/DVD image, you can present a .ISO image to a server via the virtual media applet on the iLo. We have a fully end-end build process that sets up the HP array controllers, flashes BIOS and installs the OS and drivers etc. from a CD/DVD.

We just update the boot CD .ISO file as required and its flexible and it doesn’t rely on any deployment infrastructure (PXE server, RDP server etc.) so we can port it between customers and data centres, VM’s and physical machines and do a bare-metal builds without requiring any build/network infrastructure in place.

This isn’t just limited to a Windows OS – I tried the same with an ESX installation; took over an hour (compared to 5-10 mins from a local CD)

VMWare ESX on IBM Blades

 

I’ve worked with HP c-class stuff recently, but this is a good article on the IBM equivalent and the post looks a lot simpler to read than all the official IBM docs!

In my book if you are virtualizing your infrastructure there is less of a religious argument on the underlying hardware – it’s a lot more flexible so do you care as much?

Thanks to Martins post on Bladewatch for the handy link

VLAN Tagging with ESX and HP Blade Virtual Connect Modules

 

Useful reference articles here, here and clarification from Scott here.

Looking like I’ll be doing a lot with the HP C-Class Blade chassis this year, so this is useful.