Virtualization, Cloud, Infrastructure and all that stuff in-between

I am pleased to announce that I’ve been asked to present again at this month’s VMware London user group meeting.

I’ll be giving a follow-up on my previous Home Lab : vT.A.R.D.I.S session for which I’ll be bringing a (quieter!) demo environment which I have dubbed vTARDIS:Nano Edition showing how you can build a complex 10 node clustered ESX environment with dvSwitch, iSCSI SAN and vMotion etc. on a single physical host for study and play.

I’ll also be presenting a case study for a project I worked on virtualizing a large Microsoft Windows 2003 Terminal Server estate and will be showing some interesting statistics that we gathered along the way.

I understand the event is now fully-booked which is a great turn out!

The agenda as as follows

1100 – 1200 (Optional) PowerCLI / Powershell workshop – Alan Renouf. Please bring your own curly brackets.
12:30 – 13:00 Arrive & Refreshments
13:00 – 13:20 Welcome & News – Alaric Davies
13:20 – 14:00 Sponsor Presentation – Chris Hammans, Pano Logic
Real world vSphere deployment experiences – Stuart Thompson
(Mostly) Zero downtime DC migration for Dummies – Jonathan Medd
15:00 – 15:20 Refreshment break
ESX home lab update, virtualizing Terminal Server workloads – Simon Gallagher
Thin provisioning and capacity planning in a virtual world – Chris Evans, ‘The Storage Architect’
Bringing the Cloud Down to Earth – Stuart Radnidge, vinternals
16:45 – 17:00 Close
17:00 – Pub

In future, to register your interest in attending, please send an email to londonvmug@yahoo.com with up to two named attendees from your organisation. If you do not receive a confirmation mail, please don’t just turn up since we will not be able to admit you to the meeting.

Content from the meetings will continue to be uploaded to www.box.net/londonug, NDA permitting.

Following on from Eric’s post on running VMs from SSD’s at this page, and my previous experiments at using SSD’s to run VMs I thought I would post up my initial (non-scientific) findings from the FusionIO card that I have been loaned.

FusionIO make solid state storage cards that come packaged as PCIe x8 format expansion cards, they make use of multi-level cell (MLC) NAND storage to create amazingly high speed direct-attach storage, the Duo640 device I am working with is the mid-range offering, at the higher end forthcoming versions can support up to 1TB/sec throughput. the Woz is also their chief scientist 🙂

In my test rig I am using it and the Starwind software on Windows 2008 R2 to share the FusionIO card over iSCSI to a couple of vSphere 4 hosts – in this initial test I’m just using a single GbE NIC in both the server and the vSphere client – as you’ll see from the screenshot below it can actually max out the GbE connection in these hosts if you push it with several concurrent VM cloning sessions – so there is plenty more performance to be had out of the card and high levels of concurrency, in this case the NIC was the bottleneck.

The FusionIO duo card comes with 2 banks of 320Gb of memory (640Gb in total), in my initial configurations it’s not configured to RAID across the 2 banks, but that is possible to improve performance and fault-tolerance.

The FusionIO card doesn’t yet have drivers for vSphere but they are working on them – so you can’t directly access it from an ESX host yet so I am connecting to it using the Starwind software iSCSI target software.

One issue I found with my Starwind configuration is that the Starwind software is’t able to see the FusionIO card as raw block storage like it can with normal direct attached storage (SSD/SATA HDD etc.) although it is visible to Windows disk manager as a normal disk. so to get it to work I had to format and mount the FusionIO “disks” as NTFS drives under Windows 2008 R2 Disk Manager and create a virtual disk files in these drives using the Virtual Disk feature of the Starwind software – this is then accessible both directly to the Windows 2008 host and to my ESX hosts via iSCSI.

So, based on the same software that Eric used in his post these are the out of the box numbers the FusionIO card gets – there is still significant scope for fine-tuning to increase performance – But it’s pretty impressive.

FusionIO (non-RAID configuration) inside a VM over iSCSI

FusionIO (non-RAID configuration) – direct attached to a Windows 2008 R2 host

Consumer-grade SSD – direct attached to a Windows 2008 R2 host

This is using the following SSD which I purchased last year for under £200.

In conclusion…

The FusionIO cards aren’t cheap storage; they are in the ‘000’s of £ price-range, but they are FAST! with solid state storage pricing coming down in 2010 and when combined with iSCSI target software like Starwind it’s an excellent way to build a very high performance solid state SAN using DAS technology without the enterprise SSD SAN price-range and FC/network interconnects.

When vSphere drivers become available I can see some excellent 2-node/p2p replicating cluster/vSAN configurations using either Starwind or HP Lefthand Networks vSAN VM appliances as shown below, removing the dependency on single shared storage is a great design goal.

Disclosure: FusionIO and their UK distributor have kindly lent me a 640Gb duo card to work with, I have received no financial compensation nor have they imposed any copy approval or conditions with regards to what I write about their device – it’s just great and I’m that impressed.

Ok – it’s new year’s resolution time – how about this one..

User groups (and blogs, in a less face-face manner) are an excellent way to meet like minded tech people in an informal setting and are a useful way to get information about how other people are doing things and current real-world trends/best practice.

But, none of this is possible without people stepping up and contributing – you don’t have to be a genius to contribute (look at me! :)) nor do you have to be in-charge of the largest planet munching datacentre laden with the most advanced, cutting edge tech in the world, even the humblest IT shop have something in the way of experience that they can contribute – what problems have you seen, and how did you fix them, what do you think would be useful, something creative you’ve built?

Public speaking isn’t my idea of absolute fun, it’s hard and you are putting yourself “out there” in anticipation that people will be interested; or at the very least be polite enough not to throw things at you.

my advice..

Don’t feel you have to know everything about everything, it’s ok to say I don’t know, and throw it to the floor – the man who says he knows everything actually knows nothing! it could spark an interesting debate – you don’t get that kind of thing at formal conferences.

User groups are about users, not sponsors or vendors showing their wares – they have ample online and conference time for that (although sponsors are obviously an important part of it – as they pay for it!) so take advantage of the experience in the room

It’s also easy to start a blog, it’s easy to get your thoughts out there (however serial twitter RT’ers and blog-scrapers need not apply :)).

it’s also good personal and career development, even if it’s just about making you structure your thoughts properly  – I wrote some thoughts on this a while back

If you have some ideas for user group presentation sessions, or indeed something different – just write up a proposal, it doesn’t need to be anything majorly formal – just an email with the salient points and submit it to the co-coordinators;

Title:

Format: presentation/panel/discussion

How long you would like: (keep it under 45mins)

Outline: agenda, key points and/or questions you would like to cover, what people would get from the session

Along similar lines – get a blog, get some thoughts, something you’ve fixed (with some pictures) even something you’d like to see in future versions – write it up, get it out there www.wordpress.com is all you need!

Go forth and contribute in 2010 .. 🙂

I recently presented a session at the London VMware User Group meeting about home labs, this post is a follow-up with the slides I used and some more details on the configuration.

The kit I demo’d has affectionately been named the the vT.A.R.D.I.S which stands for Trolley Attached Random Datacentre of Inexpensive Servers 🙂 or Hernia-maker – don’t feel like you actually have to strap yours to a trolley though 🙂

This is part of a series of joint postings with my esteemed colleague Mr.Techhead, my sections of the series concentrate on the details of building a virtualized ESX cluster using the vSphere 4 for learning and test & development; Techhead’s posts will focus on the best low-cost hardware to use and specific configuration steps and I will cover some of the configurations and use cases.

You may be wondering why you would want to do this? well, if you are studying for your VCP or developing scripts or utilities for managing vSphere environments you rarely have a multi-node cluster at your disposal to test against because by it’s very nature it requires a lot of {usually expensive} hardware and you miss the more advanced configurations like HA/DRS/FT that this type of environment can use.

Also consider the larger production-type environment where you want to test some automatic deployment or management scripts and tools  – this is an ideal approach which uses minimal hardware to conduct the 1st stages of test and development – if you’re an ITIL shop this is release management. Even the best equipped test labs won’t give you more than a couple of hosts to play with – this virtualized ESX approach means you can have many more ESX hosts to test against without busting the bank.

So we have put our heads together and have come up with what we think is the lowest possible cost way to build such an environment, and unsurprisingly it makes heavy use of virtualization – to allow you to study and work on without

• Being too noisy to leave switched on
• drawing too much {expensive} power
• costing the earth

The catch: Now, of course nothing is for free so to build this it will cost you some money, but it will be a lot less than your typical production environment and more into the hobbyist market – of course you get what you pay for, and I wouldn’t be going into this with the expectation that this will perform well enough for you to compete with EC2 🙂 but for your own general home use; and probably that of an SoHo/SME type organisation it’s ideal.

The photo below shows the demo kit we used for the London meeting cunningly strapped to a B&Q trolley for “portability” 🙂

<

To break it down into each major functional area and as a taster of the follow-up posts here are some of the things you need to consider..

Storage

Shared storage is a requirement for HA/DRS/FT and is usually the most expensive part in a production environment which would typically be Fibre Channel and SCSI disk SAN storage, you’ll never get this on our budget so we have taken the OpenSource and iSCSI and SATA approach, we have put this through its paces for the last 2 years in varying topologies and it performs very well and will more than service your own personal/study needs, it also has the advantage that it can probably be recycled from that pile of spare PC parts you have in the cupboard.

There are also a number of low-cost NAS devices which should be within your budget if you don’t; Techhead has a number of posts on the way around this.

Network

Building flat networks is easy – you just need a dumb switch, or even a hub and away you go; but by doing this you miss the subtle configuration problems you need to understand to do things properly in a production environment, so ideally you need something that will support VLANs and routing – you also need Gigabit ports for vSphere; although I have had vMotion working on a 100Mb switch in the past.

We have looked for a long time but there are no cheap (<£400) Gigabit switches even if you go 2nd hand.

There are numerous low-end switches that support VLANs, but can’t do the routing between VLANs so you either need an external hardware router like a Cisco 2600 or something else..

So, a compromise is needed – we opt for a low-cost Gigabit switch with VLAN support like the 8-port Linksys SLMxxx and compliment it with a virtual machine running the Vyatta community edition virtual appliance which can provide the L3 routing betweenn your VLANs (a sample of how easy to configure it is shown below)..

Server

Techhead is an avid HP-fan; and rightly so as they make great production kit but I had never really explored the lower-ends of their range such as the ML110 and ML115 range – these are single CPU socket servers with internal (non hot-swap) SATA storage, whilst they don;t have on-board redundant hardware they are quiet and more importantly – surprisingly cheap and fully ESX 3/4 compatible.

Techhead has some good deals on the ML115 G5 hardware at this link, here and here and best of all the ML115 G5 is compatible with the new Fault Tolerance feature of vSphere

if you wonder what is inside an ML115 server read this link

Hypervisor & Nested Hypervisor VM

VMware ESXi is my current weapon of choice for this environment and so will be the focus of this series of posts; unfortunately I’ve not found a way to run nested Hyper-V or Xen Virtual Machines, that would be the ultimate in flexible learning platform – unless anyone out there knows how to?

I make heavy use of the new Fault Tolerance feature of vSphere to protect the vCenter and Vyatta virtual machines in this environment.

It’s the ideal setup to test unattended deployments of ESX hosts as well as you can just delete them and start again.

Virtualized ESX Hosts – 10 ESXi hosts running on 2 physical machines

Detailed Posts Index

Rather than do one long post we have a series of break-out posts on the specific areas of this topic.

this is the list of topics to come; when articles are posted the links will be populated and become clickable.

Part 1 – Lab Hardware Overview  (coming soon @Techhead)

Part 2 – Lab Hardware Configuration (coming soon @Techhead)

Part 3 – ESXi Installation & Configuration (coming soon @Techhead)

Part 4 – Shared Storage Installation & Configuration (coming soon @Techhead)

Part 5 – Networking Configuration (VLAN’ing & Jumbo Frames) (coming soon @Techhead)

Part 6 – VM’d ESXi (Coming soon @vinf.net)

Part 7 – VM’d vCenter; auto start-up of VMs (Coming soon @vinf.net)

Part 8 – VM’d FT and FT’ing vCenter VMs (Coming soon @vinf.net)

Part 9 – FT on the ML115 series – benchmarking with some Exchange VMs (Coming soon @vinf.net)

Part 10 – VM’d Lab Manager farm environment on a pair of ML’s (VM’d ESXi) (Coming soon @vinf.net)

Part 11 – VM’d View 4 farm environment on a pair on ML’s (VM’d ESXi) (Coming soon @vinf.net)

Part 12 – Home backup – VMware data recovery / fastSCP/Veeam backup or something else low-cost with USB drives/etc. (Coming soon – joint posting)

The slides from my original VMUG presentation are available online at this link

I’ve seen a couple of instances of this in the last week where previously working Exchange 2003 servers suddenly stop serving Outlook Web Access (OWA) Requests overnight

Investigating the eventlog shows the following entry which corresponds with stopped application pools in IIS Manager;

Event Type:        Error

Event Source:    W3SVC

Event Category:                None

Event ID:              1059

Date:                     14/12/2009

Time:                    02:01:37

User:                     N/A

Computer:          EXCHANGESERVER

Description:

A failure was encountered while launching the process serving application pool ‘DefaultAppPool’. The application pool has been disabled.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

This seems to be related to a recent Microsoft Update – 973917 uninstalling the hotfix, followed by a reboot immediately resolves the problem and OWA starts working again.

I’ve also had reports from people that this update has affected some other bespoke IIS applications, again uninstalling the patch seems to resolve the problem.

This situation seems to arise from a mismatch of installed binaries – now I thought system file protection and the .msi based patch installers were supposed to avoid this situation from Windows 2003 and onwards – anyway for more information on the cause and how to resolve without uninstalling the 973917 patch see the following links

http://msdnrss.thecoderblogs.com/2009/12/16/aftermath-of-the-release-of-kb-973917-for-iis/

http://blogs.iis.net/nazim/archive/2009/12/10/issues-installing-kb-973917-on-windows-server-2003.aspx

I am currently building a new home office/lab/cave for my computer habit and have been maintaining a separate blog with all of the details – it’s online at http://geekcabin.wordpress.com so please check it out.

Don’t worry I’ll still be posting my normal content here and nothing is happening to this blog – but the other blog is for people interested in the construction etc. and keeps it away from my technical stuff

It’s turning up next week (just in time for xmas – hopefully) and I look forward to having some space to work on things again as children seem to take up so much room – especially after every xmas!

This is a much wanted feature, I haven’t checked yet – but if this is allowed for Windows instances I can see a whole heap of new use-cases in my work – check it out – this is akin to boot from SAN in a traditional infrastructure and allows for persistent OS images to be kept around when they are not running on EC2.

official announcement..

Amazon EC2 Boot from Amazon EBS

Amazon EC2 has also announced the ability to boot instances directly from Amazon EBS snapshots, providing significantly increased flexibility in how customers can manage their instances. You can still save an Amazon Machine Image (AMI) in an Amazon S3 bucket and boot it from the local instance store, but you can now also choose to save AMIs as Amazon EBS snapshots and boot directly from an Amazon EBS volume. When an instance is booted from an Amazon EBS snapshot, the root partition of the instance is created on an Amazon EBS volume. Instances booted from Amazon EBS volumes can be stopped and later restarted, preserving any of the state that is saved to your volume and allowing you to modify some properties of your instances while it is stopped. For example, you can change your instance size or update the kernel it is using, or attach your root partition to a different running instance, making it easier to do debugging when you are creating new boot images. When booting from an Amazon EBS volume, AMIs and root partitions are no longer limited to 10GB, but can be up to 1TB in size, enabling significantly more complex images. Additionally, you are not charged for stopped instance hours and you will only incur charges for your Amazon EBS volumes while your instance is stopped, allowing you to reduce your Amazon EC2 costs when you do not need your instances running. Customers can now use a newly launched API that makes it easy to bundle images without using the command line tools, and can also take advantage of the fact that the content of an Amazon EBS volume is available to the instance immediately on volume creation which can lead to much faster instance boot times. For more details on this new addition to Amazon EC2, please see the Boot from Amazon EBS Feature Guide.

The esteemed Mr Techhead and I will be presenting on how to build a low-cost lab environment for work, study and play at this month’s VMUG in London.

We will be bringing our own equipment and hope to have lots of (working!) demos of what you can do with low-cost server kit and interesting configurations

• Running virtualized multi-node ESX clusters? – check
• VMware FT on a £300 GBP server? – check
• How to get a SAN for your home? – check

If you are able to make it, it promises to be an interesting event; the full agenda is as follows;

The Steering Committee are pleased to announce the next UK London VMware User Group meeting, kindly sponsored by Symantec, to be held on Tuesday 24th November 2009. We hope to see you at the meeting, and afterwards for a drink or two.
Our meeting will be held at the Thames Suite, London Chamber of Commerce and Industry, 33 Queen Street, London EC4R 1AP, +44 (0)20 7248 4444. The nearest tube station is Mansion House, location information is available here. Reception is from 1230 for a prompt 1pm start, to finish around 5pm. Our agenda looks something like this:

1100 – 1200 (Optional) Powershell workshop – Jonathan Medd (get-tovenue | early) to cover PS one-liner solutions and new VI toolkit functionality
12:30 – 13:00 Arrive & Refreshments
13:00 – 13:15 Welcome & News – Alaric Davies
13:20 – 14:05 Sponsor Presentation – Symantec
14:10 – 14:55 ESX Home Lab/White box setup – Simons Gallagher & Seagrave
15:00 – 15:20 Refreshment Break
15:25 – 16:00 Session 3 – vSphere topics, Guy Chapman, Mike Laverick, audience
16:05 – 16:45 Session 4 – vSphere topics, Alaric Davies, Mike Laverick, audience
16:45 – 17:00 Close
17:00 – Pub

To register your interest in attending, please send an email to alaricdavies at yahoo dot com with up to two named attendees from your organisation. If you do not receive a confirmation email, please don’t just turn up since we will not be able to admit you to the meeting. Please separately mention if you intend attending Jon’s Powershell workshop at 1100. Content from the meetings will continue to be uploaded to www.box.net/londonug, NDA permitting.

My home lab has a pair of HP ML110 servers with 8Gb of RAM running vSphere 4 (more info here) it’s configured in a cluster with iSCSI storage running from an old HP D530 PC with a 1Tb hard disk running OpenFiler. it performs pretty well and meets most of my needs, I thought I’d do a quick couple of screenshots of the average performance I have seen on it over the last 3 months.

it’s running a constant load of about 17 mostly Windows virtual machines and a varying load of test environments which are suspended to disk – think the most I have ever had running on the 2-node cluster at one time was about 45 VMs and performance was ok – trying to use VUM to patch all those VMs at the same time killed things though, as all the VMs are running from a single 1Tb SATA disk over OpenFiler.

This is a list of all the VMs, you can create your own html list as follows, or you can also save it as a CSV to import into Excel to manipulate.

the following screenshots show the last 3 months of performance stats from vCenter as the number of VMs has increased and decreased as I’ve provisioned and removed VMs for testing.

Overall CPU usage for the cluster

vMotion and VM Reconfiguration activities

Cluster Memory consumption

The new overview page feature can show you a quick summary of virtual machine performance

Drilling down into the performance tab on each host gives more information on specific performance like disk and network

You can also produce a stacked graph showing guest CPU usage of each VM on a host

or identify which VMs have the busiest virtual disks

You can also view a stacked (per VM) graph showing on a per-host basis how much physical RAM the guests are consuming, relative to each other over time.

I work as a technical architect in a consultancy role in the Windows Infrastructure & Virtualization space, and as such I’m paid for my experience and opinion, but it’s also an important part of my job to proactively keep on to top of industry trends and investigate new technologies that may solve customer problems, even problems they don’t know they have {yet}.

Twitter and blogs are a useful resource for current and less formal or structured information if used carefully, but it’s all too easy to be distracted from the job at hand by the latest shiny tech or flamewar so you need to measure the amount of time you spend there and they are no substitute IMHO for traditional study and hands-on “playing” time.

Like many I’m currently studying for my VCP4 exam, and have recently upgraded my MCSE to the current MCITP:EA 2008 certification so I thought I would publish a reading list for anyone else looking to do the same as well as general industry books I have been reading recently.

Maintaining a balance between this study/pro-active time and staying a well utilised/billed out resource is hard and I find it often spills into my own personal time so you need to have a good level of personal interest/dedication otherwise you will struggle – I’m a geek at heart and I look upon this as investing in my career but it’s more important that your kids remember what you look like!

This is one area where travelling for work via public transport rather than driving works well – plenty of study time to be had with a good pair of headphones

(i.e not the rubbish ones that ship with the iPod), a good list of tunes to block everyone else out and a book/laptop with VMs.

Maintaining a good home lab environment is also critical for me, and I run all the services for my demanding demo environments as well as my home users (read: wife & kids) there and that really does teach you something about availability :)  I wrote some details of my lab setup here but I have an updated post in the pipeline as there have been some significant changes to support my vSphere study.

Anyways, enough rambling and on with the list..

VCP4 Upgrade

You can’t go far wrong with the Lowe, this is the definitive vSphere book at the moment.

VCP4 practice exams online at the SLOG here

VCP4 resource list here

if you wanted a similar book for VI3 I would recommend this one;

Microsoft Certified IT Professional (MCITP)

Windows 2008 MCITP Core Exams study guide

This is a comprehensive set of books for the 4 core exams, not as dry as previous versions and lots of practical lab excercises, as usual good set of practice tests on the included CD

For the client component of the MCITP:Enterprise Administrator cert

MOSS 2007 Exam

Not the most exciting product for me, but it does cover the exam requirements – seems to spend a lot of time explaining what each menu item is within the MOSS UI, which was a bit frustrating.

Very good book, puts a lot of real world around MOSS deployments

General Good Books to Read

Cisco UCS

Good book covering what Cisco’s new UCS blade system and consolidated I/O model is; not much information available elsewhere at the moment, although 1/2 the book is spent discussing the various CPU/memory bus architectures which is a good update to your knowledge but would have liked to have seen more time dedicated to how UCS works and some example configurations

General computing/tech

An excellent book, with a focus on open source technologies and a lot of practical insight from the building of Flickr – good for briding the infrastructure/application divide.

If you don’t get cloud computing or understand where things could go, you need to read this book, brilliant and not a long, drawn-out read and not that technical.

Aimed more at developers rather than infrastructure people but if you want to try things out for yourself it has some easy to understand examples.

I hope you found this list useful, you may notice that these are affiliate links – should you wish to purchase any of these books Amazon will pay me a nominal commission which I can use towards the normally ad-free funding of this site and my time, although you are entirely able to go and purchase any of these books directly from the Amazon site.

Baby and non-IT related books

We recently had our 2nd child and it reminded me of these great books for all fathers to-be 🙂