Virtualization, Cloud, Infrastructure and all that stuff in-between

My ramblings on the stuff that holds it all together

Nexentastor CE performance not as good as expected with SSD Cache check it is actually working

I encountered this problem in my lab – I have the following configuration physically installed on an HP Microserver for testing (I will probably put it into a VM later on however)

1 x 8Gb USB flash drive holding the boot OS

And the following configured into a single volume, accessed over NFSv3 (see this post for how to do that)

1 x 64Gb SSD Drive as a cache

4 x 160Gb 7.2k RPM SATA disks for a raid volume in a raidz1 configuration

A quick benchmark using IOmeter showed that it was being outperformed on all I/O types by my Iomega IX4-200d, which is odd, as my Nexentastor config should be using an SSD as cache for the volume making it faster than the IX4  So I decided to investigate.

If you look in data management/Data Sets and then click on your volume you can see how much I/O is going to each individual disk in the volume.

image

In my case the SSD c3d1 had no I/O at all – so if you click on the name of the volume shown in green (in my case it’s called “fast”) then you are shown the status of the physical disks.

So, from looking at the following screen Houston we have a problem – my SSD is showing as faulted (but no errors are recorded either) – so I need to investigate why (and hope it’s still under warranty, if it has actually failed this will be the 2nd time this SSD has been replaced!)

image

Attempts to manually online the disk return no error, but don’t work either so not entirely sure what happened there, I did have to shut down the box and move it so I re-seated all the connectors but it still wouldn’t let me re-enable the disk.

Worth noting that even with this fault the volume remained on-line; just without the cache enabled so I was able to storage vMotion off all the VMs and delete and re-create the volume (this time I re-created it without any mirroring for maximum performance.

Once I had storage vMotioned the test VM back (again, no downtime – good old ESX!) I ran some more Iometer tests and performance looked a lot better (see below)

image

I’ll be posting some proper benchmarks later on, but for now it was interesting to see how much better it could perform than my IX4 (although remember there is no data-protection/RAIDZ overhead so a disk fault will destroy this LUN – good enough for my home lab though, and I plan to RSYNC the contents off to the IX4 for a backup ).

Fingers crossed this isn’t a fault with my SSD… time will tell!

Nexentastor, When 1Gb just isn’t enough

 

I have been trying to get my Nexentastor SSD/SATA hybrid NAS working this last week and I’ve found that the web UI grinds to a halt sometimes for me, I couldn’t find a UNIX ‘top’ equivalent quickly but the diagnostic reports that you can generate from the setup menu command line did indicate that it was short of RAM.

The HP Micro server I am using shipped with 1Gb of RAM, and normally that would be fine for a file-server/NAS but I’m thinking that Nexentastor does a fair bit more and is based on OpenSolaris rather than a stripped down Linux or BSD; the eval guide says 768Mb is enough for testing, 2Gb better 4Gb ideal so I was already pushing my luck with 1Gb for any real use.

So, I bit the bullet and ordered 8Gb of RAM for the server, which is the maximum you can install – ironically this cost the same amount as I paid for the whole Microserver in the 1st place (after the cash-back deal) but that’s reflective of the fact it only has 2 memory slots so I had to opt for the more expensive 4Gb chips.

I went for 8Gb as at some point I will probably re-run my experiments under ESXi and deploy this host as a part of my management cluster for the vTARDIS.cloud.

I am also booting the OS from a USB flash-drive – I had several 2Gb units but it wouldn’t install to them as they didn’t have quite enough space, so I’m using an 8Gb flash drive to hold the OS – this isn’t the most performant drive either so any swapping will be further impacted by the USB speed.

I’m Pleased to report that the 8Gb RAM upgrade has resolved all the problems with navigating the UI, and should also yield further I/O performance as the Nexentastor software uses the extra RAM as extra cache (ARC) as well as the SSD (L2ARC) – there is a good explanation of that on this blog post.

I’m going to post up my I/O benchmarking when I have some further wrinkles ironed out – in the meantime there is an excellent post here with some example benchmarks running Nexentastor in a VM on a slightly more powerful HP ML110 server.

London VMware User Group VMUG Feb 10th 2011

 

It’s that time again, if you are in the UK – or anywhere nearby then register and get yourself over to the London VMUG.

I’m not sure if I can make this one, due to work commitments and this will be the 1st VMUG for about a year and a half where I won’t be presenting anything – so it’s safe to come out from behind the sofa! Smile

The Steering Committee are pleased to announce the next UK London VMware User Group meeting, kindly sponsored by Veeam is to be held on Thursday 10th February 2011. We hope to see you at the meeting, and afterwards for a drink or two, courtesy of VMware.

Our meeting will be held at the Thames Suite, London Chamber of Commerce and Industry, 33 Queen Street, London EC4R 1AP, +44 (0)20 7248 4444. The nearest tube station is Mansion House, location information is available here. Reception is from 12.30 for a prompt 1pm start, to finish around 5pm. Our agenda is below and is subject to change (but hopefully not too much!)
 
10.00 – 12.00      Roundtable Strategy Session with Martyn Storey and Mark Stockham; Enterprise Management – optional (spaces limited)
11:00 – 12:00      PowerCLI session with Alan Renouf – optional
 
12:30 – 13:00      Arrive & Refreshments
13:00 – 13:20      Welcome & News (Alaric)
13:20 – 14:00      Sponsor presentation (Veeam Systems)
14:00 – 14:30      VMware Certification – Preparing for Success (Scott Vessey)
14:30 – 15:00      Cheap Disaster Recovery using PowerShell scripts (Gabrie van Zanten http://www.gabesvirtualworld.com/)
15:00 – 15:20      Refreshment break
15:30 – 16:00      Advanced vCenter Alarming and Automation (Simon Long, VMware)
16:00 – 16:30      Transatlantic Datacentre migration (Chris Dearden, JFVI)
16:30 – 16:45      Close
17:00                     Pub
 
Notes:
If you would like to participate in Alan’s workshop, please bring a laptop, preferably with the most current PowerCLI and PowerShell binaries installed.
 
To register your interest in attending, please email londonvmug@yahoo.com with up to two named attendees from your organisation. If you do not receive a confirmation mail, please don’t just turn up since we will not be able to admit you to the meeting. Please separately mention if you intend attending Alan’s PowerCLI workshop at 11.00 or would like to be considered to attend the Enterprise Management roundtable strategy session at 10.00. Content from the meeting will be uploaded to http://www.box.net/londonug, NDA permitting.

Sincerely, and with regards,

The London VMUG Steering Committee

Building a Fast and Cheap NAS for your vSphere Home Lab with Nexentastor

 

My home lab is always expanding and evolving – no sooner have I started writing up the vTARDIS.cloud configuration than something shiny and new catches my eye! fear not I will be publishing the vTARDIS configuration notes over the next 2 months, however in the meantime I have noticed that my IX4-200d NAS has been bogging down performance a bit recently – I attribute this to the number of VMs I am running across the 5 physical (and up to 50) vESXi hosts.

The IX4 is great and very useful for protecting my photos and providing general media storage but I suspect it also uses 5.4k RPM disks and in a RAID5 configuration it performs /ok/ but I feel the need, the need for speed Smile

With the sub-£100 HP MicroServer deal that is on at the moment I spotted an opportunity to combine it with some recycled hardware into a fast NAS box, using some new software – the NexentaStor Community Edition, I’ve used OpenFiler and the Celerra VSA a lot in the past but this has some pretty intriguing features.

Nexentastor allows you to use SSD as a cache and provides a type of software RAID using Sun’s ZFS technology – you can read a good guide to configuring it inside a VM on this excellent post

I already have a number of ML115 and ML110 servers, which all boot from 160Gb 7.2k RPM SATA disks; most of the time they do nothing so an idea was born, I will switch my home lab to boot from 2Gb USB sticks (of which I have a plentiful supply) and re-use those fast SATA disks in the HP MicroServer for shared, fast VM storage

I also have a spare 64Gb SSD from my orginal vTARDIS experiments which I am planning to re-use as the cache within the MicroServer

So, the configuration looks is like this;

image

Because I want maximum performance and I don’t care particularly about data protection for this NAS I’m just going to try striping data across all the SATA disks for best performance and I hope the SSD will provide a highly performant front-end cache for VMs stored in it (if I understand how it works correctly).

Most of the VMs it will be storing are disposable or easily re-buildable but I can configure RSYNC copies between it and my IX4 for anything I want to keep safe (or maybe just use one of the handy NFR licenses Veeam are giving out)

I did consider putting ESXi on the HP MicroServer and running Nexentastor as a VM (which is supported) but I haven’t yet put any more RAM in the MicroServer, although I may do this in future and add it to my existing management cluster.

I’ll post up some benchmarks when I’m done.

New Storage Benchmarking Tool from VMware Labs

 

To quote the site..

IOBlazer is a multi-platform storage stack micro-benchmark. IOBlazer runs on Linux, Windows and OSX and it is capable of generating a highly customizable workload. Parameters like IO size and pattern, burstiness (number of outstanding IOs), burst interarrival time, read vs. write mix, buffered vs. direct IO, etc., can be configured independently. IOBlazer is also capable of playing back VSCSI traces captured using vscsiStats. The performance metrics reported are throughput (in terms of both IOPS and bytes/s) and IO latency.

IOBlazer evolved from a minimalist MS SQL Server emulator which focused solely on the IO component of said workload. The original tool had limited capabilities as it was able to generate a very specific workload based on the MS SQL Server IO model (Asynchronous, Un-buffered, Gather/Scatter). IOBlazer has now a far more generic IO model, but two limitations still remain:

The alignment of memory accesses on 4 KB boundaries (i.e., a memory page)
The alignment of disk accesses on 512 B boundaries (i.e., a disk sector).
Both limitations are required by the gather/scatter and un-buffered IO models.

A very useful new feature is the capability to playback VSCSI traces captured on VMware ESX through the vscsiStats utility. This allows IOBlazer to generate a synthetic workload absolutely identical to the disk activity of a Virtual Machine, ensuring 100% experiment repeatability.

You can download it and find more information here – I’m going to try and use this in my upcoming NexentaStor NAS project.

That was 2010 so what next

 

I’m a bit late to the Internets with this post but happy new year to all my readers – and judging by my WordPress stats there are a lot of you! over 400k visitors this year!

So – what happened in 2010? for me, it was a strange year, I knew from the start of the year that I had chosen to leave the relative comfort of the position that I had held at ioko for the previous 10 years to do “something new” – and that it was that stark, a long notice period and nothing definite lined up so I took a bit of a leap of faith and it worked out as I joined VMware the day after I left.

However, there were plenty of projects to complete before I left ioko and I also cracked on with my blog in my limited spare time whilst also juggling a new-born baby and assortment of home DIY projects

So, what’s next…?

This year I would like to focus on the following;

  • New layout and look for vinf.net – the default WP theme is getting a bit tired.
  • Spend more time bridging development and infrastructure; these are two traditionally separate camps, and particularly with “cloud” there needs to be a much better integration across the two, devops is particularly of interest to me.
  • Working with an “upcoming product” to unify the two camps for the cloud (see what I did there? Winking smile).
  • Learn to program again, it’s been a long time since I did any real hands-on development and its going to be very helpful in enabling this inf/dev bridge.
  • Keep developing the vTARDIS and do not talk about fight-club
  • Keep up with the presenting at user groups, conferences etc.
  • Learn to play bass guitar properly at least a little bit
  • Complete my VCDX on VI3 or VI4.
  • Blog at least once a week, I’ve found twitter has replaced some of my “look at this, this is cool” type blog posts that I used to enjoy writing – but it’s hard to get any kind of objective opinion into 140 chars – so I’m going back to blog basics.
  • Manage twitter distractions better, i.e close TweetDeck.
  • London – Brighton bike ride with a better time than the heavily jet-lagged one I put in this year, and not to walk the beacon.
  • London – Paris bike ride (either the 24hr race or 3 day ride)
  • At least 1,500 miles cycled (oh and upgrade my road-bike, again..)
  • Spend less of my own money on my home lab (donations accepted Smile)…

 

Have a good one!

BriForum comes to London in 2011

 

Brian Madden’s popular BriForum event is making a welcome return to Europe in 2011, and specifically to my home town of London

If you are in the server based computing space (Citrix, Terminal Services, VDI) then this is the place to be for no-vendor bull5h1t real-world information.

I’ve been to 2 previous BriForum events (2007 in Amsterdam in 2010 in Chicago, where I had the honour of presenting my vTARDIS).

there is currently a discount code available that will save you £200

Click here for more information

Pushing Packets Round The M25

 

Since starting at VMware I find myself driving a lot more on one of the UK’s greatest national landmarks – the M25, it’s such a delight to tourists and locals alike I really encourage you all to check it out (unless you actually want to get anywhere fast of course Smile) infact if you click here you can check it out on the webcams – particular “highlights” are just before 8.30am and around 5.30pm.

it’s very much a case of this

image

rather than this

image

Although there are occasional moments of entertainment…

image

I can only listen to Radio 1, Capital or Kiss or even Radio 4/LBC for so long before I want to poke pencils in my ears (think I’m getting old) and my CD collection, whilst vast and conveniently encoded onto my iPod in my car offers entertainment for only so long and is tricky to navigate and stay on the road at the same time.

I’ve been a train commuter into central London for a long time rather than a car commuter and I have come to value this time (if not personal space) for reading books/websites and generally educateralizing myself (yes, I made that word up) – this is kind of tricky in the car as reading a book whilst driving is frowned upon, and to be honest my hands are already tied up with my mobile phone, coffee and … (I jest, seriously road safety is important).

So, in an effort to make more use of this “dead” time; I started investigating podcasts.

I discovered Greg Ferro’s Packetpushers (Etherealmind.com) Podcast via his blog and from bumping into him at VMworld, I’ve always known enough to be dangerous with networking stuff but this is really useful as I (and a lot of my colleagues) live in a server/software centric world, it’s so easy to forget that networking is more than just a bit of copper and some IP addresses (although let’s not forget that cuts both ways and both sides are guilty of that!); It’s really useful to get the network-side perspective of the industry at a good from the trenches technical level to expand your knowledge, plus it’s a nice change of pace from day-day work – I’ve been listening to the IPv6 stuff recently and it’s very interesting.

Other good podcasts I regularly listen to (and occasionally participate in) is the VMware communites roundtable podcast; although as Mr Troyer says himself sometimes it’s better experienced live, as the real-time chat that accompanies the dial-in show is quite entertaining.]

Check them out

vTARDIS Cloud

Following on from my recent VMworld Europe user award I have mentioned that I’ve been working on a scaled out version of the vTARDIS, this post will act as the index for this project, there is a lot of ground to cover in terms of it’s configuration.

Disclosure/Disclaimer – I am a VMware employee, this project is not an official VMware effort, project, fling or even a thing – it’s my private-time work, documented for the community

Very little of this is an officially supported configuration, particularly the use of nested ESX – to re-iterate, this is not a VMware supported, recommended or blessed configuration – but it works well enough for my own needs – your mileage may vary and no warranty is granted, expressly or otherwise.

This is not a solution for production use, it’s suitable for lab/study work and actual performance is limited by the laws of physics

If you run into difficulties with any of this please feel free to drop me a line via the comments section of this post, however I do have a full-time day job at VMware, I’ll help where I am able.

What is vTARDIS? – see this post for details of the original vTARDIS project

What is vTARDIS.cloud?

A small, low cost physical infrastructure which is capable of supporting several multi-node ESX clusters. It provides an infrastructure representative of enterprise-grade vSphere/vCD deployments through heavy over-subscription of physical hardware as well as providing “production” home services like media streaming, data storage, DNS, DHCP etc.

Why?

My original home lab has been scaled out to support my new position at VMware and my VCDX/VCAP studies, a core part of my work is VMware vCloud Director (herein refered to as vCD) so my lab reflects that. Additionally my wife is trying to continue her IT studies, so it’s helpful to have a self-service portal for building out virtual machines for learning.

You very rarely have a large number of ESX hosts and shared storage to experiment with, testing scripts, rebuilding hosts, changing configurations. This provides a representation of a large vSphere/vCD deployment so you can carry out such work to support studies, or pre-production work.

What does it look like? – photo

The vTARDIS.cloud lives in my geek-cabin which is my home office (more info on that here) and now takes up most of a full rack.

image

What does it look like? – High-level architecture

the following diagram illustrates the layout of the vTARDIS.cloud

image

The key configurations and components of the design which I will post further details on are as follows (+more to follow);

  • Stateless ESXi deployment – Using autodeploy VM to PXE boot and configure large numbers of {virtual} ESXi hosts
  • Script to deploy large numbers of VMs and create DHCP reservations
  • Using the Distributed Virtual Switch with nested ESX – share a single dvSwitch between physical and nested ESX hosts (complicated virtual wiring!)
  • Remote Access to your home lab with a virtual appliance
  • vMotion between nested VMs
  • vMotion between nested ESX and physical hosts
  • Configuring the Cisco 3500 XL switch with VLAN, trunk ports for ESX
  • HA Layer 3 routing for the lab using Vyatta virtual appliance and FT
  • Using Distributed Power Management (DPM) with your home lab
  • Enabling Self-service with vCD
  • Backup on a budget

How much did you spend?

I cannot say, as my wife will probably kill me Smile I’ve acquired most of the hardware over the years or from eBay/factory outlet stores so it’s been a gradual expansion rather than an upfront cost. But still, it’s all been out of my own pocket – there are no sponsors or generous donations of kit (if you are reading this and would like to donate some equipment, read the disclaimer at the start and if you’d still like to talk drop me a line)

Item Approx Cost (£GBP) Status
Cisco 3500 XL 100mb switch (48 ports) £100 (eBay) in-use, VLAN-trunks from ESX hosts and office workstations connectivity
Netgear GS487T 48 port gigabit switch £100 (eBay) Spares (decent switch but too noisy for use in office)
Linksys SLM2008 8 port gigabit switch £90 (Amazon) vMotion/vStorage networks
Iomega IX4-200d 8Tb NAS in RAID5 configuration £1,000 (online, ouch!) in-use, critical, like it a lot but v.expensive
Multiple USB2 drives 500Gb-1Tb varies in-use plugged into IX4 for backup
2 x HP ML110G4 Intel Xeon, 8Gb £200 each in-use (management cluster)
special online deals, now defunct Sad smile
3 x HP ML115 G5 AMD Quad Core, 8Gb RAM, dual port GbE Intel NIC £2-300 for each serverwith RAM (varying deals)
80-100 for 8Gb RAM
40 for dual port Intel GbE NIC (job-lot on eBay)
in-use (resource cluster)
now EoL – hopefully they won’t die!
42U Rack (no-brand) free holding up servers Smile
1 x HP D530 SFF Desktop PC, 4Gb RAM, 500Gb SATA £90 (eBay) in-reserve, was ESX 3.5 host (non x64 CPU)
HP TFT 15” rack mount monitor free from skip at customer in-use
HP 4 port PS/2 KVM free from skip at customer in-use
128Gb Kingston SSD £200 (Amazon) UberVSA virtual SAN storage (was in original vTARDIS project; since cannibalised)
64Gb Transcend SSD £100 (Amazon – a while ago) UberVSA virtual SAN storage
Compaq ML570 G1, quad Xeon CPU, 12Gb RAM, external disk array multiple 18Gb SCSI disks, SmartArray £400 eBay (4 or 5 years ago) retired, non-x64 and too power hungry (was power-sucken-cluster)
[open to offers!]
spider refuge
Compaq DL360 G1, single Xeon CPU, 4Gb RAM, 2 x 18Gb HDD £500 eBay (a long time ago) retired, non-x64 and too power hungry (was power-sucken-cluster)
[open to offers!]
spider refuge
Compaq DL320 G1 – unknown spec free, from customer refresh a long time ago retired and faulty, spider refuge
Sun Netra free from a customer refresh a long time ago retired, was old firewall, spider refuge
Compaq 2 drive DLT tape-loader free from a customer refresh a long time ago retired, and probably faulty by now, spider refuge

How much does it cost to run?

This uses approx 600w of power 24/7 – it’s not that cheap here in the UK, I estimate about £6-700 per year, DPM certainly helps to reduce the power consumption of the resource cluster when it’s less-busy, although as a side-benefit the vTARDIS acts as passive heating for my garden office during the winter, that’s “green”, right?

Domestic Schedule Bliss? There’s an iPad app for that in the Cloud

 

Anyone reading this that has a partner and family will probably understand my pain… My wife & I have a pretty busy schedule; work commitments and travel that are part of any modern IT consultant job, 2 young children and their highly complex schedule of social and school, nursery, classes add to that a widely distributed network of family and friends and their social events/weekend visiting and it gets pretty complex to keep track of.

As I’ve written about before my Outlook/BES Blackberry calendar is de-facto to me; I just don’t have the mental bandwidth to track everything and the bit of my brain that deals with remembering dates over a week away is either missing or faulty and multiple personal/work calendars just mess with my head.

My wife has performed a heroic task of maintaining a paper family calendar for years, but that forced me to also maintain it manually and often things were forgotten which has led to much confusion and mis-scheduling, especially when I have been away and there have been changes to my schedule.

image

We work in IT right? there has to be a technical solution to this pain? well yes, there is; however my wife found using a laptop or small screened mobile phone to manage a shared calendar too difficult whilst juggling 2 small children and always fell back to the paper calendar and lived with it’s limitations.

So, when an iPad was introduced and quickly adopted by everyone in the household as “good-enough” for everything from quick browsing of the web to TV guides, iPlayer, recipes, games I spotted an opportunity.

I was a bit skeptical about the iPad at first and didn’t think it would be much more than a nice toy, however it was used so much by all of the family because;

  • It switched on instantly (unlike a laptop)
  • the battery lasts literally forever so you don’t have to be tethered to power (unlike a laptop or iPhone)
  • Has a usable sized screen that you could read things without having to scroll/pan about (unlike a smartphone); and having 2 young kids it’s wipe-clean and reasonably robust (parents will understand Smile).

So with some fiddling we ended up at this solution

image

  • Google calendar acting as the central “hub” reference – accessible via a web browser from anywhere and with a good API that is used by sync applications across multiple platforms.
  • My Exchange-hosted calendar is automatically synced to my fat-Outlook client and to my Blackberry via the BES.
  • the Google calendar sync application runs in the background on my Blackberry and syncs with the shared Google Calendar.
  • The iPad application CalenGoo syncs with the Google Calendar so the calendar can always be viewed and edited from the iPad in a convenient home-use form-factor.
  • My wife’s new smartphone (which will either be a BIS Blackberry or iPhone will be able to sync with the Google calendar “hub” giving her a mobile and up to date editable copy of the calendar.

For us it works pretty well, if you have a similar problem I’d suggest you investigate it, one word of warning the BB/Gmail sync app won’t sync historical appointments from your calendar so be careful if you try to get around this with a manual import; you may end up with duplicate calendar entries (at least I did, and had to de-dupe them).

There are also Google calendar sync plug-ins for Outlook and other mail clients, but I was happy enough with doing it via my BB as that nearly always has a network connection and is kept constantly in-sync with my work calendar.

With all that sync’ing there is a latency of about an hour for changes to get replicated end-end which is more than enough for our family needs.

In terms of security my family or work schedule isn’t particularly sensitive but it’s tied down to usernames/passwords where relevant and transmission is over SSL, entries from our shared calendar get synced into my corporate calendar marked as ‘private’ – but there could be better support for a more granular model both ways here, Google calendar seems to have this concept but it doesn’t federate into into the Exchange/BB world {yet}.