Virtualization, Cloud, Infrastructure and all that stuff in-between

My ramblings on the stuff that holds it all together

Category Archives: Storage

Hardware is Hard, Software is Easy. is 2011 the year of the VSA?


I have done a lot of lab-work with Virtual Storage Appliances, mainly because proper shared storage is hard to come-by for lab-time so I’ve used the following for the last few years running inside Virtual Machines



Celerra VSA Uber

HP LeftHand


The more vendors that release software versions of their kit or emulators are high on my list of things to watch as IMHO it shows they are looking-ahead

Traditional storage vendors have made a very good living in the last decade selling custom, high-performance silicon – but this comes at a cost – designing custom ASICs and code takes time, because it involves high-tech fabrication technologies, even if these are outsourced it’s very expensive and time-consuming.

It’s also harder to “turn the ship” if the market moves as a vendor has significant resources committed to product development.

Mainframes have also maintained a similar position and have seen their market share eroded by commodity x86 hardware that combined with clever software delivers the same solutions with less hardware-vendor lock-in and typically a lower cost.

Software is easy – well, relatively easy to change when compared to hardware so R&D cycles can be shorter, more agile and respond quicker to market changes.

Changes/upgrades to custom chips have development lifecycles in multiple years, and once a chip is burnt/fabricated and shipped to the masses it’s harder to make changes if a problem is found – x86 builds up on a well-used and field-proven architecture typically adopting a scale-out architecture over standardised interconnects (Infiniband/Ethernet) to achieve higher performance – why re-invent the wheel?

There will always be edge-cases where ultra low-latency interconnects can only be provided over on-die CPU traces – but for general compute, network, storage – but as x86 and it’s ancillary interconnect technologies march ever faster, can equivalent functionality not be achieved using clever software on common hardware rather than raw physics and men in white-coats?

As this cycle continues – can storage vendors continue to make those margins, respond to customer requirements and keep ahead of the competition when they are tied to a custom silicon architecture – is it more advantageous to move to a commodity platform to deliver their solutions?

Using “clever” software like a hypervisor to abstract a commodity x86 hardware architecture means you can push storage functions like snapshots, cloning, replication higher up the stack and make them less specific to hardware vendor X’s track/cylinder or backplane protocol Y

Building in x86 also means you can be selective about how you deploy – on bare hardware, or with a hypervisor like ESXi – both use-cases are equally valid and the cost to change between the two is minimal (in development terms)

EMC are already committed to an x86 scale-out architecture for their platforms for this reason and even if the badge on the outside says EMC it’s just commodity kit with clever software, rather than custom firmware running on custom chips and I expect all the competition are considering if being a niche edge-case player or a high-performance general storage player is a better business play.

The Open-source community also have some excellent projects in this space which are being spun out into commercial products, traditional storage vendors beware!

Virtual Storage Appliances (VSA) are the next logical step in de-coupling storage services from hardware.

Disclosure: I work for VMware, of whom EMC are a majority shareholder – however this isn’t an advert – it’s my opinion and experience.

Home Labbers beware of using Western Digital SATA HDDs with a RAID Controller

I recently came across a post on my favourite car forum ( asking about the best home NAS solution – original link here.

What I found interesting was a link to a page on the Western Digital support site stating that desktop versions of their hard drives should not be used in a RAID configuration as it could result in the drive being marked as failed.

Now, this I far from the best written or comprehensive technote I have ever read however I wasn’t aware of this limitation, it appears that desktop (read: cheap) versions of their drives have a different data recovery mechanism to enterprise (read: more expensive) drives that could result in an entire drive being marked as bad in a hardware RAID array – the technote is here and pasted below;

What is the difference between Desktop edition and RAID (Enterprise) edition hard drives?
Answer ID 1397   |    Published 11/10/2005 08:03 AM   |    Updated 01/28/2011 10:00 AM

Western Digital manufactures desktop edition hard drives and RAID Edition hard drives. Each type of hard drive is designed to work specifically as a stand-alone drive, or in a multi-drive RAID environment.

If you install and use a desktop edition hard drive connected to a RAID controller, the drive may not work correctly. This is caused by the normal error recovery procedure that a desktop edition hard drive uses.

Note: There are a few cases where the manufacturer of the RAID controller have designed their drives to work with specific model Desktop drives. If this is the case you would need to contact the manufacturer of that controller for any support on that drive while it is used in a RAID environment.

When an error is found on a desktop edition hard drive, the drive will enter into a deep recovery cycle to attempt to repair the error, recover the data from the problematic area, and then reallocate a dedicated area to replace the problematic area. This process can take up to 2 minutes depending on the severity of the issue. Most RAID controllers allow a very short amount of time for a hard drive to recover from an error. If a hard drive takes too long to complete this process, the drive will be dropped from the RAID array. Most RAID controllers allow from 7 to 15 seconds for error recovery before dropping a hard drive from an array. Western Digital does not recommend installing desktop edition hard drives in an enterprise environment (on a RAID controller).

Western Digital RAID edition hard drives have a feature called TLER (Time Limited Error Recovery) which stops the hard drive from entering into a deep recovery cycle. The hard drive will only spend 7 seconds to attempt to recover. This means that the hard drive will not be dropped from a RAID array. While TLER is designed for RAID environments, a drive with TLER enabled will work with no performance decrease when used in non-RAID environments.

There are even reports of people saying WD had refused warranty claims because they discovered their drives had been used in such a way, which isn’t nice.

This is an important consideration if you are looking to build or are using a NAS for your home lab like a Synology or QNap with WD HDDs or maybe this  even extends to a software NAS solution like freeNAS, OpenFiler or Nexentastor

It’s also unclear if this is just a Western-Digital specific issue or exists with other drive manufacturers.

Maybe someone with deeper knowledge can offer some insight in the comments, but I thought I would bring it to the attention of the community – these are the sort of issues are like the ones I was talking about in this post but, as with everything in life – you get what you pay for!

New Storage Benchmarking Tool from VMware Labs


To quote the site..

IOBlazer is a multi-platform storage stack micro-benchmark. IOBlazer runs on Linux, Windows and OSX and it is capable of generating a highly customizable workload. Parameters like IO size and pattern, burstiness (number of outstanding IOs), burst interarrival time, read vs. write mix, buffered vs. direct IO, etc., can be configured independently. IOBlazer is also capable of playing back VSCSI traces captured using vscsiStats. The performance metrics reported are throughput (in terms of both IOPS and bytes/s) and IO latency.

IOBlazer evolved from a minimalist MS SQL Server emulator which focused solely on the IO component of said workload. The original tool had limited capabilities as it was able to generate a very specific workload based on the MS SQL Server IO model (Asynchronous, Un-buffered, Gather/Scatter). IOBlazer has now a far more generic IO model, but two limitations still remain:

The alignment of memory accesses on 4 KB boundaries (i.e., a memory page)
The alignment of disk accesses on 512 B boundaries (i.e., a disk sector).
Both limitations are required by the gather/scatter and un-buffered IO models.

A very useful new feature is the capability to playback VSCSI traces captured on VMware ESX through the vscsiStats utility. This allows IOBlazer to generate a synthetic workload absolutely identical to the disk activity of a Virtual Machine, ensuring 100% experiment repeatability.

You can download it and find more information here – I’m going to try and use this in my upcoming NexentaStor NAS project.