Virtualization, Cloud, Infrastructure and all that stuff in-between
My ramblings on the stuff that holds it all together
Category Archives: CloudCamp
Silent Data Corruption in the Cloud and building in Data Integrity
Posted by on January 20, 2011
I was passed a link to a very interesting article on-line about silent data corruption on very large data sets, where corruption creeps undetected into the data read and written by an application over time.
Errors are common in reading from all media and this would normally be trapped by storage subsystem logic and handled lower down the stack but as these increase in complexity and the data they store vastly increases in scale this could become a serious problem as there could be bit-errors not being trapped by disk/RAID subsystems that are passed on unknown to the requesting application as a result of firmware bugs or faulty hardware – typically these bugs manifest themselves in a random manner or by edge-case users with unorthodox demands.
All hardware has a error/transaction rate – in systems up until now this hasn’t really been too much of a practical concern as you run a low chance of hitting one, but – as storage quantities increase into multiple Tb of data this chance increases dramatically. A quick scan round my home office tallys about 16Tb of on-line SATA storage, by the article’s extrapolation on numbers this could mean I have 48 corrupt files already.
This corruption is likely to be single-bit in nature and maybe it’s not important for certain file formats – but you can’t be sure, I can think of several file formats where flipping a single bit renders them unreadable in the relevant application.
Thinking slightly wider, if you are the end-user “victim” of some undetected bit-flipping what recourse do you have when that 1 flips to a 0 to say your life insurance policy doesn’t cover that illness you have just found you have – “computer says no”?
This isn’t exclusively a “cloud problem” it applies to any enterprise storing a significant amount of data without any application level logic checks, but it is compounded in the cloud world where it’s all about a centralised storage of data, applications and code, multi-tenanted and highly consolidated, possibly de-duplicated and compressed where possible.
In a market where cost/Gb is likely to be king providers will be looking to keep storage costs low, using cheap-er disk systems – but making multiple copies of data for resilience (note, resilience is different from integrity) – this could introduce further silent bit corruptions that are propagated across multiple instances as well as increasing the risk of exposure to a single-bit error due to the increased number of transactions involved.
In my view, storage hardware and software already does a good job of detecting and resolving these issues and will scale the risks/ratios with volumes stored. But, if you are building cloud applications maybe it’s time to consider a check summing method when storing/fetching data from your cloud data stores to be sure – that way you have a platform (and provider)-independent method of providing data integrity for your data.
Any such check summing will carry a performance penalty, but that’s the beauty of cloud – scale on demand, maybe PaaS providers will start to offer a web-service to offload data check summing in future?
Check summing is an approach for data reliability, rather than security but at a talk I saw at a Cloudcamp last year; a group were suggesting building DB field-level encryption into your cloud application, rather than relying on infrastructure to protect your data by physical and logical security or disk or RDBMS-level encryption (as I see several vendors are touting) build it into your application and only ever store encrypted assets there – then even if your provider is compromised all they hold (or leak) is already encrypted database contents – you as the end-user still retain full control of the keys and controls.
Combine this approach with data reliability methods and you have a good approach for data integrity in the cloud.
The Big Cloud Debate @CloudCamp London
Posted by on February 24, 2010
I noted with interest that the next CloudCamp London has been announced for Thursday 11th March, the last couple that I attended were pretty similar and other than some useful networking there was nothing really new/different that jumped out at me; so I wasn’t sure I would go to the next one as it seemed to have run out of steam a little until I noticed this on the agenda for March;
"The Big Cloud Debate " : A presidential style 4 way debate pitching 4 divergent views and approaches to cloud computing against each other. The speakers are:
- Matt Deacon – Microsoft
- Simon Wardley – Canonical
- Rod Johnston – VMware
- Chris Richardson Thoughtworks
The reason this is of interest to me? Rod Johnston came to VMware via the Springsource acquisition, I assume Matt Deacon will be discussing Microsoft’s Azure platform. plus CloudCamp London mainstay Simon Wardley from Cannonical I assume taking the EC2/Ubuntu angle and Chris Richardson (who I think has this blog; ex-springsource?
Should be an interesting debate, I think it would also be good to have a similar format debate with this panel (or the companies they represent) around the Infrastructure as a Service (IaaS) / private vs. public cloud angle – CloudCamp always has a software focus, but I think the cloud infrastructure debate has a lot of scope and a good potential audience
Registration doesn’t seem to be open to the general public yet, but I’m sure it will soon – maybe see you there.
Cloud Camp London (21st Jan 2010) now open for registrations
Posted by on December 16, 2009
You can register for the next Cloudcamp London on the 21st Jan 2010 at this link
If you don’t know what cloudcamp is about – check out one of my previous posts if you are available I recommend it.
Cloudcamp London July 2009
Posted by on July 9, 2009
Cloudcamp London is just winding down, it seems to be a lot smaller than the last London Cloudcamp and it’s in a smaller but much nicer venue (provided by Microsoft)
There were a set of lightning talks on various cloud topics, mostly on security in the cloud.
Some interesting thoughts from HP labs on on data obfuscation software as a way to better protect data in the cloud, some client side software that can encrypt and decrypt data from a service provider, almost it’s own man in the middle to translate data to/from the cloud.
Never store your data in the clear in the cloud but don’t rely on the cloud to do the encryption, it’s transparent to your apps and is prob he most agnostic approach too, an Amazon or Microsoft DB doesn’t care if the First Name field says “bob” or “"LpZ”
There was an interesting panel discussion that spent some time on the definition of cloud computing, there was a lot of bagging of the concept of private clouds not being “proper” cloud computing, it’s “just” virtualization.
That’s a favorite argument of mine and Joe Bagley from Quest software made a point that summarizes it better than my previous statements; “virtualization is just a technology, cloud is a business model” – to expand that further that business model can be applied equally to public facing services and internally facing services (inter-departmental, chargeback etc.) – it’s not all about internet scale gargantuan operations
Some breakout sessions from Cisco on IP NGN – Next Generation Network, applying tagging technology to ensure network state moves around the datacentre and globe with virtual server instances. “the network is not just the pipe”.
Rightscale were up next; SaaS gives you limited control over what you can do with your solution – PaaS/IaaS – total flexibility, rightscale add automation and management
Predictions over infrastructure sizing cost money = over-provisioning; opportunity cost, which is why cloud is so appealing to start-ups, lower barrier of entry… same principal can apply to the corporate world – cost of failure is smaller for off the wall ideas.
some examples of very peaky demand that they helped deal with on EC2;
- Animoto EC2 example 8 mins CPU for 1 min video, 25k sign ups per hour peak 4.7k EC2 instances
- Oscars Starcut, scheduling feature to bring up instances on a schedule
- Beijing olympics
- Eli Lilly — computational biology, grid in the cloud taking advantage of massive parallelism
CohesiveFT were up next (The discussion topics didn’t grab my interest this time round so I stuck with the vendor track as I needed to get a bit of market research in.
- Elasticserver.com – customize virtual (cloud) server build = likened to Dell website process for building a physical server
- Software factory, pick components – open source things like mySql, python etc. upload your own components, multiple OSes (open source)
- build, licence, market sell ISV solutions via portals
- output as EC2 or elastic hosts cloud
- or download VM in VMware/Xen/Parallels VM appliance – very cool
- community edition = free, personal or professional (paid-for
- they embed management hooks in the appliance back to elastic server service console
- On-premise versions going into beta soon, deploy as a VM appliance – nice
enStratus were up last David Bagley is an IaaS management offering
- managing infrastructure, security, reliability
- Interesting point made by a member of the audience; Amazon (+other IaaS)costs flex up and down with demand, managed services don’t map that way with most MSP, support is a fixed cost, no pay as you go
- Reason being you need people sitting there, a larger MSP should be able to do this as they have more diversity better risk/workload spread but costs don’t reflect that or have high barriers to entry.
All round a good event, bit smaller and less vocal audience than the last one I attended, if you get the chance I would definitely recommend checking out an event near you.
CloudCamp London – March 12th
Posted by on January 19, 2009
Registration is open for the next CloudCamp event in London on March 12th, I’ve attended once before and I like the format – a set of a ‘lightning talks’ around cloud topics, some networking and some detailed breakout sessions.
It’s an excellent and informal event to discuss cloud developments with your peers as well as find out what’s going on in the industry – best of all it’s free!
Information here and registration link here; if you’re UK/London based I’d recommend it, drop me a line via the comments if you are planning to attend or would like to meet up.
if you’re not UK-based; rest of the world events are here
Virtualization, Cloud, Infrastructure and all that stuff in-between 