Virtualization, Cloud, Infrastructure and all that stuff in-between
My ramblings on the stuff that holds it all together
Cannot Set Static IP in OpenFiler When Running as a VM
Posted by on February 4, 2009
As a result of a power outage last week my home lab needed a reboot as my 2 x ESX D530 boxes didn’t have auto-power on setting set in BIOS, so I dutifully braved the snow to get to the garage and power them on manually.
However nothing came back online.. ESX started but my VMs didn’t auto-restart as it couldn’t find them.
The run up to xmas was a busy month and I had vague recollections of being in the midst of using storage vMotion to move all my VMs away from local storage to an OpenFiler VM in preparation for some testing.
However, in my rush to get things working the OpenFiler box didn’t have a static IP address set and was using DHCP (see where this is going…?)
So my domain controller/DNS/DHCP and Virtual Centre server were stored on the OpenFiler VM which my ESX box was running and accessed over iSCSI. As such when ESX started it couldn’t locate the iSCSI volume hosting the VM and couldn’t start anything.
OpenFiler couldn’t start its web admin GUI if it couldn’t get an IP address, nor would it mount the shared volumes.
Once I’d figured out what was going on, it was simple enough to get things going again;
- Temporary DHCP scope on my router,
- IPCONFIG/ RENEW to get a temporary DHCP address on my laptop
- VI client directly to ESX box rather than VC and reboot the OpenFiler VM
- Web browser to OpenFiler appliance on temporary DHCP addresss
However at this point I would have expected to be able to set a static IP address and resolve the issue for the future, however I couldn’t see any NICs in the OpenFiler config screen (see screenshot below)
I thought this was a bit odd, and maybe I was looking in the wrong part of the UI, but sure enough it was the correct place.
I tried updating it to the most recent software releases via the handy system update feature, which completed ok (no reboot required – beat that Windows Storage Server! :)) but still no NICs showing up, even after a couple of reboots to be absolutely sure.
Then, I stumbled across this thread and it seems this may be a bug (tracker here) following Jason’s suggestion I used the nano text editor via the VI remote console to edit the /opt/openfiler/var/www/includes/network.inc file on the OpenFiler VM as follows;
Before:
After:
I then refreshed the system tab in my browser session and the NICs show up;
note as part of my initial troubleshooting I added a 2nd virtual NIC to the VM, but the principal should apply regardless.
And I can now set a static IP etc.
I had to reboot my ESX host to get all my VM’s back from being inaccessible, I’m sure there is a cleverer way to do that, but in my case I wanted to test that the start-up procedure worked as expected now that I’ve set a static IP and re-jigged the start-up sequence so that OpenFiler starts before any other VMs that are dependent on it for their storage.
UK Mostly Grinds to a Halt
Posted by on February 2, 2009
Lots of snow here in the UK, well 12” is lots of snow by UK standards anyway, more good travel {or lack, thereof} info here from Annie Mole
South-eastern Railway’s website has ground to a halt (not exactly unusual, even on a good day!).
For me out here in Kent, all of my local train services are all suspended, as are the buses and at least the BBC website is a bit more reliable.
All of the roads are blocked or have delays so serious as to not make it worth anyone’s while to attempt it.
My daughter’s nursery is permanently engaged on the phone so I’m suspecting that means closed – they definitely need a better push based notification/alerting system – maybe I’ll suggest I write one for them!
Welcome to 2009 :), thank goodness for technology – all of my planned meetings are now conference calls, email/IM keep the lines of communication open and the web means I’m a better informed traveller so I can make an informed decision before attempting anything.
Even 10 years ago when I can last remember this happening down south things were pretty different, the tech hasn’t changed that significantly but what has changed is the adoption and acceptance, broadband penetration and general acceptance that previously “frivolous” activities like IM the web and email are actually critical business tools these days, particularly when conditions mean it’s less efficient to get about and do things face-face.
Could Skynet be a Cloud Application, and Should I be Scared?
Posted by on January 29, 2009
Has the cloud been sent from the future to kill you?
It’s Friday… so time for something completely different, Smugmug have already built skynet here on EC2 which decided it wanted more power… and made a semi-autonomous decision to scale itself out to mammoth proportions, if you weren’t as diligent as they are and maybe don’t pay close attention maybe your EC2 bills would bankrupt you by the time you see the invoice, assuming no credit-control limit… then you’d be out on the street, maybe loose your job, etc.
Or what if your EC2 instances picked up some kind of malware that is EC2 aware and it suddenly started to become a botnet, harvesting people’s credit card details to open up new EC2 accounts and spawn more parallel instances of itself, or spread to other cloud providers or opened up online loans, credit cards and gambling accounts, trade accounts, share dealing accounts – which in turn bankrupted other people. what if it made a coordinated (or maliciously intended), distributed on-line run on a particular stock, sparking panic buying, which in turn causes credit crunch 2.0 and brought about the end of humanity? oh, wait… that’s going on now.. maybe we know what caused it 🙂
What then if EC2 did provide IP connectivity back to your own networks and it started stealing and disseminating your internal commercial data (or entire virtual servers..), what if you ignore all that security best-practice stuff and start plugging in your office HVAC system into the LAN (lots of it going on these days) and it decides that it should brute-force access into or DoS your building UPS, resulting in overloads and fires.
Maybe virtualization is that chip they found, and VMware are really Cyberdyne systems?
Ok, bit off the wall but this thought came to me on the train home today…I’ve had a nasty dose of the flu, so maybe that paracetemol was a bit stronger than it said on the box 🙂
Best to remember those firewalls, sandboxes and policies are there for a reason.. and people’s natural impatience to embrace new things can always compromise that, especially in today’s world of instant/on-demand gratification… why do I have to wait 7 days to sign my paper! credit card application form… those check-points are there for a reason, the same security principals that apply to the physical world also apply to the cloud and virtualization – just because you can do something, doesn’t mean it’s the right thing to do, you need to assess risk and mitigate accordingly*
Normal service will be resumed shortly..
*Although I would expect there would be a few eyebrows raised if your corporate risk register contained an essay on how to mitigate against a horde of cyborgs controlled by your HR department trying to exterminate you (oh, wait..:))
Microsoft Virtualization User Group Meeting (UK)
Posted by on January 29, 2009
I’ll be attending this user group event this evening in London; if you’re local and interested then I believe it’s never too late to register.
If you’re not local then you can view the webcast (details below) online
Looks to be some interesting content, and always good to speak to customers who have done it in real-life, the Microsoft virtualization user group UK site is here
Next In-Person Meeting
Microsoft Virtualisation User Group – January 2009 Meeting
Location:
Microsoft London (Cardinal Place)
http://download.microsoft.com/documents/uk/about/downloads/victoria_map.pdf
Date & Time:
Thursday 29th January 2009
18:00 – 21:30
Agenda:
18:00 – 18:15
Arrivals
18:15 – 18:45
Simon Cleland (Unisys) & Colin Power (Slough Borough Council)
Case study: Hyper-V RDP deployment at Slough Borough Council
18:45 – 19:30
Aaron Parker (TFL)
Application virtualisation – what is App-V?
Benefits of App-V & a look inside an enterprise implementation
19:30 – 20:00
Food
20:00 – 21:15
Justin Zarb (Microsoft)
Application virtualisation – in-depth look at App-V architecture
21:15 – 21:30
Q/A and wrap up
Registrations:
Register at the forums for this event here
Or email meeting@mvug.co.uk
Live Meeting:
Click Here
No need for a meeting ID
Room opens at 5.30pm – meeting at 6.30pm
Zeus ZXTM Virtual Appliance – Quick Look
Posted by on January 27, 2009
Zeus technologies from Cambridge here in the UK are one of my favourite tech companies, they are small and agile and produce an excellent bit of traffic management software called the Zeus Extensible Traffic Manager.
It’s an IP traffic manager on serious, highly-available steroids and has a very impressive list of features, I have seem many of these deployed in customer environments for some of the largest media websites in the UK (and even the world, in some instances).
For me, the beauty of them is their simplicity, it’s “just” software; not a hardware appliance like F5/Cisco etc., or mysterious black box that hums away and costs £100000’s every time you need to replace one.
It’s hardware agnostic as it runs on Linux, Solaris or FreeBSD on physical servers or a VM, or as a pre-packaged VM appliance. This really plays well in the internal/external cloud space as you want to leverage cheap, commodity x86/x64 (and even SPARC) hardware and virtualization rather than be bound to hardware that is difficult to move and redeploy… they really “get” this kind of stuff.
There is a cool article here about using the ZXTM to talk to VMware virtual center and dynamically provision web servers as it senses demand increasing, see how powerful its scripting language & Java support is?
In this quick post I’ll show you just how easy it is to set one up, by setting one up 🙂 and a quick look around the UI.
In this instance I’ve downloaded the virtual appliance from here and am running it on VMware Workstation 6.5, it also runs happily on ESX.
It defaults to a static IP address and is administered via a web browser so I’ve got one of those here;
The obligatory EULA screen, yes of course I read every word.. honest.
Configuring the IP address, note the 802.3ad support for trunking… nice
DNS Server configuration
Date & Time
Admin password, note browser and SSH access
Licence key – we’ll skip this now and upload later.
Summary screen
Basic setup complete and all done in minutes, and ready to go.
Login screen
Uploading the evaluation licence
All Done! now you just need to add your services like web servers, farms, caching etc. Almost anything it doesn’t have a button for out of the box can be implemented in its TrafficScript language. For example; making it talk to Twitter or text you – the KnowledgeHub has lots of example code and how-to’s and the FAQ is here.
To wet your appetite I’ve put some quick screen captures of the ZXTM web interface below.
ZXTM’s can be clustered together for high-availability (no special hardware required, just out of the box commodity servers and ethernet), and configuration is simple to backup and restore (as with traditional physical appliances) they scale up to multi-gigabit loads and are well suited to scaling vertically for large demand.
You can download evaluation versions of all of this from here, and they have just launched their own blog here – download it and have a play, it’s so simple.
One of my colleagues is probably Zeus’s biggest fan, his blog is here and I’m sure this will give him a nudge to blog some of the very cool stuff he’s been doing with the ZXTM.
[disclaimer] This post isn’t an advert it’s just me trying to share my experiences… whilst my employer are official Zeus partners, this is purely on the basis of merit rather than any entirely commercial grounds; we’ve deployed the ZXTM in some of the most popular TV/video on demand platforms in the UK and both the technology and support services behind the product are outstanding.. which is why I choose them time & time again and they are a core part of my internal cloud reference architecture. They’re also small enough that they are genuinely interested in what you want in the product and how you are using it.
Workload Portability: Ultimate Cloud Edition
Posted by on January 23, 2009
I like the PlateSpin range of products a lot, it really does let you take an OS instance + app stack (workload) and move it between different physical machines, hypervisors etc. in a low impact way – if you’ve not come across it before – read this post for more info I see this portability as one of the key infrastructure components if you are looking to build or manage your own internal cloud infrastructures.
This isn’t possible at present, but put your architect hat on and imagine if you could plug PlateSpin Migrate (previously known as PlateSpin PowerConvert) tool into Amazon’s EC2 cloud, or a VMWare vCloud based farm – then you could do whatever you like with your Windows and Linux servers.
By design AWS and vCloud are both supposed to be automatable with web services and APIs to control machine provisioning and control etc. EC2 seems to have all of this now (API docs and example) and vCloud is coming along. (more real details at VMworld I’m guessing).
Moving services between on and off-premise cloud infrastructures is a key concept of vCloud; but I’m guessing this will only be between vCloud based infrastructures, what if you wanted to take advantage of the capacity and scale/commodity pricing from big providers like EC2 (which is Xen based under the hood) to offload some of your internal services – to my mind, there are a couple of scenarios here that PlateSpin could fulfil;
- Disaster Recovery – using the cloud (EC2 or other) for DR capacity; pay per use – use PlateSpin Protect to sync your machine images off to Amazon S3 and have a “panic button” that converts the S3 hosted images to running AMI’s. Brent has a similar idea here around SQL, my proposition takes this to the next level and does it from the OS up; if you did have to move over to the EC2 hosted DR cloud, then you could use it to go back to physical hardware again once you’ve repaired/rebuild your internal infrastructure
- Data centre moves or serious maintenance – use a cloud like EC2 as “swing” capacity to run services whilst you pick up your DC hardware and move it somewhere else (rather than a kit refresh).
- Test & Development; the ability to sandbox new apps in EC2 could be attractive to some organisations where corporate policies hinder or prevent this type of innovation taking place in-house; What if you could do this externally then just bring the machine instances back in-house to put into internal production use (I’ve seen this happening at several customers) – of course IT security teams would probably not be to happy about it.
- Short-term Expansion Capacity; if you experience an occasional surge of demand or load for an internal service. For example; if you have an internal application that you know will get really hit for a promotion or project then you could clone instances of the relevant web/application servers off to EC2 and use some kind of very clever load balancing tech to selectively hand off load to EC2 hosted instances when internal servers start getting saturated – or vice-versa.
Maybe even if PlateSpin were to position their product as a web service itself with downloadable agents – a connector/conversion hub between clouds – now that’s an interesting proposition.
Hopefully this diagram explains some of this idea visually
Issues at present:
- PlateSpin doesn’t have an interface to EC2 (consider this my feature request :))
- There is no secure connectivity back to corp HQ – this is something that as far as I can see AWS has an issue with – out of the box there is no way to have say an IPSec VPN or dedicated private subnet managed and provided by EC2, complicated networking scenarios don’t seem to be possible – you could build your own using software based routers and firewalls on EC2 hosted server instances but this is host based – would be good if EC2 add this sort of service to the platform in future – that would definitely be a killer feature as far as I’m concerned – AWS team, consider this my feature request :))
- VM Persistence is something of an issue with EC2 and I don’t think the EC2 model currently deals with it; with EC2 you pay whilst an instance is running, if you terminate it; i.e switch it off, it’s gone – the data (and that includes OS/app configurations) that you build into the instance are lost. there is no way to archive/suspend/freeze an instance to S3 and “spin it up” as required – I’m guessing this would be feasible for Amazon to build into EC2/S3 – you pay per GB stored on S3 so there is a cost-model for it – again this would be a killer feature for me – there are ways obviously to make your instances “vanilla” and have them auto-install relevant code and data when they are created; examples here and here but that takes a lot of work and isn’t so simple for most corporate type apps.
- You can attach an EBS (Elastic Block Storage) volume to an instance, this is persisted (as long as you keep paying for it) and you can mount it to a single host as a block disk device – but the issue remains with the actual OS instance not being persisted. if its a Windows OS, this is a particular problem as the config is all held in the registry etc. which is part of the OS itself.
- This still doesn’t get you past the concerns/issues over data ownership and cloud security, there is no magic bullet in this respect, just risk management/mitigation.
Anyways. just an idea, feel free to comment and give me your feedback..
1 Month to VMworld Europe 2009
Posted by on January 21, 2009
It’s now almost one month until the start of VMworld Europe, this is the 2nd outing for the European event which is being held in Canne, France.
If you’ve never been to one of these events and work in a technical capacity I would heartily recommend it,with the current financial climate and awful GBP/EUR exchange rate it’s harder than ever to convince your employer to send you or to find the cash yourself, if it helps – this is a previous article I wrote about this sort of event and the benefit you’ll get from it. If you’re a VCP you also qualify for a significant discount as Mike has already covered here
I’ll be blogging from the event itself, and VMware are starting to post details of sessions here, there aren’t too many non-vmware/sponsor sessions posted up yet; hopefully that will happen soon – it’s always good to have a bit of an independent perspective and both VMware and Microsoft haven’t disappointed in this type of thing before.
I’ will also be attending for partner day and will post whatever I am able to (subject to any NDA’d material) – I’ll be following the technical track.
If you have 5 mins it’s definitely worth visiting the vmworld.com site – there will be lots of material posted up during the event if you aren’t able to make it in person and there is also a good community-focused site at vmworldunderground.com with it’s own social network.
CloudCamp London – March 12th
Posted by on January 19, 2009
Registration is open for the next CloudCamp event in London on March 12th, I’ve attended once before and I like the format – a set of a ‘lightning talks’ around cloud topics, some networking and some detailed breakout sessions.
It’s an excellent and informal event to discuss cloud developments with your peers as well as find out what’s going on in the industry – best of all it’s free!
Information here and registration link here; if you’re UK/London based I’d recommend it, drop me a line via the comments if you are planning to attend or would like to meet up.
if you’re not UK-based; rest of the world events are here
Easy to Understand Overview of Amazon Web Services (AWS)
Posted by on January 16, 2009
Brent Ozar has an excellent post here explaining how Amazon Web Services (AWS) can be used in a practical sense with a worked example.
It makes it easy to understand why you could use AWS if you’ve had trouble getting your head around what this could stuff is.
VMware vExpert Award Open for Nominations
Posted by on January 15, 2009
My my, it really is vEverything now isn’t it 🙂 anyways as has been widely reported over the last 24hours VMWare have announced the details of the VMWare vExpert programme, which is a way of recognising those that have contributed to the VMWare community along similar lines to Microsoft’s MVP programme.
Details and a nomination form, should you be feeling generous towards me 🙂 or indeed any other members of the blog/forum community are here.
Closing date is 6th Feb 2009, I’m guessing that the end of Feb decision on nominations may coincide with VMWorld? if so, I’ll be there blogging about it all so check back then if you want to know who won.
Here’s a quick list of my most popular posts (by hits) over the last 12 months if it helps to remind you 🙂
Running ESX & 3i under VMware Workstation
Running ESX on cheap PC hardware
Deploying a Virtual Machine from a Template with Virtual Center 2.5
Virtualization: the key to delivering cloud based architecture NOW
Virtualization, Cloud, Infrastructure and all that stuff in-between 