My ramblings on the stuff that holds it all together
Category Archives: OpenFiler
I have had a lab/test setup at home for over 15 years now, it’s proven invaluable to keep my skills up to date and help me with study towards the various certifications I’ve had to pass for work, plus I’m a geek at heart and I love this stuff 🙂
over the years it’s grown from a BNC based 10mbit LAN running Netware 3/Win 3.x, through Netware 4/NT4, Slackware Linux and all variants of Windows 200x/RedHat.
Around 2000 I started to make heavy use of VMware Workstation to reduce the amount of hardware I had (8 PCs in various states of disrepair to 2 or 3 homebrew PCs) in latter years there has been an array of cheap server kit on eBay and last time we moved house I consolidated all the ageing hardware into a bargain eBay find – a single Compaq ML570G1 (Quad CPU/12Gb RAM and an external HDD array) which served fine until I realised just how much our home electricity bills were becoming!
Note the best practice location of my suburban data centre, beer-fridge providing hot-hot aisle heating, pressure washer conveniently located to provide fine-mist fire suppression; oh and plenty of polystyrene packing to stop me accidentally nudging things with my car. 🙂
I’ve been using a pair of HP D530 SFF desktops to run ESX 3.5 for the last year and they have performed excellently (links here here and here) but I need more power and the ability to run 64 bit VMs (D530’s are 32-bit only) I also need to start work on vSphere which unfortunately doesn’t look like it will run on a D530.
So I a acquired a 2nd-hand ML110 G4 and added 8Gb RAM – this has served as my vSphere test lab to-date, but I now want to add a 2nd vSphere node and use DRS/HA etc. (looks like no FT for me unfortunately though) – Techhead put me onto a deal that Servers Plus are currently running so I now have 2 x ML110 servers 🙂 they are also doing quad-core AMD boxes for even less money here – see Techhead for details of how to get free delivery here
In the past my labs have grown rather organically as I’ve acquired hardware or components have failed; being as this time round I’ve had to spend a fair bit of my own money buying items I thought it would be a good idea to design it properly from the outset 🙂
The design goals are:
- ESX 3.5 cluster with DRS/HA to support VM 3.5 work
- vSphere DRS/HA cluster to support future work and more advanced beta testing
- Ability to run 64-bit VMs (for Exchange 2007)
- Windows 2008 domain services
- Use clustering to allow individual physical hosts to be rebuilt temporarily for things like Hyper-V or P2V/V2P testing
- Support a separate WAN DMZ and my wireless network
- Support VLAN tagging
- Adopt best-practice for VLAN isolation for vMotion, Storage etc. as far as practical
- VMware Update manager for testing
- keep ESX 3/4 clusters seperate
- Resource pool for “production” home services – MP3/photo library etc.
- Resource pool for test/lab services (Windows/Linux VMs etc.)
- iSCSI SAN (OpenFiler as a VM) to allow clustering, and have all VMs run over iSCSI.
The design challenges are:
- this has to live in my garage rack
- I need to limit the overall number of hosts to the bare minimum
- budget is very limited
- make heavy re-use of existing hardware
- Cheap Netgear switch with only basic VLAN support and no budget to buy a decent Cisco.
Luckily I’m looking to start from scratch in terms of my VM-estate (30+) most of them are test machines or something that I want to build separately, data has been archived off so I can start with a clean slate.
The 1st pass at my design for the ESX 3.5 cluster looks like the following
I had some problems with the iSCSI VLAN, and after several days of head scratching I figured out why; in my network the various VLANs aren’t routable (my switch doesn’t do Layer 3 routing). For iSCSI to work the service console needs to be accessible from the iSCSI VKernel port. In my case I resolved this by adding an extra service console on the iSCSI VLAN to get round this problem and discovery worked fine immediately
I also need to make sure the Netgear switch had the relevant ports set to T (Tag egress mode) for the VLAN mapping to work – there isn’t much documentation on this on the web but this is how you get it to work.
The vSwitch configuration looks like the following – note these boxes only have a single GbE NIC, so all traffic passes over them – not ideal but performance is acceptable.
iSCSI SAN – OpenFiler
In this instance I have implemented 2 OpenFiler VMs, one on each D530 machine, each presenting a single 200Gb LUN which is mapped to both hosts
Techhead has a good step-by-step how to setup an OpenFiler here that you should check out if you want to know how to setup the volumes etc.
I made sure I set the target name in Openfiler to match the LUN and filer name so it’s not too confusing in the iSCSI setup – as shown below;
if it helps my target naming convention was vm-filer-X-lun-X which means I can have multiple filers, presenting multiple targets with a sensible naming convention – the target name is only visible within iSCSI communications but does need to be unique if you will be integrating with real-world stuff.
Storage Adapters view from an ESX host – it doesn’t know the iSCSI target is a VM that it is running 🙂
Because I have a non routed L3 network my storage is all hidden in the 103 VLAN, to administer my OpenFiler I have to use a browser in a VM connected to the storage VLAN, I did play around with multi-homing my OpenFilers but didn’t have much success getting iSCSI to play nicely, it’s not too much of a pain to do it this way and I’m sure my storage is isolated to a specific VLAN.
The 3.5 cluster will run my general VMs like Windows domain controllers, file servers and my SSL VPN, they will vMotion between the nodes perfectly. HA won’t really work as the back-end storage for the VM’s live inside an OpenFiler, which is a VM – but it suits my needs and storage vMotion makes online maintenance possible with some advanced planning.
Performance from VM’d OpenFilers has been pretty good and I’m planning to run as many as possible of my VMs on iSCSI – the vSphere cluster running on the ML110’s will likley use the OpenFilers as their SAN storage.
This is the CPU chart from one of the D530 nodes in the last 32hrs whilst I’ve been doing some serious storage vMotion between the OpenFiler VM’s it hosts.
That’s it for now, I’m going to build out the vSphere side of the lab shortly on the ML110’s and will post what I can (subject to NDA, although GA looks to be close)
As a result of a power outage last week my home lab needed a reboot as my 2 x ESX D530 boxes didn’t have auto-power on setting set in BIOS, so I dutifully braved the snow to get to the garage and power them on manually.
However nothing came back online.. ESX started but my VMs didn’t auto-restart as it couldn’t find them.
The run up to xmas was a busy month and I had vague recollections of being in the midst of using storage vMotion to move all my VMs away from local storage to an OpenFiler VM in preparation for some testing.
However, in my rush to get things working the OpenFiler box didn’t have a static IP address set and was using DHCP (see where this is going…?)
So my domain controller/DNS/DHCP and Virtual Centre server were stored on the OpenFiler VM which my ESX box was running and accessed over iSCSI. As such when ESX started it couldn’t locate the iSCSI volume hosting the VM and couldn’t start anything.
Once I’d figured out what was going on, it was simple enough to get things going again;
- Temporary DHCP scope on my router,
- IPCONFIG/ RENEW to get a temporary DHCP address on my laptop
- VI client directly to ESX box rather than VC and reboot the OpenFiler VM
- Web browser to OpenFiler appliance on temporary DHCP addresss
However at this point I would have expected to be able to set a static IP address and resolve the issue for the future, however I couldn’t see any NICs in the OpenFiler config screen (see screenshot below)
I thought this was a bit odd, and maybe I was looking in the wrong part of the UI, but sure enough it was the correct place.
I tried updating it to the most recent software releases via the handy system update feature, which completed ok (no reboot required – beat that Windows Storage Server! :)) but still no NICs showing up, even after a couple of reboots to be absolutely sure.
Then, I stumbled across this thread and it seems this may be a bug (tracker here) following Jason’s suggestion I used the nano text editor via the VI remote console to edit the /opt/openfiler/var/www/includes/network.inc file on the OpenFiler VM as follows;
I then refreshed the system tab in my browser session and the NICs show up;
note as part of my initial troubleshooting I added a 2nd virtual NIC to the VM, but the principal should apply regardless.
And I can now set a static IP etc.
I had to reboot my ESX host to get all my VM’s back from being inaccessible, I’m sure there is a cleverer way to do that, but in my case I wanted to test that the start-up procedure worked as expected now that I’ve set a static IP and re-jigged the start-up sequence so that OpenFiler starts before any other VMs that are dependent on it for their storage.