Virtualization, Cloud, Infrastructure and all that stuff in-between

My ramblings on the stuff that holds it all together

How much do VMUG leaders contribute to VMUG globally

I’ve just attended the 1st annual VMware User Group Leader Summit, a day and a half event hosted at the VMware campus in Palo Alto to share best practice amongst the various groups around the world to better the organisation which was an excellent event.

The event was an impressive showing of commitment from VMware to the community in terms of the focus it has, but was more impressive was the scale of effort that the VMUG leaders put into their events – there aren’t many real rewards for being a VMUG leader other than some kudos and a pat on the back and it’s a very clear sign of people’s passion for the technology that they give this time freely.

Some leaders are self-employed or work full-time for an employer – but generally the time they give is their own personal, unpaid time as vacation time or work time that has to be made-up in personal time.

The London VMUG group of which I’m a leader in is currently going through a transition to a new team of leaders (I’m staying on but 3 leaders are stepping down after many years of service) and we’ve spent some time trying to quantify how much effort is required to run a VMUG group so we can set expectations appropriately for our new incoming leaders;

This is based on our experiences running 3 London (~100 attendees) and 1 UK national event (~600 attendees) each year.

Disclaimer this is very finger in the air analysis (and a little bit of fun) – but I do think it’s interesting to look at the opportunity cost of such activities (info on opportunity cost here) and other interesting {honest!} economics stuff here

Between the 4 of us we have 4 full-day meetings, so 4 man-days** contributed per meeting which we attend*, plus on average 2hrs of calls per month = 24hrs = 3 man-days/yr. (@8hrs/day) – so individually each leader contributes 7 man-days per year of effort to manage and run our events.

*I’ve not managed to sit through and enjoy a session at the London or UK VMUG meetings since I became a leader, because there is always something that needs doing, cats to herd, things to organise – not complaining, but – that’s the truth!

Our leadership team consists of 4 people, if we said the average group is 3 leaders (some have 7+, some have just 1!).

I don’t have access to all the details of the global VMUG chapters, but if you work on the basis that there was 1 leader invited from each active VMUG globally to the summit, there were 93 leaders in-attendance so let’s base our numbers on 93 ‘active’ groups – although I appreciate there are probably more as not everyone would be able to attend.

if we said an average of 3 leaders per ‘active’ group, each contributing 7 man-days per annum that’s 1,953 man-days per annum contributed by leaders to the VMUG community events. (3 x (4+3) ) x 93 = 1,953 man-days

Given there is an average of 251 working days per year that’s 7.7 man-years

Now, to make this more interesting, if we said the average salary of a VMware administrator was $80k USD (sort-of based on this article, and assuming that an VMUG leader will generally have more than 2 years of experience under their belt and will generally be in a senior-type role, the majority of VMUG leaders are in the US and salaries outside the US will obviously differ, but most VMUGs exist in well-developed 1st-word countries, rather than 2nd/3rd world emerging countries)

That would mean a VMUG leader globally earns an average of $318 per day before tax, multiply that out by the number of man-days given per year, that represents an opportunity cost that the VMUG leaders contribute to the VMware community & VMware itself of…..(drum-roll)

$622,470.12 USD.

Not too shabby 🙂 VMware, I hope you appreciate it 🙂

Anyways – just a bit of fun and not to be taken too seriously, but do go and hug a VMUG leader at your next meeting… (ok, don’t do that!)



**Yes, there are also many women who are VMUG leaders.. but man-days is an accepted term, and it’s shorter to type than person-years, apologies if it offends, it’s not meant to!



Making your OS X Terminal more useful for DOS refugees

I’m a DOS/Windows old-timer, but have been using a Mac for a number of years.. I find the OS X terminal (which is the *NIX bash shell) very flexible, but needs some tweaks to help me with my embedded DOS muscle memory – this is probably very basic and old-hat for *NIX types, but it’s here for my reference as I keep forgetting when I move to a new Mac.

If you look in your home dir “cd ~” you need to create (or edit, if it already exists) the “.bash_profile” file – you can do this with TextEdit, or use nano (“nano .bash_profile”)

Paste in the following contents


export CLICOLOR=1
export LSCOLORS=GxFxCxDxBxegedabagaced
alias dir=”ls -ahl”


Save the file (CTRL-X, yes, enter) in nano

then type “source .bash_profile” to load the changes (or start a new terminal session)

You now have a more DOS-like prompt with the full path in it, colour coding for different types of files and a “dir” command which shows the contents of the current directory by aliasing the “ls” command and adding some parameters to show it list-like.

Some really helpful references here

Also – in terminal, terminal->preferences/profiles and you can set the “pro” profile as default by hitting the “default” button at the bottom of the pick list – also remember to check “Antialias Text” for sharper text.

POSH1Liner: Find all hosts with less RAM than you expect

If you have a cluster where maybe there are some hosts with spared out RAM due to a fault or a non-standard amount of RAM you can quickly find them with this command

get-vmhost | where {$_.MemoryTotalGB -lt THE_AMOUNT_YOU_EXPECT}

For example; to find all hosts with less than 512GB of RAM

get-vmhost | where {$_.MemoryTotalGB -lt 512}

“-lt” is “less than” which is slightly less intuitive than the usual < <= operators you'd use in other languages – but handy reference here

As I go deeper with PowerShell (POSH) I like convenient things like the $_. syntax – makes it dead simple to come up with useful one-liners like this.

Joining an ESX host to the domain without rebooting

I found recently that despite the KB article saying no reboot is required that my ESX hosts would not authenticate AD users unless they were rebooted.

to work around this you can use the following PowerShell code to restart the relevant services without rebooting.

Get-VMHost $esx | Get-VMHostService | where {$_.Key -eq “lwiod”} | Restart-VMHostService -Confirm:$false
        Get-VMHost $esx | Get-VMHostService | where {$_.Key -eq “netlogond”} | Restart-VMHostService -Confirm:$false
        Get-VMHost $esx | Get-VMHostService | where {$_.Key -eq “lsassd”} | Restart-VMHostService -Confirm:$false
        Get-VMHost $esx | Get-VMHostService | where {$_.Key -eq “lbtd”} | Restart-VMHostService -Confirm:$false

Feel free to reuse the whole script, but do so at your own risk. (Download file (rename to .ps1)

# Simon Gallagher (@vinf_net)
# Script to join all ESX hosts in a vCenter to the domain, adding a specific group into a vSphere advanced setting to add the YOUR_AD_GROUP group to the local ESX admins group on the ESX host

#Version 1.0

function ESXDomainJoin ([STRING]$doVC)
connect-viserver $doVC -credential $vCenterAcct
#connect to vCenter using the credentials we stored earlier

$esxHosts = get-VMHost #list all the hosts in this vCenter, then do something with them

    foreach ($esx in $esxHosts) {

        Write-Host “Doing domain join on $esx” -ForegroundColor Green
        $esxParam = “Config.HostAgent.plugins.hostsvc.esxAdminsGroup” # the advanced setting we want to change to the AD group
        $esxValue = “YOUR_AD_GROUP” #the name of the group we want to add to the setting
        Get-VMHost $esx | Get-AdvancedSetting -Name $esxParam | Set-AdvancedSetting -value $esxValue -Confirm:$false #-WhatIf # set it and don’t ask 1st
        #set DNS domain name (required for domain join)
        Get-VMHostNetwork -VMHost $esx  | Set-VMHostNetwork -DomainName  #-WhatIf
        #join domain using build account
        Get-VMHostAuthentication -VMHost $esx | Set-VMHostAuthentication -domain -user $buildAcct.getNetworkCredential().Username     -password $buildAcct.getNetworkCredential().Password -JoinDomain -Confirm:$false  #-WhatIf
        #Restart services so that the YOUR_AD_GROUP group gets automatically ACLd on local host without a reboot
        # takes 2-5mins to apply from AD after services are restarted, but then you should be able to logon using VI client/SSH to an individual ESX host using your AD creds
        Write-Host “Restarting services on $esx..” -ForegroundColor Green
        Get-VMHost $esx | Get-VMHostService | where {$_.Key -eq “lwiod”} | Restart-VMHostService -Confirm:$false
        Get-VMHost $esx | Get-VMHostService | where {$_.Key -eq “netlogond”} | Restart-VMHostService -Confirm:$false
        Get-VMHost $esx | Get-VMHostService | where {$_.Key -eq “lsassd”} | Restart-VMHostService -Confirm:$false
        Get-VMHost $esx | Get-VMHostService | where {$_.Key -eq “lbtd”} | Restart-VMHostService -Confirm:$false
        write-host “Completed restarting services, domain logon should be available in 5mins on $esx” -ForegroundColor Green
disconnect-viserver * -Force #disconnect from all vCenters to be safe (get-VMhost connects to all vCenters you are connected to)
Write-Host “Done!” -ForegroundColor Green
} #end of function


write-host “Disconnecting from all current vCenter servers, just to be safe” –  -ForegroundColor Green
disconnect-viserver * -Force # disconnect from everything at the start, just to be safe

#build password list to work with
$vCenterAcct = Get-Credential -Message “Please enter credentials for vCenter administrator account”
$buildAcct = Get-Credential -Message “Please enter credentials to join machines to domain”

#now call the function for each vCenter in-turn

Find which SSL certificate is being used on an ESX host

If you have been through the pain of changing ESX host certificates from self-signed to real (CA signed certificates), you can check which certificate you are currently using for vCenter–>ESX host traffic by issuing the following command in an SSH session on the host

openssl x509 -noout -in /etc/vmware/ssl/rui.crt –text

You will see the details inside the SSL cert, signing authority etc. If the SSL cert contents make reference to VMware, you’re still using a self-signed certificate.

The vpxa (vCenter management) service on the ESX host is hard-coded to use /etc/vmware/ssl/rui.crt when the service is started (or restarted) so you can examine its properties using the above command to check
I can’t find any graphical way of checking this in the VI Client.

Cannot logon after setting the esxAdminsGroup advanced parameter

I found this issue recently – working on an environment where the ESX 5.5 hosts (build 1892794) had to be added to the domain. We needed to add a custom domain group in to grant it root access to the ESX host.
We followed this KB article to set the advanced parameter on each host (using PowerCLI) Config.HostAgent.plugins.hostsvc.esxAdminsGroup to be equal to the domain group we want to be granted admin access to the host e.g VMware_Admins so you can use AD credentials for SSH etc without relying on root all the time if you are a member of this group.

However – this did not work – if I connected the VI client directly to the host and logged on as root, I could not see the domain group on the permissions tab, I tried a lot of different things, combinations of DOMAIN\GROUPNAME and just GROUPNAME to no avail – the permission did not apply when I left & re-joined the domain.

It turns out that, in my environment despite the article saying no reboot required a reboot was actually required to enact the change and make it work (thanks to Julian Wood on twitter for sharing his identical experience).

I’ve provided some feedback to the KB article, but if you come across this issue – this is the cause! I did join/leave the domain a number of times with this host whilst testing so maybe this was a factor.

LonVMUG Prizes for community content aka vFACTOR London

As a fellow VMware community member I know how much I have benefited from sharing my experiences with my peers ; since I started doing so – I’ve given numerous presentations around the world on my work projects and my home lab projects I’ve met a lot of interesting people, I wrote a book about a product that VMware discontinued almost as soon as it was published  (ok, maybe gloss-over that last one!)

TL;DR version of this post

Come and present a 10min talk at London VMUG on 22nd Jan 2015 and win Apple shiny good or Amazon vouchers – submit abstract & rules here now

As a member of the London & UK VMUG steering committee I also know how hard it is for VMUGs to find people willing to talk about their experiences with their peers by giving a short talk or hosting a discussion group – so, in order to help encourage (well, ok bribe!) people to get them on their way to community stardom we are announcing EXCELLENT PRIZES for 5 first-time VMUG presenters who are end-users and are willing to deliver just a short 10-minute talk on a VMware related topic of their choosing at the next London VMUG meeting (Jan 22nd in central London)

And when I say FANTASTIC PRIZES – I mean…

1st Prize: MacBook Air
2nd Prize: iPad Air
3rd Prize: iPad Mini
4th Prize: Amazon voucher
5th Prize: Amazon voucher

You can submit your abstract using this online form here

We will pick 5 submissions from the entries to present at the event and the closing date is 19th December 2014 – everyone who presents will win something, and the audience will vote for their favourite to determine what loot you go home with from the above list?

Not convinced…? Read on.

Below are the most common reasons I hear not to do a talk… and are all my own personal opinion..

I’m nervous, I’m no Duncan Laverick, or Mike Dennemann  I can’t do public speaking

We all have been here, we’re a friendly crowd and it’s an excellent way to learn some important life & professional skills. To help with this the community at-wide offers the feed forward initiative, you can read more about this here – it can help you find a mentor to help you prepare for the day – one of our own members – Alex Galbraith did just this – read about his experiences on his blog here

I cant think of anything to talk about

We all have great experience from environments small and massive – I’ve personally found the best way to capitalise on this knowledge is to share it with others, having to distil it down to a presentation or blog post is a great way to help you focus on the important facts of what you did and how you did it – I have a tendency to whitter on (if you’ve read this blog for a while you already know this!) and doing this is an excellent tool, especially if you have to explain it to someone outside of your organisation

I work for a 3 man band company, nobody is interested in what I do

Wrong, I started my VMUGging talking about my 1 node home lab, you don’t have to have a data centre the size of the starship enterprise to have something interesting to say. Infact – I’d say from personal experience people from SMB type organisations have much more limited budgets and have to find far more creative ways to get things done

I work in a nuclear bunker, and I have access to your home address and browser history – I can’t talk about what I do

Now I know some of you maybe do work in nuclear bunkers, 3-letter govt. agencies or centres of commercial finance and are unable to talk about what you do for fear of exposing commercial secrets – I understand this, so do I.. My suggestion is that if you want to submit a talk – anonymise what you do – if you work for Goliath National Bank, just say you work for “a major bank”, or “an airline” – this is a VMUG, not Panorama. VMUG attendees are interested in the tech, what you did, how you did it & what could have gone better; not what impact it had on your Q4 financial results.

If you take this approach and remember to say you are not there as a company spokesperson, this is my own opinion etc. this is usually acceptable.

So, that’s it – submit here and I look forward to giving you your prize on 22nd January 2015.

..And to be honest – what’s the worst that could happen… 10mins of your time for a prize? …and we go to the pub afterwards anyway 🙂

If, even after all my objection handling you really can’t bring yourself to do it; that’s fine 🙂 you’re still welcome to come along and soak up the excellent content – you can register online at

What to see at UKVMUG 2014

The 4th annual UK-wide VMware User Group Conference is in a couple of weeks, on Tuesday 18th November several hundred (last year it was over 500!) of your peers will travel to the National Motorcycle Museum in Solihull.

The venue is just down the road from Birmingham airport, and a short taxi ride from the local train station – or easy to get to by road with free parking.

As part of the committee that puts this event on I’d like to point out some of the highlights of the agenda; as per-every year, we have the usual array of headlining technical super-stars from the virtualization community – Duncan Epping, Frank Denneman, Mike Laverick and Chris Wahl. But, over the years we’ve continued to build-up the community mezzanine area to deliver highly interactive face to face discussion groups and white board sessions – small groups of people talking brain-bendingly deep tech, or just introductions for n00bs – the guys staffing these tables are volunteers from all areas of industry and are the best brains in the biz – take advantage!

These are taking place on the mezzanine level and are noted as side-sessions on the main agenda. The mezzanine is the raised area at the back of the solutions-exchange & will be sign-posted on the day – there are way too many for me to detail on this blog post – 20 sessions in total!!!

    • VMware GSS are running a drop-in clinic – this is your chance to ask the tricky questions of VMware support, they are excellent supporters of VMUG and are sending two of their top level support & escalation engineers to field your questions face to face.
    • Designing real-world workflows for vRealize automation center (vCAC) with Jon Medd – if, like me – you’ve struggled to find practical examples & resources for this sort of project – Jon is your man; he’s done it for real – ask him the tricky questions and he’s an absolute wizz at PowerCLI!
    • Ask the CTO anything; this is your chance to have a face to face discussion with VMware EMEA CTO Joe Baguley – Joe is an avid VMUG supporter and general all-round expert at all things technical – he’s game for any questions – even non-VMware related ones (hint: ask him about fireworks or Robot Wars, or even SDDC) – how often do you get that opportunity! (he is also our keynote speaker).

We start early on the Tuesday, and as such we are repeating the popular vCurry the night before – check the box when you sign up to attend; we’ll be laying on curry and there will be a light-hearted quiz, this was a hoot last year – although we have already decided the guys from VMware GSS are excluded from winning 🙂 – although beware they may set some of the questions :).

Please note the event is on a Tuesday (not the typical Thursday) – but it’s early enough in the week to hopefully get your boss to agree that you can go to what is effectively a mini-VMworld in the UK.

You can see the full agenda online at and you can sign-up at

Press F8 to enter CIMC configuration does not work on Cisco c-class rack mount server

I encountered this today, on a system that is not managed by UCS manager, to setup the CIMC (HP iLo equivalent) you need to connect to the physical console with a screen and keyboard to set the initial IP address.

You do this by pressing F8 at the BIOS screen, however I couldn’t get this to work

the fix was simple, if a bit weird – I had accidentally cabled the management NIC to the serial console port on the back of the server, if you do this – it recognises you pressing F8 but then boots to a flashing prompt – I assume this is because it switches to some sort of serial console interface unstead of displaying the UI on the screen.

for reference – connect the Ethernet management cable to the correct NIC! as below


Move over vTARDIS here comes theBORG


Many of you will be familiar with my vTARDIS home lab project – this has run for a number of years and even won a couple of awards, making heavy use of nested ESX and virtual appliances for storage; however in the last 12 months RAM has been a massive problem – it’s hard to get SME type quiet hosts that can have more than 32GB of RAM.
If you consider most of VMware’s management stack consists of several 8GB virtual appliances and add in vCloud Director, vCAC etc. you can see my issue – I run out of capacity even before I start to run an actual workload.
In addition I’m doing a lot of work with the fantastic Atlantis ILIO and USX products, which can use RAM as storage infront of your storage array to de-dupe and deliver some very clever optimizations for storage (more on that in a later post).
So – I bit the bullet and purchased a blade chassis for home.
Did I read that right?
Are you rich/mad/deranged? (well definitely not rich but maybe the others..)

I was on the hunt via eBay for a DL380 or similar rackmount with 128GB of RAM – they were all pretty expensive, and for approx. the same money I managed to pick up a loaded up HP C7000 blade chassis with a total of >68 CPU cores and 318GB of RAM

Yes, it’s LOUD, yes it sucks a lot of power (this is with all the swich + FC modules installed, but only 1 full height blade powered on).



But – think of it this way – my total acquisition cost (including 4 x Cisco blade switches and 2 Cisco MDS FC blade switches was less than £2,000 GBP

It will cost (approx) £100 per month in electricity in the UK to run, I can handle that – consider it this way – purchased server cheap vs. spending £5,000 for a more modern set of kit to give the same capacity (and only ½ the running costs).

In my house we call this man-maths (and if you’re a Pistonheader – you may recognize the phrase) – it’s like when you buy an older, but very fast & cool car for a low (already depreciated) price but higher running costs (fuel, repairs etc.) but you enjoy it MUCH more than driving a more economical eurobox car for the same overall price(ish)


Anyway it makes sense to me and it currently runs about 40 nested ESX hosts, and I am only using 2 of the 14 blades ☺, it has a built in iLo-everything and it all comes in one box

(Disclaimer – the C7000 does not live in my house, as my wife would divorce me and the kids would have to wear ear-defenders).


More about what I’m using it for in future posts, yes that’s also an EMC VNXe for storage (again, subject of a future blog post – and to be honest it was the main reason for me needinf more compute capacity to push it – I blame Chad).