Virtualization, Cloud, Infrastructure and all that stuff in-between

My ramblings on the stuff that holds it all together

Move over vTARDIS here comes theBORG


Many of you will be familiar with my vTARDIS home lab project – this has run for a number of years and even won a couple of awards, making heavy use of nested ESX and virtual appliances for storage; however in the last 12 months RAM has been a massive problem – it’s hard to get SME type quiet hosts that can have more than 32GB of RAM.
If you consider most of VMware’s management stack consists of several 8GB virtual appliances and add in vCloud Director, vCAC etc. you can see my issue – I run out of capacity even before I start to run an actual workload.
In addition I’m doing a lot of work with the fantastic Atlantis ILIO and USX products, which can use RAM as storage infront of your storage array to de-dupe and deliver some very clever optimizations for storage (more on that in a later post).
So – I bit the bullet and purchased a blade chassis for home.
Did I read that right?
Are you rich/mad/deranged? (well definitely not rich but maybe the others..)

I was on the hunt via eBay for a DL380 or similar rackmount with 128GB of RAM – they were all pretty expensive, and for approx. the same money I managed to pick up a loaded up HP C7000 blade chassis with a total of >68 CPU cores and 318GB of RAM

Yes, it’s LOUD, yes it sucks a lot of power (this is with all the swich + FC modules installed, but only 1 full height blade powered on).



But – think of it this way – my total acquisition cost (including 4 x Cisco blade switches and 2 Cisco MDS FC blade switches was less than £2,000 GBP

It will cost (approx) £100 per month in electricity in the UK to run, I can handle that – consider it this way – purchased server cheap vs. spending £5,000 for a more modern set of kit to give the same capacity (and only ½ the running costs).

In my house we call this man-maths (and if you’re a Pistonheader – you may recognize the phrase) – it’s like when you buy an older, but very fast & cool car for a low (already depreciated) price but higher running costs (fuel, repairs etc.) but you enjoy it MUCH more than driving a more economical eurobox car for the same overall price(ish)


Anyway it makes sense to me and it currently runs about 40 nested ESX hosts, and I am only using 2 of the 14 blades ☺, it has a built in iLo-everything and it all comes in one box

(Disclaimer – the C7000 does not live in my house, as my wife would divorce me and the kids would have to wear ear-defenders).


More about what I’m using it for in future posts, yes that’s also an EMC VNXe for storage (again, subject of a future blog post – and to be honest it was the main reason for me needinf more compute capacity to push it – I blame Chad).


Do you want to know what VMware has planned for its next release?

Then you need to come to the London VMUG on July 17th (in London, UK), we are very lucky to be one of the few VMUGs that are able to deliver NDA content to VMware customers.

As this is NDA type content There are some caveats as to who can attend  – as listed below, but for those that aren’t eligible we have something else planned that will definitely be of interest.

And, we also have the usual array of awesome COMMUNITY, VMware and sponsor technical content. e.g no sales spin, no men in shiny suits and bonus schemes, real. interesting. relevant. technical content for YOU (ok, and me too!).

Sign up and view the full agenda here:

As I’m sure you understand VMware closely guard their roadmap and NDA content, we are very lucky to be able to deliver it at a VMUG, but there are some rules to prevent NDA material being leaked. If this happens VMware won’t be willing to help us deliver this content in future – so it’s in your interest to help safeguard the content – please respect this and the community spirit VMUG represents. You will have to sign a one-way personal NDA prior to being allowed entry to the NDA session.

All official VMware partners can request the content via their account manager at VMware, but to help make it clear we’ve put together an FAQ as follows.

  • I am an employee of a commercial or public sector organisation and we buy VMware products, or plan to, can I go the NDA session? Yes.
  • I’m a contractor and I work for a customer/consultancy, can I go to the NDA session? Yes
  • I work for a storage/network/server vendor, and am a VMware partner can I go to the NDA session? No. but you can request a private session via your VMware partner manager.
  • I work for a software vendor, and we are a VMware partner; can I go to the NDA session? No, but you can request a private session via your VMware partner manager.
  • I work for a reseller/consultancy who are a VMware partner, can I go the NDA session? No, but you can request a private session via your VMware partner manager.
  • I work for a cloud/managed/hosting service provider and we are VMware partners, can I go to the NDA session? No, but you can request a private session via your VMware partner manager.

This list isn’t exhaustive, if in-doubt of your status, bring a business card on the day andVMware can make a call, but please don’t be disappointed if we can’t admit you; we have an interesting update on VMware’s vision and strategy.

HP LaserJet 400 M475dw slow to connect to office365

I have one of these printers, it’s literally the best printer I have ever owned. it does everything, 1st time with no bother, aggro or hassle, it duplexes, copies, scans, prints very good colour pictures.


Granted, it’s not cheap or small but having purchased and thrown away literally tens of inkjet type printers over the years you get what you pay for with this printer.

The only small niggle is scanning to e-mail, I use Office365 for my personal and business email and I configured it to point at using this guide. however for some reason it took almost 3-5 minutes to connect and send to the SMTP server – I couldn’t find any obvious cause until I stumbled across this forum post, the solution? disable IPv6 – I tried this and it now connects & sends instantly.

For future reference, this is where you disable IPv6 on the web-UI.


VMworld accommodation on the cheap…

VMworld can be expensive, especially if, like me you are self-employed and don’t have a generous employer to cover the cost (or, even if your employer is cheap but won’t send you because of the cost).

For the last couple of VMworlds I have tried to save money on the accommodation as it can be a significant cost.

San Francisco, and Barcelona aren’t cheap cities, especially when they get wind of a conference – all the hotel prices go up as demand escalates.

My top tips..

1) Book early, before registration opens – early bird catches the worm, there are no last minute deals, ever.
2) Distance – whilst it’s nice to be near the venue, this can be very expensive – consider going further away and using the excellent public transport options in each city.
3) Lower your expectations, the Hilton is nice, but consider you’ll likely only be sleeping there between the conference and post-event parties – do you really care about the (premium) on-site spa facilities? and most cheaper places (even the hovels) have free WiFi – in your face $14.99/day Hilton!
4) Skip breakfast (at the hotel) and either eat locally to Moscone/Fira or there is usually breakfast laid on at VMworld (granted, not always the best food – WTF is a breakfast burrito for example, but still…)
5) Share, VMworld hotels are EXPENSIVE; especially if they are close to the venue; consider using AirBnB or look for a shared apartment on Expedia (easier in Barcelona than SF in my experience) & use the power of social media and gather up some like minded geeks + split the cost, there are significant savings here – provided you can avoid sharing with any serial-killers. On the latter point you can sort of vet would-be sharers via Social Media before saying yes (if they frequently post pics of shrunken heads or selfies with corpses, maybe think twice :)).

As evidence of this, I’ve stayed in some hovels in the past, stayed the other side of SF +commuted & this year I’m staying on a boat in the harbour with a mate via AirBnB + it’s 1/2 the cost of even a hovel hotel.

Good hunting!

Cannot logon to vCAC portal and you get Internal error when logging on as the root tenant

If you came to London VMUG last week and were one of the people who attended my vCAC real world session, you’ll have seen the demo gods not shining kindly on me as I couldn’t logon to my vCAC demo environment.

This was particularly annoying as I was testing it not 15mins before my session started and it was working 100%

With some quick on-stage debugging I found that I was unable to logon as any of the tenants on my vCAC deployment so was unable to show my cool demo 😦

After some further digging I found that I could logon to the root org using the administrator@vsphere.local account but nothing else and I couldn’t logon to any of my tenant organisations despite the credentials being 100% correct.

When logged onto the root org as administrator I could see the following error… which translates as “not good, your tenant has disapeared and I have an internal problem”


After some digging this seems this is a sort of time-bomb bug that has only been recently discovered, it occours after 90 days and will stop tenants being able to login using LDAP credentials.

When attempting to log into a tenant, a blank page is displayed with a Submit button in the upper left corner.
You receive a System Exception error when accessing the tenant identity store configuration page and the identity store configuration may have disappeared.
You cannot log in to a tenant using an LDAP account.
You are unable to add a new identity store configuration to the affected tenant.
The tenant identity store disappears from the SSO Administrator login.


This KB article details the issue and a temporary workaround

I assume a fix will follow in due-course.

In my case I worked out that it was exactly 90 days until about 15mins before my vCAC session started last Thursday – which is bad-timing taken to the extreme.

Once I have verified this fix in my environment and resumed access I will post a video of my intended real-world demo, where I’m using vCAC to deploy a VM and talk to a massive “enterprise system” via a REST API (e.g Twitter :)) – whilst an odd use-case this is an easy way of demonstrating the extensibility of vCAC (well, when it works anyway)

So, in summary if you have a vCAC implementation that you deployed since 6.0.1 came out you’re going to run into this install date +90 days.



How to Remove LinkedIn contact birthdays from your iPhone Calendar

Contacts of LinkedIn, I love you all. honestly I do – but there are a lot of you.

When I added the LinkedIn app to my iPhone it “helpfully” populated by iPhone calendar with everyone’s birthday – which is kind of a problem when you’re challenged for screen estate on the iPhone, and frankly as much as I would love to send you all a card and gift it’s not going to happen, so it’s a bit annoying.


You can remove these by tapping on Calendars at the bottom of the Calendar display.

Then scroll down past you usual calendar settings, look for “other” and uncheck Birthdays.


And presto, they’re gone.

*PS Kenny – I’ll buy you a bourbon next time I see you instead 🙂

Make a Windows application use OS X full screen mode when using Coherence Mode in Parallels

This has been bugging me for ages.. but it turns out it was there all the time! I have a Mac with the latest version of Parallels desktop (sorry VMware, Fusion + unity was getting too buggy for me).

I use coherence mode so I can run windows apps and OS X apps seamlessly side by side, but ever since Lion came out with the OS X full screen mode (and then mountain Lion actually made it useful) I have been using full screen mode for certain OS X apps when not using my external screens as it makes the most of my 13” screen.

However, I couldn’t quickly figure out how to make Windows apps go OS X full screen (and thus be visible in Mission Control/expose or whatever it is called now) so still got the task bar at the top and couldn’t 3-finger swipe between them (very handy feature) also a maximised Windows application covered over OS X apps on my primary desktop so I had to tab around to find it.

You can actually make individual Windows apps use OS X full screen with Parallels using a key-combination. When you have the Windows application active press CMD-CTRL + F and presto it’s using OS X full screen and you can see it in Mission Control and 3-finger swipe between them (the Outlook Window in the screenshot is a Windows app in coherence mode) or you can use the view menu as shown below to switch in/out of individual applications.

image image


Is your local VMUG your launchpad to a VMworld session?

My main role in helping to run the UK & London VMware User Group (VMUG) is finding, arranging, press-ganging and cajoling people into giving sessions at one of our quarterly VMUG meetings – I look for primarily for solid, interesting content, especially if it’s “outside of the box” and secondly a professional speaker – I don’t care who you are or if the last presentation you delivered was when you played the carrot in the cub scout christmas play aged 8 (yes, that was me – before you ask!).


You’ll get a lot out of presenting at a VMUG – even if you’re nervous, or new to presenting we can help to coach you through to a successful session and VMUG audiences, even helping you to find a co-presenter to help you out – the #FeedForward initiative is also a massive supporter of YOU and the audience are ALWAYS friendly and like-minded.


You don’t have to be captain of a datacentre the size of the starship enterprise to have something interesting to say, my 1st VMUG presentation 5 years ago was on my 2-host lab environment, and ever since I’ve iterated that vTARDIS presentation and delivered sessions on it at VMworld and VMUGs and other conferences around the world (Melbourne, Copenhagen, Chicago) and it even won the VMworld best of show award –  it’s been a hugely rewarding and enjoyable experience and I would encourage you all – no matter how timid or insignificant you think your experiences are to get in touch with your local VMUG and volunteer to do a presentation – VMUGs are about community and content – and we (the UK/LDN committee) work VERY hard to ensure the right balance between VMware, sponsor and COMMUNITY content.


In recent London VMUG’s we’ve started doing a slot for lightning-talks where you deliver a short, sharp & to-the-point 5-15min presentation on a topic of your choosing, these are a great way to jump-start your presenting career locally, and maybe even globally as they’re much less daunting than a normal 45-60min slot and are an excellent way to dip your toe in the water, there are also panel & round-table discussions that are a good way to get involved.


Being able to present, articulate a point and hold a confident discussion on a topic is a critical career skill if you want to stand out from the crowd, or keep your seat in a world of increasing off-shoring and out-sourcing.


So, in short get in touch (details on about page), present something, go LARGE by submitting your talk for VMworld via the call for papers process –  Let’s be realistic, VMworld sessions are traditionally hard to get accepted for unless you’re a sponsor or a well recognised name and there is a LOT of competition, on the other hand VMUGs sometimes struggle to find people willing to present, a lot of those recognised names started their vSpeaking career by getting involved at a local VMUG first – you do the maths, literally what have you got to lose?


BTW Next London VMUG meeting (#LonVMUG on twitter) is 15th May 2014.

Where are the vCAC log files

I’ve had some SRs open with VMware GSS lately on some vCAC issues, and the first thing they usually ask for is logs – to save yourself (and GSS) some time it’s worth gathering up the following logs and providing them immediately when you open your SR (using the upload tool) – this helps GSS to hit the ground running with your issue


Log description Log File Name Default Location
vCAC Installation log MMDDYYHHMMSS.txt c:\vcacLog
vCAC Server (everything) log All.log C:\Program Files (x86)\VMware\vCAC\Server\Logs
vCAC Server (errors only) log Error.log C:\Program Files (x86)\VMware\vCAC\Server\Logs
vCAC Agent logs vSphereAgent.log (name based on Agent type) C:\Program Files (x86)\VMware\vCAC\Agents\<AgentName>\logs
vCAC DEM Orchestrator (everything) log DEM Orchestrator Name_All.log C:\Program Files (x86)\VMware\vCAC\Distributed Execution Manager\<DEM-OrchestratorName>\Logs
vCAC DEM Orchestrator (errors only) log DEM Orchestrator Name_Errors.log C:\Program Files (x86)\VMware\vCAC\Distributed Execution Manager\<DEM-OrchestratorName>\Logs
vCAC DEM Worker (everything) log DEM Worker Name_All.log C:\Program Files (x86)\VMware\vCAC\Distributed Execution Manager\<DEM-WorkerName>\Logs
vCAC DEM Worker (errors only) log DEM Worker Name_Errors.log C:\Program Files (x86)\VMware\vCAC\Distributed Execution Manager\<DEM-WorkerName>\Logs
vCAC DEM logs – Additional logging Windows Event Viewer N/A
vCAC Repository logs Repository.log C:\Program Files (x86)\VMware\vCAC\Server\Model Manager Web\Logs
vCAC Website logs Windows Event Viewer N/A

How to monitor what your vCAC IaaS Server is doing in real-time

vCAC is a complex product with a lot of moving parts, it’s a pain to have to switch back & forth between monitoring views in the UI.

The IaaS server is a Windows box, and does most of the heavy lifting so you can’t just SSH in and tail a log file (well, you sort of could…  but)

I like to have all this info on a secondary monitor scrolling past in realtime, mainly so I can look like I am in the Matrix but it’s useful when you’re trying to debug a weird edge-case that only happens in real-time, I achieve this by..

Using mTail on my Windows administrative workstation.

Mapping a drive to the C$ share on my IaaS server

Open the following log files in mTail and put it on a secondary monitor so I can watch what’s going on in real-time (don’t forget to hit start to monitor the file)

\\IaaSserver\c$\Program Files (x86)\VMware\vCAC\Agents\<NAME OF YOUR AGENT(S)>\logs\vSphereAgent.log

\\IaaSserver\c$\c$\Program Files (x86)\VMware\vCAC\Server\Logs\All.log

For good measure you can also open your vCenter log files in the same way if you want to.

Handy hint – if you are tryign to catch a specific errror in the noise of the logs, check the Filter box and put the search expression in – then it will only show when that message occurs in the logs (like doing tail –f <file> | grep “string” in *NIX). you can use the following strings to limit to certain message categories.