Advertisements

Virtualization, Cloud, Infrastructure and all that stuff in-between

My ramblings on the stuff that holds it all together

Password sync Warning: no recent synchronization on Office365

If you manage an Office365 tenant like I do for my lab, and are security minded you may decide to change the password of the account you configured AAD Connect to use to talk to your on-prem Active Directory. For example if maybe you were lazy and used the default domain administrator account in your lab…. tut, tut :)) you need to update AAD Connect to reflect the new password otherwise you’ll get “Password sync Warning: no recent synchronization” on your admin page and no password changes will sync to Office365.

*I* thought you did this by running the Azure AD Connect tool and re-entering the password there, refreshing the directory. nope and other error logging is a bit sparse, other than the warning in the o365 tenant admin portal.

Password sync Warning: no recent synchronization on Office365

There are some excellent PowerShell utils for debugging this stuff in this post

in my case I got an error back like the following;

AAD Tenant - MyTenant.onmicrosoft.com
Password hash synchronization cloud configuration is enabled

AD Connector - MyDomain.tld
Password hash synchronization is enabled
No password hash synchronization heartbeat is detected

Connectivity:
=============
Directory Partition - MyDomain.tld
Password synchronization agent had a problem to resolve a domain controller in the domain "MyDomain.tld" at: 07/
11/2017 16:38:19 UTC
Please make sure AD Connector account username and password are correct
Only Use Preferred Domain Controllers: False
Checking connectivity to the domain...
Domain "MyDomain.tld" is reachable

Would you like to diagnose single object issues? [y/n]: n

For more help:
+ Please see - https://go.microsoft.com/fwlink/?linkid=847231 or
+ Open a service request through Azure Portal or Office 365 Admin Portal.

Which led me to think maybe AAD Connect was still using the old password.

To actually change the password and configure more details there is another utility outside of the Azure Connect wizard called “Synchronization service” which resides under “Azure AD Connect” on your start menu, run this. select the connectors to MyDomain.tld hit properties/Connect to AD Forest and update the password for the account you use to connect to on-prem AD.

You can also use this utility to configure a preferred domain controller if you don’t want it to follow the normal DC discovery process (useful if you have a segregated environment)

Blogged for when I have to do this again and invariably forget how..

Advertisements

Where there are geeks, there are gadgets – a cautionary vBeers tale

At the London VMUG we’ve held social beers in a pub after the event for over 10 years, in what has become known as vBeers.

In all that time I’m pleased to say that we’ve never had any problems, it’s all been good social fun. As you’d expect most of our attendees come with gadgets. Laptops, tablets, phones, watches etc. and we’ve never had any issues – other than the occasional identical bag or phone picked up by mistake. but quickly resolved.

Unfortunatley at our most recent event we were targetted by an oportunist thief who helped themselves to a selecton of gadgets from the bag pile – directly under the pub CCTV system.

The pub think they have identified the thief on CCTV and will be handing it over to the police.

Hopefully that won’t detract from future events or discourage you from attending some crime is unfortunatley inevitable in a large city like London but please do be careful and don’t make yourself a target. Keep an eye on your bag as you never know who is keeping an eye on your bag for you.

Cohesity at vRetreat

Last month I had the luck to be invited to the vRetreat event put together by Patrick Redknap, this was a day to get some bloggers face to face with some presenters looking at some cool, new tech.

In the interests of disclosure I should point out that there was a Porsche track day, food and a hotel involved. On the day we were very privileged to have 1:1 instruction around the purpose built track at Silverstone race course (you can pay a visit for this yourself http://www.porsche.com/silverstone/). There is no directive on what content I have to write or pressure to write something positive. Evidence of this is that it was over a month ago and work commitments have meant I’ve not had chance to write a blog post about the event until now, Patrick was very cool about it.

This delay did, however give me some time to give some serious thought to the info we received during the day, Veeam, Zerto and Cohesity presented on the day. I’ve worked hands-on with Zerto and Veeam before but Cohesity were new to me and it piqued my interest – the delay in writing this blog post meant I thought about some very cool real-world use-cases.

As a side note; Zerto also demonstrated something very cool, they have a ready to use appliance in Azure, I’ve been doing a lot of experimentation with Azure recently and whilst they were explaining some of the finer points I managed to deploy it before the end of the session! Now that’s the real power of the cloud! A little disappointing that the Azure ready to run appliance at the time was a Windows VM with some links, but I understand this has was due to some license constraints and the full appliance will be ready to download from Azure soon.

Anyways, back to Cohesity, who bill themselves as a Hyperconverged platform for secondary storage. I have to say I rolled my eyes a little at the mention of “another” hyperconverved platform, it’s secondary storage – everyone de-dupes etc. Jam in more capacity and shove some data on it to forget about it, or let it rot. We spent a lot of time talking about storage functionality that to be honest is plain storage, But I have to say regardless of how their magic is served up software or hardware offering – the last 5mins were the coolest part; it has a very cool plug-in architecture to allow Java apps to run on the appliance itself, just think about that analytics, search, applications themselves running on the storage itself. Now that is the cool bit, and it’s USP as far as I’m concerned, it’s new and still being developed, but bear with me..

A long time ago I worked for a company that did video on demand solutions, about 10 years ago one of the most interesting (but now defunct) vendors I worked with had scale-out storage solution (think cheap, x86 pizza boxes with very clever software way ahead of its time) that could store large amounts of video content but also transcode it to different formats at the same (or near-realtime) something that is very computationally expensive; storage is boring (sorry, but it is..) but if you can make it do something with the data it holds at the same time then to me, it’s very clever.

Cohesity offers this with its plugin and analytics plug-ins, what if you have a compliance use-case and you need to prove to regulators that all your data doesn’t contain credit-card numbers or other Personally Identifiable Information (PII); scanning large volumes of data with an application can only run a) periodically on a schedule, and b) at the speed of NFS/SMB etc. as data has to be read off the array, scanned, and in some cases written back. in this case the storage can do it efficiently on the array itself using all that spare CPU power.

Plug that together with an API on the Cohesity array or via the plug-in application and you not only have a really powerful scale-out storage device, but you have an application with vast amounts of data adjacent and on-tap (sorry, rubbish NetApp joke).

I like to think I maintain an even, unbiased professional line and I guess like me you’ll eye start-up storage vendors with the evil-eye as there is a risk that they go belly-up leaving you with an un-supportable storage headache that you need to replace at massive cost and migrate off, but I think this one has legs and something that nobody else on the market has, and to me that’s a reasonable bet they’ll get snapped up or live on their own.

Anyways, back to the Porsche bit. As a life-long Porsche fan the Porsche Experience is a great day out and you get to do some cool stuff like skidding about and a wet skidpan to test your driving skills (verdict: needs work!). I’ve always been a sceptic of those new fangled J pretend automatic/sort of manual gearboxes in sports cars after a bad investment in a BMW e46 M3 with an SMG gearbox (don’t do it kids). But a couple of hours of proper driving with a PDK gearbox is starting to convert me. My wife said specifically to me when I left the house for this event to not come back wanting to change my car, err..

Also got a ride in Joe Baguley’s Tesla Model X… those things are like rocket ships!

Sad post-note: Several days of very creative man-maths have not yet resulted in me being able to buy one, back to the grind-stone Gallagher.

if you want to get a flavour of what went on, check out this video

London and UK VMUG Dates 2016 and 2017

Dates for your diary for our future events, keep an eye on http://vmug.com/london for details of the agenda – Also feel free to join our LinkedIn group or follow our London VMUG Twitter feed and UKVMUG Feed to be kept up to date, we will post/tweet when the agenda and registration link is live as well as any logistical info.
2016

23 June at TechUK in London – followed by Luxury vBeers at a brewery
November 17th National Motorcycle Museum for the national UKVMUG

2017

19 January TechUK, London
6 April TechUK, London
22 June at TechUK, London
#UKVMUG 16th November at the National Motorcycle Museum, Solihull (Near Birmingham)

We (the committee) pride ourselves on promoting community content, we’re all about the U in VMUG – if you have an idea for a session at one of these events – you can use our handy call for papers form

Look forward to seeing you at a future meeting

 

How much do VMUG leaders contribute to VMUG globally

I’ve just attended the 1st annual VMware User Group Leader Summit, a day and a half event hosted at the VMware campus in Palo Alto to share best practice amongst the various groups around the world to better the organisation which was an excellent event.

The event was an impressive showing of commitment from VMware to the community in terms of the focus it has, but was more impressive was the scale of effort that the VMUG leaders put into their events – there aren’t many real rewards for being a VMUG leader other than some kudos and a pat on the back and it’s a very clear sign of people’s passion for the technology that they give this time freely.

Some leaders are self-employed or work full-time for an employer – but generally the time they give is their own personal, unpaid time as vacation time or work time that has to be made-up in personal time.

The London VMUG group of which I’m a leader in is currently going through a transition to a new team of leaders (I’m staying on but 3 leaders are stepping down after many years of service) and we’ve spent some time trying to quantify how much effort is required to run a VMUG group so we can set expectations appropriately for our new incoming leaders;

This is based on our experiences running 3 London (~100 attendees) and 1 UK national event (~600 attendees) each year.

Disclaimer this is very finger in the air analysis (and a little bit of fun) – but I do think it’s interesting to look at the opportunity cost of such activities (info on opportunity cost here) and other interesting {honest!} economics stuff here

Between the 4 of us we have 4 full-day meetings, so 4 man-days** contributed per meeting which we attend*, plus on average 2hrs of calls per month = 24hrs = 3 man-days/yr. (@8hrs/day) – so individually each leader contributes 7 man-days per year of effort to manage and run our events.

*I’ve not managed to sit through and enjoy a session at the London or UK VMUG meetings since I became a leader, because there is always something that needs doing, cats to herd, things to organise – not complaining, but – that’s the truth!

Our leadership team consists of 4 people, if we said the average group is 3 leaders (some have 7+, some have just 1!).

I don’t have access to all the details of the global VMUG chapters, but if you work on the basis that there was 1 leader invited from each active VMUG globally to the summit, there were 93 leaders in-attendance so let’s base our numbers on 93 ‘active’ groups – although I appreciate there are probably more as not everyone would be able to attend.

if we said an average of 3 leaders per ‘active’ group, each contributing 7 man-days per annum that’s 1,953 man-days per annum contributed by leaders to the VMUG community events. (3 x (4+3) ) x 93 = 1,953 man-days

Given there is an average of 251 working days per year that’s 7.7 man-years

Now, to make this more interesting, if we said the average salary of a VMware administrator was $80k USD (sort-of based on this article, and assuming that an VMUG leader will generally have more than 2 years of experience under their belt and will generally be in a senior-type role, the majority of VMUG leaders are in the US and salaries outside the US will obviously differ, but most VMUGs exist in well-developed 1st-word countries, rather than 2nd/3rd world emerging countries)

That would mean a VMUG leader globally earns an average of $318 per day before tax, multiply that out by the number of man-days given per year, that represents an opportunity cost that the VMUG leaders contribute to the VMware community & VMware itself of…..(drum-roll)

$622,470.12 USD.

Not too shabby 🙂 VMware, I hope you appreciate it 🙂

Anyways – just a bit of fun and not to be taken too seriously, but do go and hug a VMUG leader at your next meeting… (ok, don’t do that!)

 

 

**Yes, there are also many women who are VMUG leaders.. but man-days is an accepted term, and it’s shorter to type than person-years, apologies if it offends, it’s not meant to!

 

Making your OS X Terminal more useful for DOS refugees

I’m a DOS/Windows old-timer, but have been using a Mac for a number of years.. I find the OS X terminal (which is the *NIX bash shell) very flexible, but needs some tweaks to help me with my embedded DOS muscle memory – this is probably very basic and old-hat for *NIX types, but it’s here for my reference as I keep forgetting when I move to a new Mac.

If you look in your home dir “cd ~” you need to create (or edit, if it already exists) the “.bash_profile” file – you can do this with TextEdit, or use nano (“nano .bash_profile”)

Paste in the following contents

———–

export CLICOLOR=1
export LSCOLORS=GxFxCxDxBxegedabagaced
alias dir=”ls -ahl”

———–

Save the file (CTRL-X, yes, enter) in nano

then type “source .bash_profile” to load the changes (or start a new terminal session)

You now have a more DOS-like prompt with the full path in it, colour coding for different types of files and a “dir” command which shows the contents of the current directory by aliasing the “ls” command and adding some parameters to show it list-like.

Some really helpful references here http://natelandau.com/my-mac-osx-bash_profile/

Also – in terminal, terminal->preferences/profiles and you can set the “pro” profile as default by hitting the “default” button at the bottom of the pick list – also remember to check “Antialias Text” for sharper text.

POSH1Liner: Find all hosts with less RAM than you expect

If you have a cluster where maybe there are some hosts with spared out RAM due to a fault or a non-standard amount of RAM you can quickly find them with this command

get-vmhost | where {$_.MemoryTotalGB -lt THE_AMOUNT_YOU_EXPECT}

For example; to find all hosts with less than 512GB of RAM

get-vmhost | where {$_.MemoryTotalGB -lt 512}

“-lt” is “less than” which is slightly less intuitive than the usual < <= operators you'd use in other languages – but handy reference here http://ss64.com/ps/syntax-compare.html

As I go deeper with PowerShell (POSH) I like convenient things like the $_. syntax – makes it dead simple to come up with useful one-liners like this.

Joining an ESX host to the domain without rebooting

I found recently that despite the KB article saying no reboot is required that my ESX hosts would not authenticate AD users unless they were rebooted.

to work around this you can use the following PowerShell code to restart the relevant services without rebooting.

Get-VMHost $esx | Get-VMHostService | where {$_.Key -eq “lwiod”} | Restart-VMHostService -Confirm:$false
        Get-VMHost $esx | Get-VMHostService | where {$_.Key -eq “netlogond”} | Restart-VMHostService -Confirm:$false
        Get-VMHost $esx | Get-VMHostService | where {$_.Key -eq “lsassd”} | Restart-VMHostService -Confirm:$false
        Get-VMHost $esx | Get-VMHostService | where {$_.Key -eq “lbtd”} | Restart-VMHostService -Confirm:$false

Feel free to reuse the whole script, but do so at your own risk. (Download file (rename to .ps1)

# http://vinf.net Simon Gallagher (@vinf_net)
# Script to join all ESX hosts in a vCenter to the domain, adding a specific group into a vSphere advanced setting to add the YOUR_AD_GROUP group to the local ESX admins group on the ESX host

#Version 1.0

function ESXDomainJoin ([STRING]$doVC)
{
connect-viserver $doVC -credential $vCenterAcct
#connect to vCenter using the credentials we stored earlier

$esxHosts = get-VMHost #list all the hosts in this vCenter, then do something with them

    foreach ($esx in $esxHosts) {

        Write-Host “Doing domain join on $esx” -ForegroundColor Green
        $esxParam = “Config.HostAgent.plugins.hostsvc.esxAdminsGroup” # the advanced setting we want to change to the AD group
        $esxValue = “YOUR_AD_GROUP” #the name of the group we want to add to the setting
        Get-VMHost $esx | Get-AdvancedSetting -Name $esxParam | Set-AdvancedSetting -value $esxValue -Confirm:$false #-WhatIf # set it and don’t ask 1st
        #set DNS domain name (required for domain join)
        Get-VMHostNetwork -VMHost $esx  | Set-VMHostNetwork -DomainName your.domain.com  #-WhatIf
        #join domain using build account
        Get-VMHostAuthentication -VMHost $esx | Set-VMHostAuthentication -domain your.domain.com -user $buildAcct.getNetworkCredential().Username     -password $buildAcct.getNetworkCredential().Password -JoinDomain -Confirm:$false  #-WhatIf
        #Restart services so that the YOUR_AD_GROUP group gets automatically ACLd on local host without a reboot
        # takes 2-5mins to apply from AD after services are restarted, but then you should be able to logon using VI client/SSH to an individual ESX host using your AD creds
        Write-Host “Restarting services on $esx..” -ForegroundColor Green
        Get-VMHost $esx | Get-VMHostService | where {$_.Key -eq “lwiod”} | Restart-VMHostService -Confirm:$false
        Get-VMHost $esx | Get-VMHostService | where {$_.Key -eq “netlogond”} | Restart-VMHostService -Confirm:$false
        Get-VMHost $esx | Get-VMHostService | where {$_.Key -eq “lsassd”} | Restart-VMHostService -Confirm:$false
        Get-VMHost $esx | Get-VMHostService | where {$_.Key -eq “lbtd”} | Restart-VMHostService -Confirm:$false
        write-host “Completed restarting services, domain logon should be available in 5mins on $esx” -ForegroundColor Green
    }
disconnect-viserver * -Force #disconnect from all vCenters to be safe (get-VMhost connects to all vCenters you are connected to)
Write-Host “Done!” -ForegroundColor Green
} #end of function

#———–Start

write-host “Disconnecting from all current vCenter servers, just to be safe” –  -ForegroundColor Green
disconnect-viserver * -Force # disconnect from everything at the start, just to be safe

#build password list to work with
$vCenterAcct = Get-Credential -Message “Please enter credentials for vCenter administrator account”
$buildAcct = Get-Credential -Message “Please enter credentials to join machines to domain”

#now call the function for each vCenter in-turn
ESXDomainJoin(“FQDN_OF_YOUR_VCENTER”)

Find which SSL certificate is being used on an ESX host

If you have been through the pain of changing ESX host certificates from self-signed to real (CA signed certificates), you can check which certificate you are currently using for vCenter–>ESX host traffic by issuing the following command in an SSH session on the host

openssl x509 -noout -in /etc/vmware/ssl/rui.crt –text

You will see the details inside the SSL cert, signing authority etc. If the SSL cert contents make reference to VMware, you’re still using a self-signed certificate.

The vpxa (vCenter management) service on the ESX host is hard-coded to use /etc/vmware/ssl/rui.crt when the service is started (or restarted) so you can examine its properties using the above command to check
I can’t find any graphical way of checking this in the VI Client.

Cannot logon after setting the esxAdminsGroup advanced parameter

I found this issue recently – working on an environment where the ESX 5.5 hosts (build 1892794) had to be added to the domain. We needed to add a custom domain group in to grant it root access to the ESX host.
We followed this KB article http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2075361 to set the advanced parameter on each host (using PowerCLI) Config.HostAgent.plugins.hostsvc.esxAdminsGroup to be equal to the domain group we want to be granted admin access to the host e.g VMware_Admins so you can use AD credentials for SSH etc without relying on root all the time if you are a member of this group.

However – this did not work – if I connected the VI client directly to the host and logged on as root, I could not see the domain group on the permissions tab, I tried a lot of different things, combinations of DOMAIN\GROUPNAME and just GROUPNAME to no avail – the permission did not apply when I left & re-joined the domain.

It turns out that, in my environment despite the article saying no reboot required a reboot was actually required to enact the change and make it work (thanks to Julian Wood on twitter for sharing his identical experience).

I’ve provided some feedback to the KB article, but if you come across this issue – this is the cause! I did join/leave the domain a number of times with this host whilst testing so maybe this was a factor.