Virtualization, Cloud, Infrastructure and all that stuff in-between
My ramblings on the stuff that holds it all together
Category Archives: Windows
Mapping a drive to a VSS Snapshot & General DFS-R woes
Posted by on September 14, 2008
Microsoft’s volume snapshot service is pretty handy right? quick hardware independent snaps of a file system – all free and out of the box, well it’s now officially saved my bacon…. whilst it’s a bit klunky (more on this in a bit) it was damned useful.
I had a pain of a problem to deal with this weekend, helping out a friend doing some server re-organising (plan was to migrate these guys from VMWare Server 1.x to ESXi – but didn’t get that far due to some other Windows issues that took all of our time as we checked everything was ok before the move)
Firstly, if you use DFS-R (as comes with Win2003 R2) never, ever, ever, ever use the “distributed file system” applet to administer DFS, we needed to add a new replica of a large DFS-R set to another server and because (in our defence) the server was a fresh R2 install, we forgot to install the newer DFS-R components via control panel, but original DFS was still installed by default and we were in a hurry (read: not paying attention) we used the “Distributed File System” applet to add a new target, and followed the wizard which actually re-created the DFS volume (note to self – pay more attention when clicking!) from scratch.
It proceeded to delete all the contents of all the DFS shares and moved them to a folder called NtFrs_PreExisting___See_EventLog and started afresh, that wouldn’t be so bad except for some inexplicable reason it then purged the contents of that folder from all replicas so we had no quick cut & paste file copy solution.
This was not going to be a fun weekend.
Don’t use this one 
use this one!
So, basically it was our (my) fault – but it was compounded by some weird corruption in one of the directories that looked like it had been there a while that meant recovery wasn’t going to be straightforward.
The data Backup was about 24hrs older than the last VSS snapshot on the central file server (hub & spoke replication topology) so as we now had a flat, deleted DFS volume with no data (thanks!) we decided to try and revert to the most recent VSS snapshot for the relevant directories.
But no dice, it just threw an error – can’t copy, I can view the files and see the contents and can drag and drop one or two a time, but any more and it would throw an error.
Not good, I can only assume that this was because of some logical corruption within the file system as there was one whole directory tree I couldn’t access (more on how I recovered this later).. there were over 60k files so I wasn’t going to do that by hand – so a command line was in order as at least XCopy can ignore errors etc. and just pull out the good data.
I found these excellent articles here and here and documentation here but some of them were more geared towards taking a snapshot and extracting data in-situ rather than from a persistent snapshot like you get with VSS.
so, none of them worked for me ; and even a lot of hacking with Vshadow and MOUNTVOL I couldn’t get the VSS Snap to mount at all and time was short
I did discover the following though, if you view a snapshot using the Previous Versions tab (remember this only works if you browse for files to restore via UNC path) it opens the snap in Explorer, but you can’t map a drive to it or run a command line copy against it…. or can’t you 🙂
When you open it in explorer this way it does create a sort of hidden temporary share – easiest way I found to expose the name of the share was to try and zip a file in the explorer session that is looking at the snapshot using WinZip, if you follow the wizard at some point it will expose a UNC path like \\SERVERNAME@GMT-DD-MM-YY-{GUID} if you can cut & paste that you can then map a network drive to it
NET USE * \\servername@gmt-dd-mm-yy-{guid}
And you can then run xcopy etc against that mapped drive to copy out all the good data – in reality we used SyncBackSE – which is great for complex file copies and we already had it installed.
All of these Windows servers were installed as VM’s in VMWare Server(s), so it actually made our lives a lot easier as we could quickly clone a known-broken server as-is (do no further harm) and then spin it up disconnected from the network to recover data using this method and also undeleted files using Get Data Back NTFS etc. and then use that data to re-seed the DFS-R volume – but much easier than if it were a physical box and at no real risk of making things worse.
So, in conclusion this was human error, rather than a 100% technical problem and should have been better planned and prevented by maintenance and a better recovery plan- but here it is, with the solution we found to get things back in all its gory details… and mainly as a footnote so I don’t make the DFS mistake again and in my defence this is a shoe-string charity operation rather than a blue-chip org with significant money and time to invest in such efforts.
This solution worked for us, but you need to have your own tried & tested solution – don’t rely on this as far as I can tell it’s unsupported, use at your own risk!
Virtualization – the key to delivering "cloud based architecture" NOW.
Posted by on June 23, 2008
There is a lot of talk about delivering cloud or elastic computing platforms, a lot of CxO’s are taking this all in and nodding enthusiastically, they can see the benefits.. so make it happen!….yesterday.
Moving your services to the cloud, isn’t always about giving your apps and data to Google, Amazon or Microsoft.
You can build your own cloud, and be choosy about what you give to others. building your own cloud makes a lot of sense, it’s not always cheap but its the kind of thing you can scale up (or down..) with a bit of up-front investment, in this article I’ll look at some of the practical; and more infrastructure focused ways in which you can do so.
Your “cloud platform” is essentially an internal shared services system where you can actually and practically implement a “platform” team that operates and capacity plans for the cloud platform; they manage it’s availability and maintenance day-day and expansion/contraction.
You then have a number of “service/application” teams that subscribe to services provided by your cloud platform team… they are essentially developers/support teams that manage individual applications or services (for example payroll or SAP, web sites etc.), business units and stakeholders etc.
Using the technology we discuss here you can delegate control to them over most aspects of the service they maintian – full access to app servers etc. and an interface (human or automated) to raise issues with the platform team or log change requests.
I’ve seen many attempts to implement this in the physical/old world and it just ends in tears as it builds a high level of expectation that the server/infrastructure team must be able to respond very quickly to the end-“customer” the customer/supplier relationship is very different… regardless of what OLA/SLA you put in place.
However the reality of traditional infrastructure is that the platform team can’t usually react as quick as the service/application teams need/want/expect because they need to have an engineer on-site, wait for an order and a delivery, a network provisioning order etc. etc (although banks do seems to have this down quite well, it’s still a delay.. and time is money, etc.)
Virtualization and some of the technology we discuss here enable the platform team to keep one step ahead of the service/application teams by allowing them to do proper capacity planning and maintain a pragmatic headroom of capacity and make their lives easier by consolidating the physical estate they manage. This extra headroom capacity can be quickly back-filled when it’s taken up by adopting a modular hardware architecture to keep ahead of the next requirement.
Traditional infrastructure = OS/App Installations
- 1 server per ‘workload’
- Silo’d servers for support
- Individually underused on average = overall wastage
- No easy way to move workload about
- Change = slow, person in DC, unplug, uninstall, move reinstall etc.
- HP/Dell/Sun Rack Mount Servers
- Cat 6 Cables, Racks and structured cabling
The ideal is to have an OS/app stack that can have workloads moved from host A to host B; this is a nice idea but there are a whole heap of dependencies with the typlical applications of today (IIS/apache + scripts, RoR, SQL DB, custom .net applications). Most big/important line of business apps are monolithic and today make this hard. Ever tried to move a SQL installation from OLD-SERVER-A to SHINY-NEW-SERVER-B? exactly. *NIX better at this, but not that much better.. downtime required or complicated fail over.
This can all be done today, virtualization is the key to doing it – makes it easy to move a workload from a to b we don’t care about the OS/hardware integration – we standardise/abstract/virtualize it and that allows us to quickly move it – it’s just a file and a bunch of configuration information in a text file… no obscure array controller firmware to extract data from or outdated NIC/video drivers to worry about.
Combine this with server (blade) hardware, modern VLAN/L3 switches with trunked connections, and virtualised firewalls then you have a very compelling solution that is not only quick to change, but makes more efficient use of the hardware you’ve purchased… so each KW/hr you consume brings more return, not less as you expand.
Now, move this forward and change the hardware for something much more commodity/standardised
Requirement: Fast, Scalable shared storage, filexible allocation of disk space and ability to de-duplicate data, reduce overhead etc, thin provisioning.
Solution: SAN Storage, EMC Clariion, HP-EVA, Sun StorageTek, iSCSI for lower requirements, or storage over single Ethernet fabric – NetApp/Equalogic
Requirement: Requirement Common chassis and server modules for quick, easy rip and replace and efficient power/cooling.
Solution: HP/Sun/Dell Blades
Requirement: quick change of network configurations, cross connects, increase & decrease bandwidth
Solution: Cisco switching, trunked interconnects, 10Gb/bonded 1GbE, VLAN isolation, quick change enabled as beyond initial installation there are fewer requirements to send an engineer to plug something in or move it, Checkpoint VSX firewalls to allow delegated firewall configurations or to allow multiple autonomous business units (or customers) to operate from a shared, high bandwidth platform.
Requirement: Ability to load balance and consolidate individual server workloads
Solution: VMWare Infrastructure 3 + management toolset (SCOM, Virtual Centre, Custom you-specific integrations using API/SDK etc.)
Requirement: Delegated control of systems to allow autonomy to teams, but within a controlled/auditable framework
Solution: Normal OS/app security delegation, Active Directory, NIS etc. Virtual Center, Checkpoint VSX, custom change request workflow and automation systems which are plugged into platform API/SDK’s etc.
the following diagram is my reference architecture for how I see these cloud platforms hanging together
As ever more services move into the “cloud” or the “mesh” then integrating them becomes simpler, you have less of a focus on the platform that runs it – and just build what you need to operate your business etc.
In future maybe you’ll be able to use the public cloud services like Amazon AWS to integrate with your own internal cloud, allowing you to retain the important internal company data but take advantage of external, utility computing as required, on demand etc.
I don’t think we’ll ever get to.. (or want) to be 100% in a public cloud, but this private/internal cloud allows an organisation to retain it’s own internal agility and data ownership.
I hope this post has demonstrated that whilst, architecturally “cloud” computing sounds a bit out-there, you can practically implement it now by adopting this approach for the underlying infrastructure for your current application landscape.
Running ESX 3.5 and 3i Under VMWare Workstation 6.5 Beta Build 91182
Posted by on May 18, 2008
Following on from my earlier post I upgraded my installation to the new build of 6.5. it un-installed the old build and re-installed the latest without a problem, took about 30mins and required a reboot of the host OS.
All my previously suspended XP/2003 VM’s resumed ok without a restart but needed an upgrade to the VMTools which did require a restart of the guest OS – all completed with no problems.
Now, onto installing ESX….
I used the settings from Eric’s post here to edit my .vmx file
ethernet0.virtualDev = “e1000”
monitor.virtual_exec = “hardware”
monitor_control.restrict_backdoor = “true”
Note – you need to select an x64 Linux version from the VM type drop down, if you have to go back and change it via the GUI after you’ve edited the .vmx file it overwrites the Ethernet card “e1000” setting to “vlance” so you need to edit again otherwise the ESX installer won’t find a compatible NIC and won’t install.
it was initially very slow to boot; 5mins on my dual core laptop with only one error – which was expected..
To improve the performance I changed my installation to run the non-debug version of the Workstation binaries (rename the vmware-vmx.exe to vmware-vmx-debug.exe)
note: this isn’t recommended unless you know what you are doing, VMWare will rely on the output from the debug version of the code if you need to report any issues)
It also seems to work for the installable version of ESX 3i… (although I’ve not quite figured out the point of that version yet :)).
Install prompt
it did fail with an error the 1st time round..
this was because I had specified an IDE disk as per the ESX instructions, I changed it to a SCSI one and it worked ok.
Finished..
The ESX 3i install has a footprint of about 200Mb on disk, and ESX 3.5 uses 1.5Gb.
I’m going to keep the 3.5 install on my laptop and will try to use linked clones to maintain a couple of different versions/configs to save disk space.. I’m sure I could knock up a quick script to change the hostname/IP of each clone – if I do I’ll post it here.
Why would you want to do this? well because you can, of course 🙂 and its handy for testing patch updates and scripts for ESX management etc.
I will also try to get a ESX DRS cluster running under workstation with a couple of ESX hosts and shared storage over iSCSI using something like OpenFiler as shown here. won’t exactly be production performance, but useful for testing and demo’ing.
New VMWare Workstation 6.5 Build(s) and ability to run ESX 3.5
Posted by on May 18, 2008
As a result of this post from Eric Sloof I note there is a new build of Workstation 6.5 available; I hadn’t noticed this as I haven’t had much time to follow the forums and my beta/RC (as used in this post and installed here is build 84113) hasn’t notified me there is a new release as all the previous 4.x/5.x beta’s have.
Oddly I checked this morning before I saw Eric’s post and it reported no new builds available – assume this is because its still a beta programme.
Anyway – if you downloaded the previous build before 14th May then go to this page and you can update your registration for the new build (below).
I’ll be trying this out in the coming week and hopefully will be able to get ESX running on my laptop under VMWare Workstation (very handy mobile demo platform).
Misc bits of Useful, Recent VMWare News
Posted by on May 12, 2008
I’ve been really busy the last couple of weeks and I’ve had to trim down my incoming RSS feeds, as there was too much noise and I was missing important things like the following;
- Scott Lowe’s summary of sessions from VMWare’s partner Exchange, some useful information on Site Recovery Manger
- The new VMWare Certified Design Expert (VCDX) certification – next step up from VCP, will have to have a look into it now I’ve finally managed to re-schedule my cancelled QA course – official VM announcement here.
- Official Microsoft Clustering Support with ESX 3.5 Update 1 here
- Some workarounds for deploying Windows Server 2008 with virtual center here – would have been nice if support was in an official update from VMWare soon; it’s not like it’s been beta’ing for a while is it (errr!)
VMWare Workstation 6.5 Beta – Run Multiple Copies of Outlook/Exchange via Unity
Posted by on April 30, 2008
I use a single laptop for my day-day use, it has all the stuff i need, I run Vista and Office 2007, for our corporate mail we use Exchange like everyone else and I use Outlook Cached Mode to work online/offline..
My own personal email is also an Exchange mailbox – provided by fasthosts (why – well, because..ok?) the problem with this is that I can’t have a single copy of Outlook connected to more than one Exchange server at the same time or run multiple instances of Outlook (I’ve tried all the hacks and Thinstall etc.), and to be honest even if I could it would probably violate the security policies of all the involved organisations as it would be quite simple for an Outlook-aware worm to try to propagate itself across multiple organisations or harvest confidential details.
The problem is further compounded by the fact that I often work on long-term customer projects and have to have a mailbox on their Exchange system as well… which leads to multiple diary sync nightmare, maybe I’ll blog about that some other time).
So at present I have 4 Exchange mailboxes that I need to keep track of, auto-forwarding mail between them is a no-no, I used to be an Exchange admin and I’ve lost many bank holidays due to corporate->Hotmail NDR mail loops!
So, up until now I’ve had to run one full Outlook client and multiple OWA clients in a browser, which is ok as long as I’m connected to the Internet, but no good if I’m on a train unless I want to close and restart Outlook with multiple profiles, which is a pain especially when you are collaborating on a project between multiple organisations. To be honest as good as OWA 2003 is it’s no substitute for a full outlook client. (still waiting for Fasthosts to go to Exchange 2007, oh and enable EAS!).
So, anyway a solution – VMWare Unity, this is a feature like Parallels for the Mac which lets you “float” an application window out of a guest VM to the host desktop meaning you can use the applications without working within a single VM’d desktop window.
VMWare Fusion also has the same feature, but Workstation 6.5 is the 1st time its been available on the PC platform.
To use Unity you need to have upgraded the virtual machine to 6.5 “hardware” by right clicking on the VM in the sidebar pane (below) and install the latest VM Tools – it also only seems to support XP at present, or at least it didn’t work on the Server 2003 VM I had.
Boot the VM… and install the latest VM tools.
VM Workstation Screen – note VM is set to “Unity mode”
My Vista desktop (yes, I have the start bar at the right hand side – widescreen laptop!) with the popup menu for the VM, showing all the start menu for applications installed within in it.
the following screen shot is Calculator running from inside the XP VM but in a single window on the Vista desktop – note the red border and the 
icon, denoting that its presented via Unity.
It even shows up on the start bar with the correct icon; although this doesn’t seem to work until its been run a couple of times; I assume it needs to cache an icon or something.
it also seems to respect the window snapshots you get whilst Win-Tab between applications, even for pop-up windows
Technically I can use this to run n x Windows XP/Outlook 2003 VM’s presenting Outlook through to my Vista desktop and comply with all organisations security policies, as each VM and its respective copy of Outlook runs in isolation from each other with the relevant company-specific AV client (or at worst, the same level as if I were using a machine connected to a public network in that they all share a vm network) – I don’t enable shared folders between the VMs.
It’s still a beta feature at the moment, and there seem to be a few bugs particularly when resizing windows sometimes it doesn’t work properly and double clicking to expand to full screen overlays the start-bar on my vista machine.
And it does seem to get confused sometimes and not allow keyboard input, so you have to flick back to non-unity mode and then back to continue, and sometimes a reboot of the guest VM but it is an early build so I would guess this will be resolved.
As an added bonus VM Workstation seems to allow the Vista host OS to go into sleep mode even whilst VMs are running, this is something I’ve not had much luck with in the past – it would generally refuse to sleep when I closed the lid (but thats not a scientific comparison… it may have just been bad luck!)
So, the pay-off – 2 copies of Outlook (2003 and 2007) seemingly running on the same desktop, alt-tab works ok and you have access to all the functionality of both without having to switch between or run multiple OWA sessions and from a security perspective it’s not really any different from having 2 physical PCs in front of you (slight memory overhead, but my laptop has 4Gb RAM, so not a huge issue).
Opening attachments is obviously going to be a bit of an issue, as you’ll technically need an individually licenced instance of Office 2003 in each VM as they can’t (yet) exchange data between them… and that would compromise the security principal.
VMWare Server Performance – A Practical Example
Posted by on April 19, 2008
The following screen dump is from an HP DL380G5 server that runs all the core infrastructure under VMWare Server (the free one) for a friend’s company which I admin sometimes.
It is housed in some co-lo space and runs the average range of Windows servers used by a small but global business, Exchange SQL, Windows 2003 Terminal Services.
As a result of some planned (but not very well communicated!) power maintenance the whole building lost power earlier today, when it was restored I grabbed the following screenshot as the 15 or so Virtual Machines automatically booted.
interesting to note that all the VM’s had been configured to auto-start with the guest OS, meaning there wasn’t any manual intervention required, even though it was a totally dirty shutdown for both the host and guest OS’es (No UPS, as the building and suite is supposed to have redundant power feeds to each rack – in this instance the planned maintenance was on the building wiring so required taking down all power feeds for a 5 yearly inspection..)
There are no startup delay settings in the free version of VMWare Server so they all start at the same time, interesting to note the following points..
The blue line that makes a rapid drop is the pages/second counter, and the 2nd big drop (green) is the disk queue length. the hilighted (white) line is the overall %CPU time, note the sample frequency was 15 seconds on this perfmon.
After it had settled down, I took the following screenshot, it hardly breaks a sweat during its working day. there are usually 10-15 concurrent users on this system from around the world (access provisioned via an SSL VPN device) and a pretty heavily used Exchange mail system.
The box is an HP DL380 G5 with 2 x quad core CPUs (8 cores in total) and 16Gb of RAM, it has 8 x 146Gb 15k HDDs in a single RAID 5 set + hot-spare, it was purchased in early 2007 and cost c.£8,000 (UK Prices)
It runs Windows 2003 Enterprise Edition x64 edition with VMWare Server 1.0.2 (yes, its an old build.. but if it ain’t broke..) and they have purchased multiple w2k3 ent-edition licences to take advantage of the virtualisation use-rights to cover the installed virtual OS’es.
It’s been in-place for a year and hardly ever has to be touched, its rock-solidly available and the company have noticed several marked improvements since they P2V’d their old servers onto this platform, as follows;
- No hardware failures – moving from lots of low-end servers (Dell) and desktops to a single box (10:1 consolidation)
- The DL380 has good redundancy built in, but it’s also backed up with a h/w maintenence contract, and they also have a spare cold-standby server to resume service from backups if data is lost.
- Less noise, the old servers were dotted around their old offices in corners, racks etc – this is the main thing they liked!
- Simple access anywhere – using a Juniper SA2000 SSL VPN, its easy to get secure access from anywhere
- Less reliance on physical offices and cheap DSL-grade data communications, now the servers are hosted on the end of a reliable, data centre class network link with an SLA to back it up. if an individual office looses its ADSL connection, no real issue – people pick up their laptop(s) and work from home/starbucks etc.
- Good comms are cheaper in data centres than in your branch offices (usually)
Hopefully this goes to show the free version of VMWare’s server products can work almost as well if budget is a big concern, ESX would definitely give some better features and make backup easier, they are considering upgrading and combining with something like Veeam Backup to handle failover/backup.
HP Rapid Deployment Pack – PXE Settings for Deploying Windows OS
Posted by on April 17, 2008
The followign screens show a working configuration from the RDP 3.80 PXE Configuration Manager
Have had lots of problems with this deploying Windows OS’es and VMWare ESX 3.5 onto an HP c7000 Blade chassis, still not resolved all the problems, but this definitely works for deploying Windows!
The documentation reads like you should always use the Linux PE configuration and it handles switching between WinPE/LinuxPE depending on which OS job you drop on a target. in my experience this doesn’t work and you need to manually change the PXE configuration to default to LinuxPE or WinPE depending on the OS you want to target.
And
Still a work in progress as I have a c7000 to which I want to deploy a mix of Windows and ESX/Redhat OS’es….
I did get a previous installation to install ESX 3.5 by hacking the default ESX 3.02 job, but its since been re-installed and I can’t do it now
RDP 6.90 seems to list Windows 2008 and ESX 3.5 in the quickspecs, but I’ll be damned if I can find where to download it, going to have to call HP methinks!
As I’ve posted before installing via iLo is just a non-starter if you really do want a flexible and fast deployment configuration – so it has to be RDP.
More later…
VMWare Workstation 6.5 Beta Open Now
Posted by on April 2, 2008
As of 1st April (not an April fools joke!) you can register for, and download the VM Worksation 6.5 beta programme here..
Main new features are;
- Use Unity to integrate your guest apps with your host
- More Powerful VM Record and Replay
- Support for Smart Cards & Smart Card Readers
- Enhanced ACE Authoring
- Link State Propagation Networking
- Improved 3D graphics Support
I’m most looking forward to being able to use Unity… very cool, have used it on the Mac with Fusion and Parallels equivalent, maybe now I will be able to run 2 instances of full-fat Outlook 2007 side by side, plugged into different Exchange mailboxes and orgs (maybe a bit overkill – but I have my reasons!)
How to Monitor VMWare ESX Servers from Microsoft System Center Operations Manager 2007 (SCOM)
Posted by on March 26, 2008
Virtualization, Cloud, Infrastructure and all that stuff in-between 
