Virtualization, Cloud, Infrastructure and all that stuff in-between

Microsoft’s volume snapshot service is pretty handy right? quick hardware independent snaps of a file system – all free and out of the box, well it’s now officially saved my bacon…. whilst it’s a bit klunky (more on this in a bit) it was damned useful.

I had a pain of a problem to deal with this weekend, helping out a friend doing some server re-organising (plan was to migrate these guys from VMWare Server 1.x to ESXi – but didn’t get that far due to some other Windows issues that took all of our time as we checked everything was ok before the move)

Firstly, if you use DFS-R (as comes with Win2003 R2) never, ever, ever, ever use the “distributed file system” applet to administer DFS, we needed to add a new replica of a large DFS-R set to another server and because (in our defence) the server was a fresh R2 install, we forgot to install the newer DFS-R components via control panel, but original DFS was still installed by default and we were in a hurry (read: not paying attention) we used the “Distributed File System” applet to add a new target, and followed the wizard which actually re-created the DFS volume (note to self – pay more attention when clicking!) from scratch.

It proceeded to delete all the contents of all the DFS shares and moved them to a folder called NtFrs_PreExisting___See_EventLog and started afresh, that wouldn’t be so bad except for some inexplicable reason it then purged the contents of that folder from all replicas so we had no quick cut & paste file copy solution.

This was not going to be a fun weekend.

Don’t use this one

use this one!

So, basically it was our (my) fault – but it was compounded by some weird corruption in one of the directories that looked like it had been there a while that meant recovery wasn’t going to be straightforward.

The data Backup was about 24hrs older than the last VSS snapshot on the central file server (hub & spoke replication topology) so as we now had a flat, deleted DFS volume with no data (thanks!) we decided to try and revert to the most recent VSS snapshot for the relevant directories.

But no dice, it just threw an error – can’t copy, I can view the files and see the contents and can drag and drop one or two a time, but any more and it would throw an error.

Not good, I can only assume that this was because of some logical corruption within the file system as there was one whole directory tree I couldn’t access (more on how I recovered this later).. there were over 60k files so I wasn’t going to do that by hand – so a command line was in order as at least XCopy can ignore errors etc. and just pull out the good data.

I found these excellent articles here and here and documentation here but some of them were more geared towards taking a snapshot and extracting data in-situ rather than from a persistent snapshot like you get with VSS.

so, none of them worked for me ; and even a lot of hacking with Vshadow and MOUNTVOL I couldn’t get the VSS Snap to mount at all and time was short

I did discover the following though, if you view a snapshot using the Previous Versions tab (remember this only works if you browse for files to restore via UNC path) it opens the snap in Explorer, but you can’t map a drive to it or run a command line copy against it…. or can’t you 🙂

When you open it in explorer this way it does create a sort of hidden temporary share – easiest way I found to expose the name of the share was to try and zip a file in the explorer session that is looking at the snapshot using WinZip, if you follow the wizard at some point it will expose a UNC path like \\SERVERNAME@GMT-DD-MM-YY-{GUID} if you can cut & paste that you can then map a network drive to it

NET USE * \\servername@gmt-dd-mm-yy-{guid}

And you can then run xcopy etc against that mapped drive to copy out all the good data – in reality we used SyncBackSE – which is great for complex file copies and we already had it installed.

All of these Windows servers were installed as VM’s in VMWare Server(s), so it actually made our lives a lot easier as we could quickly clone a known-broken server as-is (do no further harm) and then spin it up disconnected from the network to recover data using this method and also undeleted files using Get Data Back NTFS etc. and then use that data to re-seed the DFS-R volume – but much easier than if it were a physical box and at no real risk of making things worse.

So, in conclusion this was human error, rather than a 100% technical problem and should have been better planned and prevented by maintenance and a better recovery plan- but here it is, with the solution we found to get things back in all its gory details… and mainly as a footnote so I don’t make the DFS mistake again and in my defence this is a shoe-string charity operation rather than a blue-chip org with significant money and time to invest in such efforts.

This solution worked for us, but you need to have your own tried & tested solution – don’t rely on this as far as I can tell it’s unsupported, use at your own risk!