27322: ABR 10 Virtual Edition - Volsnap 27, Disk 51 and FilterManager 3 errors thrown during nightly VM backups.

No replies
Matt Sleno
Offline
Beginner
Joined: 2011-12-02
Posts: 1

Problem

Volsnap 27, Disk 51 and FilterManager 3 errors thrown during nightly VM backups.

Environment

Host - Server 2008 R2 Enterprise SP1
VMS - Three in total, the two that throw errors are a 2008 R2 SP1 and a 2003 R2 SP2
Backup Software - Acronis Backup and Recovery 2010 Virtual Edition (latest build)
Backup Device - QNAP NAS with 2TB capacity

Explanation
We are imaging the VMs on a nightly basis and all internal logs for the Acronis software show successful backups with no errors. However on the Host server we are receiving the following errors / warnings when two out of the three machines finish their snapshots

Backups finish and a "snapshot complete" is logged in Acronis - at an identical time on the host server we get:

Level - Source - EventID - Message

Information - volsnap - 64 - Volume \\?\Volume{bc724979-d76b-11e0-82d4-18a9053c4895} is being reverted to the state of a previous shadow copy.

Information - volsnap - 66 - The reverting of volume \\?\Volume{bc724979-d76b-11e0-82d4-18a9053c4895} to the state of a previous shadow copy is complete.

Warning - Disk - 51 - An error was detected on device \Device\Harddisk2\DR20 during a paging operation. This can occur multiple time - I've seen up to 15 at a time, ususally 3.

Error - volsnap - 27 - The shadow copies of volume \\?\Volume{bc724979-d76b-11e0-82d4-18a9053c4895} were aborted during detection because a critical control file could not be opened.

Sometimes we also receive this:

Error - FilterManager - 3 - Filter Manager failed to attach to volume '\Device\HarddiskVolume32'. This volume will be unavailable for filtering until a reboot. The final status was 0xc03a001c.

This then repeats for the second VM with different volume GUIDs and DRXX numbers.

Additional Notes

• No errors or warnings appear in the guest VMs.
• During the backups the Virtual Disk Service and the Volume Shadow Copy Service seem to constantly stop / start every 3-5minutes (System Log). On the 2003 VM we are also seeing the Logical Disk Manager Administrative Service starting and stopping during the backups, again every few minutes (System Log)
• On the 2008 VM, just prior to the above errors / warnings the following appears:
Information - VSS - 8220 - Ran out of time while deleting files. Operation: OnPostSnapshot event PostSnapshot EventContext: Execution Context: Shadow Copy Optimization Writer Execution Context: Writer Writer Class Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}

Solutions Tested (with no resolution)

1) Cleaned up old hidden volume binds in the registry

2) Found that hotfix http://support.microsoft.com/kb/982210 is not applicable to this server

3) Disabled AV on Host server

4) Restarted the Host server

5) Disabled Shadow Copies on the Host drive that contains the VHDs

6) Upgraded the integration services on all VMs

7) chkdsk /F has been run on the host drive where the VHDs are stored

8) chkdsk /F has been run internally on the guest VM's (they only have one drive each)

9) Added Registry Key HKey_Local_Machine\ SOFTWARE\Microsoft\Windows NT\CurrentVersion\Virtualization, New Reg_DWORD, Name "VssRevertTimeout", Value 360 (decimal).
On the Host.
[ See Sources *1 *2 *3]

10) Added Registry Key "HKey_Local_Machine\Software\Microsoft\Windows NT\CurrentVersion\SPP"
DWord : CreateTimeout
Value decimal: 1200000    (20*60*1000 = 20 mins).
On the Host.
[ See Sources *1 *2 *3]

Sources

*1 Windows Server Backup Forum post referencing the same note errors when using Windows Server Backup: http://social.technet.microsoft.com/Forums/en-US/windowsbackup/thread/2a...

*2 Secondary Windows Server Backup thread confirming above values: http://social.technet.microsoft.com/Forums/en/windowsbackup/thread/313b4...

*3 MS Blog triple confirming above values: http://blogs.technet.com/b/asiasupp/archive/2011/08/01/windows-server-ba...

So - were do we head from here?

Cheers

Greg