Skip to main content

Corrupted .tib file

Thread needs solution
Beginner
Posts: 2
Comments: 1

Hey

As i'm on the lastest software (Acronis Backup for VMware Agent (build 9.2.10535)) and i'm still getting the same corrupted .tib file again and again, I'm making this post so you could help me find the fault(s).

Details:

  • vSphere 5.1
  • AB for VMware (build 9.2.10535)
  • Deduplication = ON
  • CBT backup = ON
  • Backup to NAS (LaCie 2big Network 2: Firmwareversion 2.2.12.2)

Error code:

It's slightly different every time but the end result is a corrupted .tib file. So corrupted that AB can't even find the file (for expl. validate archive).

Little sketch:

Day 1:

Back-up completed succesfull for serveral days!

Day 2:

VMware_VDDK: Insufficient permissions in the host operating system
Additional info:
--------------------
Error code: 208
Module: 83
LineInfo: c61573f663f5d76f
Fields: code : 3014
Message: VMware_VDDK: Insufficient permissions in the host operating system
--------------------

Day 3:

Task 'Back up to 'Network'' failed: 'Failed to create a backup.
Additional info:
--------------------
Error code: 3
Module: 435
LineInfo: 555b5abba0950340
Fields:
Message: Failed to create a backup.
--------------------
Error code: 32786
Module: 114
LineInfo: 28314c961de7d337
Fields:
Message: Failed to prepare for backing up.
--------------------
Error code: 353
Module: 149
LineInfo: a71592046cb2c5f6
Fields:
Message: Failed to back up the group.
--------------------
Error code: 2
Module: 218
LineInfo: 338a407ad20e0987
Fields:
Message: Error occurred while running the backup and recovery engine.
--------------------
Error code: 1080
Module: 1
LineInfo: d1ab7fa1e56ec8bf
Fields:
Message:
--------------------
Error code: 13
Module: 149
LineInfo: d1ab7fa1e56eca4e
Fields:
Message: Failed to perform the requested operation.
--------------------
Error code: 103
Module: 83
LineInfo: a859dd78cc91df40
Fields:
Message: Failed to open the virtual machine ([datastore1 (2)] ns419a (7)/ns419a (7).vmx).
--------------------
Error code: 253
Module: 83
LineInfo: c7610e0a857bedf4
Fields:
Message: VMware error: 'Remote method call failed.'.
--------------------
Error code: 32
Module: 0
LineInfo: c7610e0a857bedf4
Fields:
Message: Awaiting task 'CreateSnapshot' has failed. Reason: An error occurred while quiescing the virtual machine. See the virtual machine's event log for details..
--------------------'.

Day 4: +- 5 different messages saying the archive is corrupted

Failed to perform the operation with archive 'avfs:/smb?//192.168.1.12/vmbackup/Archive.TIB'. Error: 'Write error.'.
Additional info:
--------------------
Error code: 4
Module: 7
LineInfo: 9d2de1f088cbc4d7
Fields:
Message: Failed to perform the operation with archive 'avfs:/smb?//192.168.1.12/vmbackup/Archive.TIB'. Error: 'Write error.'.
--------------------
Error code: 13
Module: 4
LineInfo: 1f40bc5f43e9adeb
Fields:
Message: The file is corrupted.
--------------------

Solution:
Super dirty but the only thing that works.
Going to NAS and renaming the "Archive.tib" file to "Archive.old" so AB makes a new file.

Notes:

  • This is going on for quite a while, first I tought it was because of the .tib file size. When it was reaching 750-800gig in size it "happend".
  • After a while it even happend on a 300gig file.
  • When it happens its always the "ns419a" server (SBS 2011 with Exchange 2010 on it)

Thoughts:

Is it possible these problems are occuringing because of drives in the NAS are faulty, this week 1 drive has failed, so maybe replacing both of them could solve this problem.
The drive status on both drives (provided by LaCie) was always "OK" until the failure this week ofcourse.

Thanks, let's hope we can solve this problem!

 

0 Users found this helpful
Acronis Program Manager
Posts: 22
Comments: 3742

Hi,

The main problem according to your description is the actual backup location (LaCie NAS) which probably has either faulty disks or  some other problem. To localize the issue it makes sense to try saving the backup to some other location (for example some Windows-based share on an external USB drive attached to this Windows machine) and monitor it for several days to see whether the problem appears once the archive size reaches 300-700GB. This check would let us localize the problem.

Concerning the other errors you mentioned:

1) "VMware_VDDK: Insufficient permissions in the host operating system" - it can be safely ignored as explained in my reply here http://forum.acronis.com/forum/56385#comment-174391

2) "Awaiting task 'CreateSnapshot' has failed. Reason: An error occurred while quiescing the virtual machine. See the virtual machine's event log for details." - this is a typical quiesced snapshot failure which may appear randomly. The problem is that the affected VM cannot be snapshotted by VMware (could happen at peak load hours for example) itself and thus the backup of it is impossible. To check it you can create the snapshot manually from vSphere client to see if it works or not (snapshot with quiesced=on, memory=off parameters). The general rule is as follows: when snapshot works - backup works too, when snapshot fails - backup will fail too.

Thank you.

--

Best regards,

Vasily

Acronis Virtualization Program Manager

Beginner
Posts: 2
Comments: 1

Hey Vasily

I installed two brand new WD RED 2TB HDD (raid 1), first 2 days went ok. On the third day I got the following errors:

starting at 01.00, backup begins at 0u30. Errors end at 01u05.

1--------------------------------------------------

Failed to perform the operation with archive 'avfs:/smb?//192.168.1.12/vmbackup/Archive.TIB'. Error: 'Write error.'.
Additional info:
--------------------
Error code: 4
Module: 7
LineInfo: 9d2de1f088cbc4d7
Fields:
Message: Failed to perform the operation with archive 'avfs:/smb?//192.168.1.12/vmbackup/Archive.TIB'. Error: 'Write error.'.
--------------------
Error code: 13
Module: 4
LineInfo: 1f40bc5f43e9adeb
Fields:
Message: The file is corrupted.
--------------------

2--------------------------------------------------

The operation 'Backing up' has failed.
Additional info:
--------------------
Error code: 26
Module: 100
LineInfo: 48afbd3608a410ca
Fields:
Message: The operation 'Backing up' has failed.
--------------------
Error code: 1080
Module: 1
LineInfo: b43e776571144dee
Fields:
Message: Commit of operations result is failed.
--------------------
Error code: 4
Module: 7
LineInfo: 9d2de1f088cbc4d7
Fields:
Message: Failed to perform the operation with archive 'avfs:/smb?//192.168.1.12/vmbackup/Archive.TIB'. Error: 'Write error.'.
--------------------
Error code: 13
Module: 4
LineInfo: 1f40bc5f43e9adeb
Fields:
Message: The file is corrupted.
--------------------

3--------------------------------------------------

VMware_VDDK: Insufficient permissions in the host operating system
Additional info:
--------------------
Error code: 208
Module: 83
LineInfo: c61573f663f5d76f
Fields: code : 3014
Message: VMware_VDDK: Insufficient permissions in the host operating system
--------------------

4--------------------------------------------------

Internal backup error: 458,784 (The archive is corrupted.).

5--------------------------------------------------

The operation 'Backing up' has failed.
Additional info:
--------------------
Error code: 26
Module: 100
LineInfo: 48afbd3608a410ca
Fields:
Message: The operation 'Backing up' has failed.
--------------------
Error code: 1080
Module: 1
LineInfo: 145a191bfe9050d1
Fields:
Message: Failed to execute the operation.
--------------------
Error code: 1080
Module: 1
LineInfo: bb03ce40e9a37fb0
Fields:
Message: Failed to open archive.
--------------------

6--------------------------------------------------

Internal backup error: 458,784 (The archive is corrupted.)

7--------------------------------------------------

The operation 'Backing up' has failed.
Additional info:
--------------------
Error code: 26
Module: 100
LineInfo: 48afbd3608a410ca
Fields:
Message: The operation 'Backing up' has failed.
--------------------
Error code: 1080
Module: 1
LineInfo: 145a191bfe9050d1
Fields:
Message: Failed to execute the operation.
--------------------
Error code: 1080
Module: 1
LineInfo: bb03ce40e9a37fb0
Fields:
Message: Failed to open archive.
--------------------

8--------------------------------------------------

VMware_VDDK: Insufficient permissions in the host operating system
Additional info:
--------------------
Error code: 208
Module: 83
LineInfo: c61573f663f5d779
Fields: code : 3014
Message: VMware_VDDK: Insufficient permissions in the host operating system
--------------------

9--------------------------------------------------

Task 'Back up to 'Network'' failed: 'Failed to create a backup.
Additional info:
--------------------
Error code: 3
Module: 435
LineInfo: 555b5abba0950340
Fields:
Message: Failed to create a backup.
--------------------
Error code: 32786
Module: 114
LineInfo: 28314c961de7d337
Fields:
Message: Failed to prepare for backing up.
--------------------
Error code: 353
Module: 149
LineInfo: a71592046cb2c5f6
Fields:
Message: Failed to back up the group.
--------------------
Error code: 2
Module: 218
LineInfo: 338a407ad20e0987
Fields:
Message: Error occurred while running the backup and recovery engine.
--------------------
Error code: 1080
Module: 1
LineInfo: 145a191bfe9050d1
Fields:
Message: Failed to execute the operation.
--------------------
Error code: 1080
Module: 1
LineInfo: bb03ce40e9a37fb0
Fields:
Message: Failed to open archive.
--------------------'.

END----------------------------------------------------------------------------------------------------

I hope that we can find a solution.
The only thing I haven't tried yet is rebooting the vmware host. Will do this tonight.

 

Acronis Program Manager
Posts: 22
Comments: 3742

Hi,

This is still a problem accessing the backup location on the network share (LaCie NAS) - the archive was detected as corrupted on the 3rd day. To confirm whether it is really corrupted or there is a problem with share access you can try moving the .tib file to some other location and then run archive validation. Likely it will succeed, but if not then this means that archive is corrupted and needs to be re-created. It may also be a random issue (caused by network collisions), so on 4th day the backups may again be successful.

Thank you.

--

Best regards,

Vasily

Acronis Virtualization Program Manager

Beginner
Posts: 0
Comments: 7

Having same problem(s) here on one (out of two) virtual servers (W2012R2 std) on same VMware host.

Running Acronis Backup for VMware Agent (build 9.2.10571)

/tmp on Acronis appliance has 3.9G free space.
Backup location has 1.8T free space.
.TIB size for 'failing server' ~22G
.TIB size for 'ok server' ~ ~104G

Got the following error messages:

Task 'Backup_DC' failed: 'Failed to create a backup.
Additional info: 
--------------------
Error code: 3
Module: 435
LineInfo: 555b5abba0950340
Fields: 
Message: Failed to create a backup.
--------------------
Error code: 32786
Module: 114
LineInfo: 28314c961de7d337
Fields: 
Message: Failed to prepare for backing up.
--------------------
Error code: 353
Module: 149
LineInfo: a71592046cb2c5f6
Fields: 
Message: Failed to back up the group.
--------------------
Error code: 2
Module: 218
LineInfo: 338a407ad20e0987
Fields: 
Message: Error occurred while running the backup and recovery engine.
--------------------
Error code: 1080
Module: 1
LineInfo: d1ab7fa1e56ec8bf
Fields: 
Message: 
--------------------
Error code: 13
Module: 149
LineInfo: d1ab7fa1e56eca4e
Fields: 
Message: Failed to perform the requested operation.
--------------------
Error code: 103
Module: 83
LineInfo: a859dd78cc91df40
Fields: 
Message: Failed to open the virtual machine ([HolmeStore] HD-DC/HD-DC.vmx).
--------------------
Error code: 253
Module: 83
LineInfo: c7610e0a857bedf4
Fields: 
Message: VMware error: 'Remote method call failed.'.
--------------------
Error code: 32
Module: 0
LineInfo: c7610e0a857bedf4
Fields: 
Message: Awaiting task 'CreateSnapshot' has failed. Reason: An error occurred while saving the snapshot: Failed to quiesce the virtual machine..
--------------------'.

ACTIONS:

  1. Seemed to be a problem with VMware tools not running (at least not stable). Reinstalled VMware tools OK
  2. Reboot virtual server
  3. Made snapshot & removed snapshot OK.
  4. New backup with error messages:

The operation has failed.
Additional info: 
--------------------
Error code: 26
Module: 100
LineInfo: 48afbd3608a410ca
Fields: 
Message: The operation has failed.
--------------------
Error code: 537
Module: 149
LineInfo: fc2924f06484b5f1
Fields: 
Message: Meta collection during application-aware backup failed.
--------------------
Error code: 358
Module: 149
LineInfo: 8e1d384601b2ab4b
Fields: 
Message: VSS metadata is missing or corrupt.
--------------------
Error code: 13
Module: 4
LineInfo: 86137b9d60c180c7
Fields: 
Message: The file is corrupted.
--------------------

ACTIONS:

  1. Renamed .TIB file
  2. New backup. Same error message.
  3. Backup of the other virtual server on same host - to same directory on network storage: OK
  4. Validation of the renamed .TIB file: OK
  5. Snapshot (mem off, quiesce on) for server with failing backup: OK
  6. Removed shapshot for server with failing backup: OK
  7. New backup. Error messages:

Failed to run the task with ID '9396FFFE-7606-4C5D-BC9C-F1515E0D7DEE' (internal run).
Additional info: 
--------------------
Error code: 41
Module: 144
LineInfo: 5ee2c4bb8ba04bdd
Fields: 
Message: Failed to run the task with ID '9396FFFE-7606-4C5D-BC9C-F1515E0D7DEE' (internal run).
--------------------
Error code: 3
Module: 435
LineInfo: 5ee2c4bb8ba0489f
Fields: IsReturnCode : 1
Message: Failed to create a backup.
--------------------
Error code: 32786
Module: 114
LineInfo: 28314c961de7d337
Fields: 
Message: Failed to prepare for backing up.
--------------------
Error code: 353
Module: 149
LineInfo: a71592046cb2c5f6
Fields: 
Message: Failed to back up the group.
--------------------
Error code: 2
Module: 218
LineInfo: 338a407ad20e0987
Fields: 
Message: Error occurred while running the backup and recovery engine.
--------------------
Error code: 537
Module: 149
LineInfo: fc2924f06484b5f1
Fields: 
Message: Meta collection during application-aware backup failed.
--------------------
Error code: 358
Module: 149
LineInfo: 8e1d384601b2ab4b
Fields: 
Message: VSS metadata is missing or corrupt.
--------------------
Error code: 13
Module: 4
LineInfo: 86137b9d60c180c7
Fields: 
Message: The file is corrupted.
--------------------

ACTIONS:

  1. Reboot of VM with failing backup
  2. Changed backup job to make new backup file name
  3. Re-run of backup

   (got one AD15455C-........847.lck and one AD15455C-........847_backup.lck in destination directory, but no file with specified .TIB file name)
    Error messages:

Task 'Backup_DC' failed: 'Failed to create a backup.
Additional info: 
--------------------
Error code: 3
Module: 435
LineInfo: 555b5abba0950340
Fields: 
Message: Failed to create a backup.
--------------------
Error code: 32786
Module: 114
LineInfo: 28314c961de7d337
Fields: 
Message: Failed to prepare for backing up.
--------------------
Error code: 353
Module: 149
LineInfo: a71592046cb2c5f6
Fields: 
Message: Failed to back up the group.
--------------------
Error code: 2
Module: 218
LineInfo: 338a407ad20e0987
Fields: 
Message: Error occurred while running the backup and recovery engine.
--------------------
Error code: 537
Module: 149
LineInfo: fc2924f06484b5f1
Fields: 
Message: Meta collection during application-aware backup failed.
--------------------
Error code: 358
Module: 149
LineInfo: 8e1d384601b2ab4b
Fields: 
Message: VSS metadata is missing or corrupt.
--------------------
Error code: 13
Module: 4
LineInfo: 86137b9d60c180c7
Fields: 
Message: The file is corrupted.
--------------------'.

ACTIONS:

  1. Reboot of Acronis Appliance
  2. Re-run of backup task with same result as last run.
  3. Reboot of VMware host
  4. Re-run of backup task, same result as last run.
  5. Go to bed, hope it will be OK tomorrow.....

****

Best regards,

Dag-Olav Bjorøy
Senior Consultant
Data-Consult Nesbyen as
 

 

 

Acronis Program Manager
Posts: 22
Comments: 3742

Hi Dag-Olav,

--------------------
Error code: 358
Module: 149
LineInfo: 8e1d384601b2ab4b
Fields: 
Message: VSS metadata is missing or corrupt.
--------------------
Error code: 13
Module: 4
LineInfo: 86137b9d60c180c7
Fields: 
Message: The file is corrupted.
--------------------'.

This error message appears when the vss_manifest***.zip file doesn't contain valid VSS writers output. This file is generated in the VM folder on the datastore after you take a quiesced snapshot of the VM. We use these outputs to take application-aware backups of the VM. Normally the .zip will contain backup.xml + multiple writer[0-N].xml files.

Please see the following KB article: https://kb.acronis.com/content/31347 . Check the 6-7 steps from the Solution section.

Note that the quick workaround is to disable application-aware backup processing in the backup settings - in this case the backup will succeed.

Thank you.

Beginner
Posts: 0
Comments: 7

Thanks Vasily,

I will try this when the customer has left his office today.

The two virtual server on this host are

  1. DC - with AD - failing
  2. DB - with SQLServer - Ok

There are some strange issues with VMware Tools on DC; we have had error messsages while trying to take snapshots. The "Failed to quiesce the virtual machine.." error message has occured 5-6 times during the last month. I also will check if there could be a new build of VMware Tools - if that could help. Certainly, reinstallation of VMware Tools has not helped so far.

Will keep you informed.

Acronis Program Manager
Posts: 22
Comments: 3742

Hi,

The VMware quiesced snapshot failure is typically caused by a combination of multiple factors. In particular it is:

1) High CPU/RAM consumption on the ESXi host where the VM runs

2) Intensive I/O load onto the VM storage (datastore)

3) Intensive I/O load inside the VM (for example intensive DB operations)

That's why it appears randomly and may not strike at certain hours during a day, since these factors may become true only during peak hours or during some specific events (some storage maintenance tasks for example).

In Acronis Backup for VMware there is an option to re-try failed VM processing (Backup Options->Error Handling) which will re-try the quiesced snapshot for pre-defined number of attempts. It makes sense to enable this option since quite often a quiesced snapshot attempt will succeed right after failure if it failed randomly before.

Thank you.

Beginner
Posts: 0
Comments: 7

Hi Vasily,

same problem today - just started off trying to get a backup without doing any changes to the system.

Could not find the vss.....zip file in the vm's directory on the datastore (scr01.jpg) at any time of the backup. Could this be the problem: the vss....zip file is not created?

One possibly interesting detail from the Recent Task window in vSphere (scr02.jpg). Why is Acronis trying to remove the snapshot twice?

Made a snapshot (no mem, quiesce file system), OK (no vss....zip).

Defined a new backup task to different location, same result.

Uninstalled VMware Tools from virtual server. Reboot of virtual server.

Downloaded "Latest" VMware Tools (from jun 2016) and installed on virtual server (with reboot).

Started backup task once more. The "Create virtual machine snapshot" says progress 0% for much longer time than earlier attempts, then flashing over to 100% and 'hanging' there for a while. Now we have got a vss....zip file in the datastore directory of the vm (scr03.jpg)! And the progress bar for the backup slowly moves from 0% in the Acronis web interface. (Since we have asked for a new file, that is ok....)

New .TIB file (new name) is created in backup destination (scr04.jpg).

Backup job succeeded!

Thanks for good ideas to solve the problem!

 

 

 

 

 

 

Attachment Size
370247-130924.jpg 117.41 KB
370247-130927.jpg 70.1 KB
370247-130930.jpg 133.31 KB
370247-130933.jpg 69.31 KB
Beginner
Posts: 0
Comments: 7

Hi Vasily,

same problem today - just started off trying to get a backup without doing any changes to the system.

Could not find the vss.....zip file in the vm's directory on the datastore (scr01.jpg) at any time of the backup. Could this be the problem: the vss....zip file is not created?

One possibly interesting detail from the Recent Task window in vSphere (scr02.jpg). Why is Acronis trying to remove the snapshot twice?

Made a snapshot (no mem, quiesce file system), OK (no vss....zip).

Defined a new backup task to different location, same result.

Uninstalled VMware Tools from virtual server. Reboot of virtual server.

Downloaded "Latest" VMware Tools (from jun 2016) and installed on virtual server (with reboot).

Started backup task once more. The "Create virtual machine snapshot" says progress 0% for much longer time than earlier attempts, then flashing over to 100% and 'hanging' there for a while. Now we have got a vss....zip file in the datastore directory of the vm (scr03.jpg)! And the progress bar for the backup slowly moves from 0% in the Acronis web interface. (Since we have asked for a new file, that is ok....)

New .TIB file (new name) is created in backup destination (scr04.jpg).

Backup job succeeded!

Thanks for good ideas to solve the problem!

Beginner
Posts: 2
Comments: 2

Hello

I would like to contribute my experiences to one of these errors, I also suddenly had problems to quiesce a VM, while all backups before succeeded. Restarting both the VM and the Acronis appliance did not help.

-------------------- 
Error code: 32 
Module: 0 
LineInfo: c7610e0a857bedf4 
Fields: 
Message: Awaiting task 'CreateSnapshot' has failed. Reason: An error occurred while saving the snapshot: Failed to quiesce the virtual machine.. 
--------------------

 

Then I tried to create a Snapshot in vCenter - also failed, so problem was within VMware, probably VMtools. Analyzing of Windows event logs on the VM led to Event ID 7011 and to this thread: https://communities.vmware.com/thread/533736?tstart=0

Check on the VM if the service "VMware Snapshot Provider" is running and stop it. Then try to do the backup again - it worked instantly in my case. The problem is rooted in a bug in the update 5.5u3 in VMtools 10.

Since beginning of June there is a new version of VMtools 10.0.9, which should fix this issue. I have installed it last week and so far did not have this issue again.

 

Best regards,

Ben