Skip to main content

[fixed] vSphere 6.0 critical CBT issue

Thread needs solution
Acronis Program Manager
Posts: 22
Comments: 3742

Dear Acronis customers,

There was a major problem discovered by VMware which affects Changed Block Tracking (CBT) functionality on vSphere 6. It has been discovered and documented by VMware in KB2136854 . 

This problem affects all VMs running on ESXi 6.0: the incremental backups of these VMs may not be recoverable. There are no viable workarounds for this problem except for either downgrading ESXi host down to 5.5 version or disabling CBT option in backup settings of Acronis Backup (Advanced) for VMware. Note that taking incremental backups will take longer with CBT option disabled, however the size of them will still be "incremental".

[Fixed]: VMware has released a patch with fix for this issue. See KB2136854 and KB2137546.

Thank you.

--

Best regards,

Vasily

Acronis Virtualization Program Manager

0 Users found this helpful
Beginner
Posts: 7
Comments: 5

Hi Vasily,

To clarify, does this affect ALL vSphere 6 hosts, or only VMs on hardware version 11? The reason I ask is because everything else I have read says it's all vSphere 6 hosts and the only fix is to disable CBT or go back to vSphere 5.5. The VMware article sounds like you have to downgrade the VM hardware version from 11 to 10 in order to be able to run the VM on vSphere 5.5. Can you please confirm?

Thanks,

Andrew

Acronis Program Manager
Posts: 22
Comments: 3742

Andrew,

Thank you for the note - apparently you're right and any VM running on ESXi 6.0 is affected regardless of the virtual hardware version, so the only viable option is to disable CBT when doing backups. 

P.S. Looks like the VMware KB article was changed a couple of days ago (on 17th of Nov) and probably I've misread the previous version while making this sticky post.

Thank you.

--

Best regards,

Vasily

Acronis Virtualization Program Manager

Beginner
Posts: 7
Comments: 5

Acronis Program Manager
Posts: 22
Comments: 3742

Andrew, thank you for the note. I've corrected the original post and added link to respective KB.

JoV
Beginner
Posts: 5
Comments: 7

Hi Vasily

Once the patch has been applied to ESX, can we then just re-enable CBT against the Acronis backups? Or does the ctkEnabled setting need to be disabled in the VM Properties in vSphere, a backup taken and then re-enabled?

Thanks

Jo

Acronis Program Manager
Posts: 22
Comments: 3742

Hi Jo,

The VMware patch fixes wrong reporting of sectors range between 2 CBT records, therefore after applying the patch it should be enough to simply create a new full backup (into new archive for example). This will ensure that all following incremental backups will be correctly tracking the changes. There should be no need to re-enable or reset CBT from what I can see since it logically doesn't make sense (according to patch description) and plus there is no such recommendation from VMware.

Thank you.

--

Best regards,

Vasily

Acronis Virtualization Program Manager

JoV
Beginner
Posts: 5
Comments: 7

Hi Vasily

Further to my question above, we have received the following advice from the Veeam forum:

VMware has released the patch for the latest vSphere 6.0 CBT issue. We have completed a few rounds of testing using the same lab setup that made the original issue easily reproducible, and the patch appears to resolve the issue when installed on ESX 6.0 Update 1a build 3073146. However, we found that simply installing the patch is not sufficient, and CBT reset is additionally required for all of your VMs. This is because existing CBT maps may already contain issues, which may result in inconsistent full backups in future. Having CBT reset will also make the following job run perform "full scan" incremental, thus fixing any existing inconsistencies in backups and replicas. Provided CBT reset has been performed, Active Full backup is not required. However, performing Active Full backups by itself cannot be considered as a substitute to CBT reset with this particular issue. I repeat, CBT reset is mandatory in every scenario.

Acronis Program Manager
Posts: 22
Comments: 3742

Hi Jo,

I believe Veeam is referring to CBT* (so-called "CBT with star") functionality which depends on CBT and allows to retrieve a full list/map of occupied sectors for a given virtual disk and is used when capturing full backups. In Acronis implementation we don't use CBT* feature and we retrieve the list of occupied sectors by analyzing file system inside the virtual disk snapshot instead of relying on VMware API, so in our case it's not necessary to reset CBT before capturing new full backup (new full backup is still required as I mentioned in my previous comment). The subsequent incremental backups will be tracking changes between the 1st CBT record (appeared after capturing full backup) and newer CBT records which correspond VMware snapshots taken for new incremental backups.

P.S. To summarize: CBT* allows to retrieve list of all occupied sectors. CBT allows retrieve list of changed sectors between 2 snapshots (2 CBT records).

P.P.S. Resetting CBT for any machine won't do any harm, so it can be done to be 100% confident that the backups are ok.

Thank you.

--

Best regards,

Vasily

Acronis Virtualization Program Manager

JoV
Beginner
Posts: 5
Comments: 7

Thank you for clarifying Vasily

Jo

Beginner
Posts: 7
Comments: 5

Something still seems wrong, even after applying the patch. My backups now take much longer than before, and the "bytes transmitted" and "speed" counters are very high, as if it is doing a full backup each time. I've got it set to only keep 5 days of backups and CBT is enabled. I am seeing this issue on 2 separate appliances on 2 separate hosts. I have deleted all recovery points from my archives. Do I need to create an entirely new archive? I assume that with the appliance this means I will need to create and setup new local disks? (The appliances have a 2TB locally attached disk with is the RAID array in the host server).

Acronis Program Manager
Posts: 22
Comments: 3742

Hi Andrew,

The symptoms you describe may look like CBT-related, so it's either a problem with specific VMs which have broken CBT which needs to be reset (see https://kb.acronis.com/content/45682 ) or a problem with the archive. Archive file should be re-created as I mentioned in my previous comment (you might need to pick up a new name for it). The backup logs should also show whether there are any issues with CBT - there must be a specific Information record about it.

Thank you.

--

Best regards,

Vasily

Acronis Virtualization Program Manager