Cyber Backup Consistently Failing Recently
I am at my wits end troubleshooting this.
Issue: Backups are failing for 3 Linux cPanel servers at one location all going to the same backup storage server with the error "Backup has failed due to errors on storage side or a network connectivity issue."
Setup: The backup storage server (abgw) that hosts the backups has 8 different servers linked to it out of which backups are only failing for 3 servers. The storage server and the servers that are to be backed up are all in a local network within the same datacenter with no firewall between them.
Troubleshooting: Extensive troubleshooting has been done over a period of 2 weeks. The connectivity has been determined to be ok with using the Connectivity Verification Tool from Acronis, which tested and showed no connectivity issues between any components.
The backup starts and backup size also goes on increasing however it fails at random intervals sometimes at 34%, sometimes at 40 or even at 80%.
Completely uninstalling the agent and removing the server and adding it back again has also been tried, but it resulted in the same issue. Deleting all old backups and creating a full new backup has also been tested with no success.
Both servers can reach each other and the Acronis cloud portal without any issues. ABGW and Agent are both updated to the most latest version. The problem started occurring 2 weeks ago with backups running flawlessly for over 3 months prior to that.
All logs are attached below.
I run the storage server myself, it is running the vstorage-abgw service.
There is no details as to what is the error in accessing the backup file on the storage. Network connectivity checks out just fine. Smartctl drive checks, fine. Raid status fine. FSCK on the system, fine.
Hello Jay.
There is no details as to what is the error in accessing the backup file on the storage.
In the psc.log the very first line indicates that the software can't open a file on that storage:
2020-08-28T09:12:57:185-04:00 140156779108096 I00000000: service_process(21750): io#1: io#1rq#286: readfile = {.suffix = '1', .name = 'uksrv1.websiteserverbox.com-cPanel-Weekly-SaturdayA.tibx',
.offset = 0x264287b000, .length = 8192, .lock_id = 0x100000003e071}, astor_client = 1The mms.log also refers to this path:
| error 0x29b138d: Input/output error
| line: 0x30ba355f9fd4fffe
| file: d:/419/core/resizer/archive3/utils.cpp:429
| function: CoroutineFunc
| function: archive_stream_write_shbuf
| path: /1/uksrv1.websiteserverbox.com-cPanel-Weekly-SaturdayA.tibx
| $module: disk_bundle_lxa64_23140
|
| error 0x29b0006
| line: 0x30ba355f9fd4fffe
| file: d:/419/core/resizer/archive3/utils.cpp:429
| function: CoroutineFunc
| function: astor_file_append
| path: astor://uk-cdp2.websiteserverbox.com:44445/34#754118::/1/uksrv1.websiteserverbox.com-cPanel-Weekly-SaturdayA.tibx
| $module: disk_bundle_lxa64_23140
vstorage-abgw
This is Acronis Cyber Infrastructure's storage. Here in Acronis Cyber Backup 12.5 we also do not have access to the logs on that side.
I redirect your thread to the respective Acronis Cyber Infrastructure forum.
I am a service provider. I am running my own Acronis Backup server, which is running the vstorage-abgw gateway, the backup file is present and verified to be there in that path. I have also attempted to delete that file and create a new and fresh backup, same results,.
Attached is the abgw log just before the backup starts to fail.
Any help is greatly appreciated.
The archive looks corrupted:
04-09-20 16:44:12.390 s#692.r#2784960: file size is greater than last sync offset (822800547840 and 822779576320); discarding unsynced data
04-09-20 16:44:12.390 s#692.r#2784960: failed to truncate '/34#754118/1/uksrv1.websiteserverbox.com-cPanel-Weekly-Saturday-509109EF-5E9A-4637-95C7-3DB2C85771CBB.tibx' to last sync offset: -6 (Misc error while local file io)
04-09-20 16:44:12.390 s#692.r#2784960: completed with err = -6 (Misc error while local file io)
Please check it by running following command on any node in ABGW cluster:
archive_ctl --astor 127.0.0.1 --cert /mnt/vstorage/vols/acronis-backup/certs/abgw.pem -m readonly -f /34#754118/1/uksrv1.websiteserverbox.com-cPanel-Weekly-Saturday-509109EF-5E9A-4637-95C7-3DB2C85771CBB.tibx -s
There is no details as to what is the error in accessing the backup file on the storage.
Best regards,
Maria Belinskaya | Acronis Forum Support Specialist
Information provided AS-IS with no warranty of any kind.
To contact support, please follow http://www.acronis.com/en-us/support/