what is local disk size limit in Acronis Backup Agent Appliance
what is local disk size limit in Acronis Backup Agent Appliance (ESXi appliance)? We used vmkd file to the appliance and configured as local disk. We assigned 3 TB in size and we are able to see the Agent console detected 3 TB. However, we are getting "there is not enough disk space". When we check on the agent console, we have enough disk space:
# df -h
Filesystem Size Used Available Use% Mounted on
/dev/sda1 19.2G 19.0G 165.9M 99% /
/dev 1.9G 8.0K 1.9G 0% /dev
tmpfs 1.9G 32.0K 1.9G 0% /run
/dev/loop0 17.0M 17.0M 0 100% /lib/modules
/dev/loop1 25.0M 25.0M 0 100% /lib/firmware
/dev/sdc1 3.0T 2.0T 779.7G 73% /mnt/local_storages/D:
# fdisk /dev/sdc1 -l
fdisk: device has more than 2^32 sectors, can't use all of them
Disk /dev/sdc1: 2199.0 GB, 2199023255040 bytes
255 heads, 63 sectors/track, 267349 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdc1 doesn't contain a valid partition table
Can you please clarify which exact operation returns "there is not enough disk space" error? Note that 'fdisk' utility cannot be used for management of >2TB volumes so outputs from it are not of concern. The >2TB disks are initialized as GPT ones and are formatted into ext4 file system through appliance local GUI, so your initial configuration steps seem to be right, while the error might be caused by something else (need to understand the context of the operation first).
I got the "there is not enough disk space" when a scheduled backup plan was executed. I noticed that the management console showing was enough free space for any VM in the backup plan. Anyway, after I rebooted the agent appliance and unmount/remount the local storage, it's working now.
unfortunately, I got following error today, "database or disk is full" and a backup stuck at 100%.
Failed to check out the resource with ID '/vmwesx/host-44673/vm-214/[NAS04B_SAS_2] bwxsp-03.mgmt.comwave/bwxsp-03.mgmt.comwave.vmdk' from the repository.
Error code: 60
Message: Failed to check out the resource with ID '/vmwesx/host-44673/vm-214/[NAS04B_SAS_2] bwxsp-03.mgmt.comwave/bwxsp-03.mgmt.comwave.vmdk' from the repository.
Error code: 11
Message: ASYNC: Exception has occurred during work execution.
Error code: 4
Message: ASYNC: Execution flow control job.
Error code: 4
Message: Statement 'COMMIT' cannot be executed. Error code: 13.
Error code: 17
Message: An error occured while using database '/var/lib/F4CEEE47-042C-4828-95A0-DE44EC267A28.db3'.
Error code: 13
Message: database or disk is full
turn out I found the Appliance / disk went out of space. When I looked further, I found most of the space were filled by "Activity" files in "/var/lib/Acronis/BackupAndRecovery/MMS/Activity". What can I do with to free up the disk and stop from happening? The '/' disk of this appliance already has 20 GB instead of the default 5 GB and 17 GB used by the "Activity" folder.
The activities logs are rotated within 1 year period and normally contain only text files which are very small in size. 17GB is definitely not normal size for these logs unless appliance was running for 10+ years executing backups every 5 minutes. It makes sense to contact our support team and analyze why there is such huge size for these logs by getting them exported from appliance. You can connect to appliance via WinSCP (as described in "Connecting using WinSCP utility" section in https://kb.acronis.com/content/61859) and copy the contents of the /var/lib/Acronis/BackupAndRecovery/MMS/Activity folder for further analysis with our support team. These files can be safely deleted from appliance permanently and it won't affect other operations.
I already contacted support and i was told to update the VM's vmdk files because the support want the logs on the appliance but unable to tell the exact location. Anyway, it's good to now that the logs are rotated within 1 year. It will be nice if Acronis can build a internal monitoring to 1. reserve some space for working (so that it won't crash any process); 2. giving some warning more specific instead just saying out of space; 3. set the log retention or purge it as needed.
I am getting this same problem all the time. A backup of guests won't complete with the disk is full error and then I have to delete the Appliance and redeploy.
It's incredibly annoying.
Having appliance overfilled with logs is not normal situation and to properly address the issue we need to understand which particular logs are consuming so much space, e.g. it should be investigated with our support team involved.
There are multiple Acronis modules which write independent logs and all have rotation logics included when the logs increase some threshold (10MB for "snapapi" module logs for example), then they are archived into .zip automatically and get permanently deleted after 1 year. Packing text files in .zip achieves very good compression ratio, so the logs are not taking much space under normal circumstances.
In your cases there could be some faulty module which violates the designed retention logics and this has to be properly analyzed (reveal which particular module is causing the issue).
FYI... I've uploaded the whole VM files (acronis backup agent appliance) to the cftp.acronis.com as per requested from your support. In case you are interested to look at it, the case number is 03527724 and the zipped file name is Case_03527724.zip. Thanks,
The logs are growing due to unexpected garbage error message which appears when appliance guest OS (Linux) is trying to initialize the local disks on the SCSI controllers. There seems to be some specific in the locally attached storage disk in your case (which is >2TB and probably has some other specifics) which in turn causes such behavior. Faulty lines spammed in the logs:
2018-10-23T02:23:17:642-04:00 139779079350016 C0000001B: Assertion failed: !"Unknown scsi host type", file e:/232/enterprise/virtualware/raw/vmware/scsi/impl/host.cpp, line 105
2018-10-23T02:23:17:644-04:00 139779079350016 C0000001B: /bin/libDiskBundle.so ( + 0x84cf13) [0x7f2121908f13]
These are the problematic messages which are overfilling the logs under /var/lib/Acronis/BackupAndRecovery/MMS/Activity/ folder. These logs can be safely removed completely (all files) as a temporary measure.
For futher troubleshooting we need to understand the type of SCSI controllers discovered by the appliance guest OS. This can be done via the following command executed from local appliance UI (from command line interface):
Where X varies from 0 to [number of SCSI host adapters], in my below example it's from 0 to 3:
The expected (by Acronis engine) SCSI host adapters classes names are bolded below:
If the class name for SCSI controller on appliance does not match one of the expected ones this results in error quoted above and it leads to unexpected growth of the log files.
Please check the classes names in your environment (it cannot be checked by us, since it's tied not to appliance VM itself which I retrieved the logs from, but rather to specifics of the particular disks attached in real environment) and send this information to our support engineer.
P.S. I will also coordinate these troubleshooting steps with our support team internally.
please see attached output from the appliance. It's 100% the same as yours
the local attached disk in the appliance, in fact, a vmdk. Therefore, it's using the same scsi controller as the appliance. I didn't customize any VM's setting other than adding an extra 3 TB HDD as local storage
Thank you for the details. The issue is definitely related to locally attached storage used, however our first theory (specifics of SCSI host adapter) didn't confirm, so there should be extended logging added for further troubleshooting. This can be done by our support team (our maintenance developers). Please continue working with our support engineers - I've passed them all required technical details related to the case.