ACI Critical Alert

Hello Bill,

Alert 'Chunk replication is blocked or too slow.' appears in case if increase of Prometheus metrics mdsd_cluster_replication_stuck_chunks and mdsd_cluster_replication_touts_total was detected, related documentation on metrics: https://dl.acronis.com/u/software-defined/html/AcronisCyberInfrastructu…

The side symptom  - constant appearing of 'Replication timeout' warning messages in storage cluster event, e.g.:

[root@aci-node-03 ~]# vstorage -c $(cat /mnt/vstorage/.vstorage.info/clustername) get-event | grep -i 'replication timeout' | tail
connected to MDS#3
2021-01-18 23:10:21.608 MDS WRN: Replication timeout on CS#1030 [+...]
2021-01-18 23:11:12.124 MDS WRN: Replication timeout on CS#1026
2021-01-19 00:02:11.045 MDS WRN: Replication timeout on CS#1037

One of the possible reasons may be some network issues, you may check Grafana dashboard and review the graphs of 'Hardware nodes overview' dashboard for network errors detected on nodes interfaces.

In case if you will have doubts with finding the reasons of chunk replication issues, feel free to contact support:

https://dl.acronis.com/u/software-defined/html/AcronisCyberInfrastructu…

Have a good day!

Thanks for the input

We have not seen any network errors in Graphana

Hi Bill,

Please contact support for further investigation as advised: https://dl.acronis.com/u/software-defined/html/AcronisCyberInfrastructu…