Question
How to check if Cassandra is corrupted on the Managed node?
Answer
- Open the Cassandra log file, usually located in :/opt/dynatrace-managed/cassandra/log/cassandra.log
- Check if there are any corrupted messages in logs like those below
<span class="">ERROR </span>[CompactionExecutor:544] 2019<span class="">-02</span><span class="">-19</span> 13:26:55,920 CassandraDaemon.java:231 - Exception in thread Thread[CompactionExecutor:544,1,main]
org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: /metricsDisk/cassandra/ruxitdb/Ts2Final01Min<span class="">-967699</span>b0f56011e7ad0403861a5cb15c/ruxitdb-Ts2Final01Min-ka<span class="">-356332</span>-Data.db
<span class="">ERROR </span>[CompactionExecutor:566] 2019<span class="">-02</span><span class="">-19</span> 13:30:20,207 CassandraDaemon.java:231 - Exception in thread Thread[CompactionExecutor:566,1,main]
org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: /metricsDisk/cassandra/ruxitdb/Ts2Volatile01Min<span class="">-960</span>efe40f56011e7ad0403861a5cb15c/ruxitdb-Ts2Volatile01Min-ka<span class="">-65282</span>-Data.db
<span class="">ERROR </span>[CompactionExecutor:564] 2019<span class="">-02</span><span class="">-19</span> 14:05:25,489 CassandraDaemon.java:231 - Exception in thread Thread[CompactionExecutor:564,1,main]
org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: /metricsDisk/cassandra/ruxitdb/Ts2Final01Min<span class="">-967699</span>b0f56011e7ad0403861a5cb15c/ruxitdb-Ts2Final01Min-ka<span class="">-345130</span>-Data.db
- Corrupted files where the filename contains Ts are not critical, since only time-series data is involved (like the sample from above).
- Corrupted files where the filename contains Me are critical since configuration data is involved. An example of such file name would be following
<span class="">/var/</span>opt<span class="">/dynatrace-managed/</span>cassandra<span class="">/ruxitdb/</span>Me2Data-<span class="">0763</span>a420293b11e8946add4c85a31f71/ruxitdb-Me2Data-tmp-ka-<span class="">202976</span>-Data.db
In such cases, Cassandra must be repaired!
Note timezone in Cassandra logs is the local time of the host Cassandra is running at