How to check if cassandra is corrupted on Managed node?

Question

How to check if Cassandra is corrupted on the Managed node?

Answer

  • Open the Cassandra log file, usually located in :/opt/dynatrace-managed/cassandra/log/cassandra.log
  • Check if there are any corrupted messages in logs like those below
<span class="">ERROR </span>[CompactionExecutor:544] 2019<span class="">-02</span><span class="">-19</span> 13:26:55,920 CassandraDaemon.java:231 - Exception in thread Thread[CompactionExecutor:544,1,main]
org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: /metricsDisk/cassandra/ruxitdb/Ts2Final01Min<span class="">-967699</span>b0f56011e7ad0403861a5cb15c/ruxitdb-Ts2Final01Min-ka<span class="">-356332</span>-Data.db
<span class="">ERROR </span>[CompactionExecutor:566] 2019<span class="">-02</span><span class="">-19</span> 13:30:20,207 CassandraDaemon.java:231 - Exception in thread Thread[CompactionExecutor:566,1,main]
org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: /metricsDisk/cassandra/ruxitdb/Ts2Volatile01Min<span class="">-960</span>efe40f56011e7ad0403861a5cb15c/ruxitdb-Ts2Volatile01Min-ka<span class="">-65282</span>-Data.db
<span class="">ERROR </span>[CompactionExecutor:564] 2019<span class="">-02</span><span class="">-19</span> 14:05:25,489 CassandraDaemon.java:231 - Exception in thread Thread[CompactionExecutor:564,1,main]
org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: /metricsDisk/cassandra/ruxitdb/Ts2Final01Min<span class="">-967699</span>b0f56011e7ad0403861a5cb15c/ruxitdb-Ts2Final01Min-ka<span class="">-345130</span>-Data.db
  • Corrupted files where the filename contains Ts are not critical, since only time-series data is involved (like the sample from above).
  • Corrupted files where the filename contains Me are critical since configuration data is involved. An example of such file name would be following
<span class="">/var/</span>opt<span class="">/dynatrace-managed/</span>cassandra<span class="">/ruxitdb/</span>Me2Data-<span class="">0763</span>a420293b11e8946add4c85a31f71/ruxitdb-Me2Data-tmp-ka-<span class="">202976</span>-Data.db

In such cases, Cassandra must be repaired!

Note timezone in Cassandra logs is the local time of the host Cassandra is running at

7 Likes

Thank you @bharat_pitta for this troubleshooting tip/trick :slightly_smiling_face:

Hi @bharat_pitta

I assume the log file is debug.log and not cassandra.log? I only see system.log, debug.log and launch.log in the default /opt/dynatrace-managed/cassandra/log/ directory :slightly_smiling_face:

It looks like debug.log is the one that would contain the info you mentioned.

3 Likes