How to check if cassandra is corrupted on Managed node?

bharat_pitta — Tue, 03 Jan 2023 02:43:45 GMT

Question

How to check if Cassandra is corrupted on the Managed node?

Answer

Open the Cassandra log file, usually located in :/opt/dynatrace-managed/cassandra/log/cassandra.log
Check if there are any corrupted messages in logs like those below

ERROR [CompactionExecutor:544] 2019-02-19 13:26:55,920 CassandraDaemon.java:231 - Exception in thread Thread[CompactionExecutor:544,1,main]
org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: /metricsDisk/cassandra/ruxitdb/Ts2Final01Min-967699b0f56011e7ad0403861a5cb15c/ruxitdb-Ts2Final01Min-ka-356332-Data.db
ERROR [CompactionExecutor:566] 2019-02-19 13:30:20,207 CassandraDaemon.java:231 - Exception in thread Thread[CompactionExecutor:566,1,main]
org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: /metricsDisk/cassandra/ruxitdb/Ts2Volatile01Min-960efe40f56011e7ad0403861a5cb15c/ruxitdb-Ts2Volatile01Min-ka-65282-Data.db
ERROR [CompactionExecutor:564] 2019-02-19 14:05:25,489 CassandraDaemon.java:231 - Exception in thread Thread[CompactionExecutor:564,1,main]
org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: /metricsDisk/cassandra/ruxitdb/Ts2Final01Min-967699b0f56011e7ad0403861a5cb15c/ruxitdb-Ts2Final01Min-ka-345130-Data.db

Corrupted files where the filename contains Ts are not critical, since only time-series data is involved (like the sample from above).
Corrupted files where the filename contains Me are critical since configuration data is involved. An example of such file name would be following

/var/opt/dynatrace-managed/cassandra/ruxitdb/Me2Data-0763a420293b11e8946add4c85a31f71/ruxitdb-Me2Data-tmp-ka-202976-Data.db

In such cases, Cassandra must be repaired!

Note timezone in Cassandra logs is the local time of the host Cassandra is running at

Re: How to check if cassandra is corrupted on Managed node?

ChadTurner — Tue, 10 Jan 2023 13:25:00 GMT

Thank you @bharat_pitta for this troubleshooting tip/trick 🙂

Re: How to check if cassandra is corrupted on Managed node?

andre_vdveen — Wed, 07 Jun 2023 15:48:30 GMT

Hi @bharat_pitta

I assume the log file is debug.log and not cassandra.log? I only see system.log, debug.log and launch.log in the default /opt/dynatrace-managed/cassandra/log/ directory 🙂

It looks like debug.log is the one that would contain the info you mentioned.

article How to check if cassandra is corrupted on Managed node? in Troubleshooting

How to check if cassandra is corrupted on Managed node?

Question

Answer

Re: How to check if cassandra is corrupted on Managed node?

Re: How to check if cassandra is corrupted on Managed node?