Overview
This article describes a scenario where log ingestion is silently rejected by the Extension Execution Controller (EEC) on an ActiveGate host due to the log persistence storage being full. Logs will not appear in Dynatrace even though the monitored sources appear healthy.
When can this problem be observed?
This problem typically surfaces in the following situations:
- High log ingest volume - a large number of extensions or monitored sources are sending logs through a single ActiveGate, causing persistence to fill faster than it can be flushed to the cluster.
- Prolonged connectivity issues - if the ActiveGate cannot reach the Dynatrace cluster for an extended period, buffered log data accumulates in persistence until it reaches the configured limit.
- After onboarding new log sources - adding new extensions or increasing log verbosity on existing ones can push an otherwise healthy ActiveGate over the persistence threshold.
Symptoms
The following indicators point to this issue:
Metrics
dsfm:extension.engine.logs_ingest.records_rejected shows a non-zero and/or increasing count of rejected log records.dsfm:extension.engine.eec_status split by dt.extension.status shows the value LOG_PERSISTENCE_FULL.
Datasource logs
The following message may appear in the extension or datasource logs on the ActiveGate host:
Log persistence full, rejecting logs
Root Cause
The EEC uses a local disk-based persistence mechanism (the reliability mechanism) to buffer log data. This ensures that logs collected from monitored sources are not lost in the event of:
- Temporary network issues between the ActiveGate and the Dynatrace cluster
- ActiveGate or OneAgent restarts
- Transient data ingest overloads
When the persistence reaches its storage limit, the EEC can no longer accept new log records and begins rejecting them. This results in log data gaps in Dynatrace.
The persistence is located at:
| OS |
Path |
| Windows |
%PROGRAMDATA%\dynatrace\remotepluginmodule\agent\runtime\extensions\persistence |
| Linux |
/var/lib/dynatrace/remotepluginmodule/agent/runtime/extensions/persistence |
By default, this directory is capped at 600 MB. The full reliability mechanism also requires an additional buffer of 1.5GB, meaning a minimum of 2100 MB of total free disk space is required on the host at default settings.
Resolution
There are two approaches to resolving this issue. Choose the one that best fits your environment.
Option 1: Increase the persistence size limit (Recommended)
If the host has sufficient free disk space, increase the maximum persistence volume by editing the EEC configuration file.
Configuration file location:
| OS |
Path |
| Windows |
C:\ProgramData\dynatrace\remotepluginmodule\agent\conf\extensionsuser.conf |
| Linux |
/var/lib/dynatrace/remotepluginmodule/agent/conf/extensionsuser.conf |
Add or update the following property (value is in kilobytes):
persistence.total_limit_kb = 614400
The default value is 614400 (600 MB). Increase it to a value your disk can support. Remember to account for the required buffer of 1.5GB on top of whatever limit you set.
After editing the file, restart the ActiveGate service for the change to take effect.
Note: Before increasing the limit, verify that the host has enough free disk space. The minimum requirement is persistence.total_limit_kb + 1.5GB of total free space.
Option 2: Disable reliable mode (Use with caution)
If disk space cannot be increased on the host, you can disable the reliability mechanism entirely. In this mode, log data is transmitted directly without being persisted locally, meaning logs will be lost if there is a communication failure between the ActiveGate and the cluster.
In the same extensionsuser.conf file, set:
persistence.reliable_mode = false
⚠️ Warning: Setting persistence.reliable_mode = false means that logs will not be stored locally during ActiveGate restarts, network disruptions, or ingest overloads. This can result in permanent log data loss. Use this setting only as a temporary workaround or in environments where occasional log gaps are acceptable.
Restart the ActiveGate service after making this change.
Verification
After applying the fix, confirm the issue is resolved by checking:
- The metric
dsfm:extension.engine.logs_ingest.records_rejected should drop to zero or stop increasing.
- The
dsfm:extension.engine.eec_status dimension dt.extension.status should no longer show LOG_PERSISTENCE_FULL.
- Log ingestion should resume and new log records should appear in Dynatrace Log Viewer.
What's next
If this article didn't help, please open a support ticket, mention that this article was used, and provide the following:
- Link to the tenant
- A description of the problem
- A list of troubleshooting steps that have been performed
