Summary
Use Log ingest overview, Top log producers, and OpenPipeline self-monitoring metrics to identify which source caused an unexpected volume increase, inspect its records, correct the ingest rule scope, and set up alerting to catch future spikes.
Problem
Use this article when log consumption increased unexpectedly, or when one source appears to be dominating ingested volume without an obvious explanation.
This workflow also covers the reverse scenario an unexpected volume decrease since both start with the same investigation path.
Troubleshooting steps
Step 1: Establish the affected timeframe
Before opening any dashboard, record:
- When the change started (and whether it is still ongoing)
- The timezone you will use consistently throughout the investigation
- Whether the change is continuous or limited to a specific window
- Any application deployments, logging configuration changes, or ingest-rule changes during that period
Without a clean timeframe, volume comparisons will be misleading.
Step 2: Identify the top producer
- Open Dashboards → Log ingest overview.
- Set the timeframe to cover the spike window.
- Review the Top log producers panel.
- Identify which source, host, host group, Kubernetes cluster, or workload is associated with the volume change.
- Use the Actions menu (⋯) on the relevant row to open that source directly in the Logs app.
Note: The Log ingest overview dashboard will be upgraded with SFM event tiles from Dynatrace version 1.342+. Until then, also import the Log module self-monitoring dashboard for the full component-health view — see Monitor log source health with SFM events for instructions.
Step 3: Inspect the records from the top producer
Open the identified source in the Logs app and review the records from the spike window.
Determine whether the increase is explained by:
- Debug or trace-level records that should not be in a production ingest rule
- Repetitive application errors generating high-frequency identical records
- A new deployment or application that was not previously covered by an ingest rule
- A change in application logging verbosity or destination
- A recent ingest-rule or matcher change that expanded scope
Do not infer the cause from volume alone. Confirm it from the actual records.
Step 4: Check OpenPipeline self-monitoring metrics
If you want to verify the volume change at the pipeline level rather than just the source level, use the OpenPipeline self-monitoring metrics. These count records at each stage of the pipeline — from ingestion to storage.
| Metric | What it shows |
|---|---|
dt.sfm.openpipeline.ingest_sources_in.records |
Records entering at the ingest source, per data type and source path |
dt.sfm.openpipeline.routing.records |
Records processed through routing rules, per route and target pipeline |
dt.sfm.openpipeline.pipelines_out.records |
Records leaving a pipeline after processing, per data type and storage bucket |
dt.sfm.openpipeline.not_stored.records |
Records that were dropped, not persisted, or invalid — check pipeline_id to determine whether it happened at ingest or within a pipeline |
The OpenPipeline usage overview ready-made dashboard provides a preconfigured view of these metrics. Use it to identify where in the pipeline the volume change entered or dropped out.
Step 5: Review the ingest rule that covers the top producer
Open:
Settings app > Collect and capture > Log monitoring > Configure log module > Sources
Locate the source and review its active and inherited rules.
Check whether:
- A matcher intended for one scope also matches an unintended source or environment
- Current source attributes differ from the attributes that existed when the rule was created
- A recent matcher change expanded coverage beyond the intended scope
- The rule applies at the correct level (environment, host group, host)
Correct only the specific rule or matcher that caused the unintended coverage. Do not broaden exclusions without confirming the intended policy first.
Step 6: Verify the correction
After adjusting the rule:
- Return to Log ingest overview.
- Set the timeframe to the period after the change.
- Review the Top log producers panel.
- Confirm that intended logs remain available and searchable.
- Confirm that the high-volume unintended records are no longer dominating the producer list.
Step 7: Set up alerting to catch future spikes
Once the current issue is resolved, create a log metric for the affected source and configure an anomaly detector for future volume thresholds.
To do this, use Settings > Log monitoring > Log metrics for Managed and for Saas Metrics are created in OpenPipeline (via the OpenPipeline configuration or Settings > Log Monitoring > Log metrics
to define a metric scoped to the relevant source, then configure an anomaly detection rule or Davis anomaly detector against that metric.
You can also use dt.log.status_per_entity_count is a built-in, pre-aggregated Dynatrace log metric that counts ingested log records per entity and per log status (e.g., ERROR, WARN, INFO).
For context on how log-based metrics behave, see Dynatrace Log Monitoring: Metric Shows No Data.
For a volume decrease instead
If the scenario is a volume drop rather than a spike, follow the same path to Step 2 to identify the affected scope, then:
- Check Ingest components health on the Log ingest overview dashboard for any unhealthy component.
- If a component is unhealthy → My logs went silent where do I start?
- Confirm the source still writes new records at the original system.
- Check source coverage in the Sources tab → What is Dynatrace actually capturing right now?
- Check for delay or rate limiting:
What's next
Opening a support case with below details
- Approximate start time and timezone of the volume change
- Affected source, host group, cluster, workload, or ingestion method
- Screenshot or export from Top log producers showing the spike window
- Screenshot or export from Ingest components health (if a decrease was also observed)
- OpenPipeline self-monitoring metric values for the affected pipeline and timeframe
- Sanitized sample records from the affected source during the spike period
- Applicable active and inherited ingest rules with matchers and scopes
- Recent logging, application, deployment, or rule changes
- Confirmation that the original source still produces records
- Applicable OneAgent, Dynatrace Operator, or collector version
Related reading
