Troubleshooting for Ingest sources that send data along static routes to Readymade pipelines

[jgrant]

• Summary
• Problem
• Troubleshooting steps
• Resolution
• What's next

Summary 

Some Ingest sources are configured to static routes, so all incoming logs of that source route directly to a readymade pipeline, bypassing any configured dynamic routes. Ingest sources are read-only, so if the readymade pipeline is tied to the configuration's owner, and you upgrade an extension or source to use OpenPipeline, we may find that our previously configured dynamic routes stop working, and if the previously logs routed to the default pipeline, for example, classic settings > Log monitoring > Log processing, classic rules will stop working too.

Problem

Newer releases of extensions own a dedicated readymade pipeline with a statically routed Ingest source. This extension owned ingest source supersedes the generic, built-in Extensions ingest source, which sends data to Dynamic routing. After an upgrade of a monitoring configuration in the extension, dynamic routes that we added for that source's logs report 0% distribution, and matching log processors in our custom pipeline stop working, because the new static route sends all logs of the extension's monitoring configurations to the extension's own Readymade pipeline, before dynamic routing can be evaluated.

When identifying why processors stop working, we should check for Ingest sources that sends data along a static route in OpenPipeline. We can review:

• Revision history on the readymade pipeline to confirm when it was created or re-created, which often aligns with an updated monitoring configuration that installs the newer extension release.
• Revision history on a classic log event rule can confirm the owner, so we know if we should manually migrate. Only rules created by an extension may be re-created in the Readymade pipeline.
• If logs are stored in an unexpected bucket, processed by an unexpected pipeline, or processing stops working, that is a possible indicator that a new Ingest source is bypassing our Dynamic routing by sending the data along a static route.

When working with Ingest sources sources sending data along a static route in OpenPipeline:

• A generic catch-all Ingest source like Extension is superseded by extension-specific Ingest sources, like that of SQL Server or IBM i.
• We can't modify the static route or the Readymade pipeline it points to, since these Ingest sources and pipelines are managed by an Owner that is an extension. 
• We can't convert the static route in the Ingest source to a dynamic route.
• Ingest sources send logs along either dynamic routing or a static route.
• We should create Pipeline groups, so we can include our own processing on data routed to a Readymade pipeline without needing to downgrade.
• We should migrate classic rules into our base pipeline, since the classic pipeline is exclusive to member and base pipelines, and is unavailable in Pipeline Groups.

Troubleshooting steps

When you're restricted from overriding configurations managed by an owner like an extension, Pipeline groups is an option to insert your own processing into the data pipeline. A pipeline group can have a read-only member pipeline that can process logs with a user-defined base pipeline that includes your custom processors. To understand this scenario's resolution, we should remember that:

• Base pipelines prior to the member pipeline placeholder run first. [1]
• A base pipeline can exist standalone but is inactive unless part of a pipeline group. [2]
  
  • Readymade pipelines like those created by extensions are member pipelines.
• When it is part of a pipeline group, a member pipeline receives data routed by the first matching route and processes that data through the stages enabled by the group. [3]
  • Readymade pipelines process data sent via a static route through the stages enabled by the group.
• A (member) pipeline belongs to only one pipeline group at a time. [4]
  • You can't add your Readymade pipeline to more than one pipeline group. 
• There is no need for any additional routing. You can imagine the base as a wrapper of the member via the pipeline group:
                                           +------pipeline group-------+
                                         | +--->[ base pipeline ]    |
                                         | |            |            |
                                         | |            V            |
[ extension data ] ---- static mapping --|-+  [ extension pipeline ] |
                                         |              |            |
                                         |              V            |
                                         |       [ base pipeline ]   |
                                         +---------------------------+

To resolve this scenario for Readymade pipelines, you should add the Readymade pipeline to the composition of your pipeline group, then add your base pipeline.

1. Confirm that your custom pipeline is a base pipeline that you can add to your pipeline group. 

   

   1. Only base pipelines are listed when you add base pipeline to a Pipeline group composition of a group that you're editing in Settings > OpenPipeline > Pipeline Groups.

      

   2. Move your custom processors to your base pipeline.

      1. Re-create any classic processors in your base pipelines, since classic log processing is exclusive, not a base or member pipeline candidate.

2. Add the Readymade pipeline(s) for the extension's ingested data that you want your base pipelines to process.

3. Enable the stages for the processors of each pipeline that you want to execute on matching data.

   

Resolution

With a Pipeline group added for custom processing of your logs from an extension owned Ingest source, you can avoid downgrading all monitoring configurations, and upgrade your monitoring configurations without any blockers.

If the static route was introduced by a monitoring configuration update of an ingest source like an extension, downgrading all monitoring configuration of that extension by re-installing the extension as a lower version that doesn't include the static route, will remove the static route and readymade pipeline, so the Ingest source is Extensions, which sends the logs to dynamic routing again. Pipeline groups allows you to keep monitoring configurations with or without breaking changes introduced by the extension's own Ingest source in OpenPipeline, so we avoid any future upgrade blockers caused by a complete downgrade of an extension.

What's next

If your log data is still not transformed by your migrated processors in a base pipeline added to a pipeline group, or the pipeline group doesn't appear to be working, please don't hesitate to review other articles in our Logs Troubleshooting Map or open a support case with Dynatrace, mentioning the steps that you completed, the result, and the expected result, for example, logs in a notebook versus the expected transformation, extracted metric or events.

More articles can be found on the Logs Troubleshooting Map