article Jenkins Pipeline Analytics using OpenTelemetry and Dynatrace in Troubleshooting

Jenkins Pipeline Analytics using OpenTelemetry and Dynatrace

anton_konikov — Fri, 27 Sep 2024 07:48:12 GMT

Prerequisites

Jenkins as CI/CD tool
OpenTelemetry plugin for Jenkins - https://plugins.jenkins.io/opentelemetry/
Generate API token in scopes: openTelemetryTrace.ingest, metrics.ingest, logs.ingest.

To illustrate, we’ll employ Podman as our container runtime and set up Jenkins within it. For that, you could use the following command:

podman run --name jenkins -d -e OTEL_LOGS_EXPORTER="otlp" -p 8080:8080 jenkins/jenkins:lts

Once the container is up and running in your favorite browser, let's navigate to http://localhost:8080/ and, follow the shown instruction and login into Jenkins.

As the second step, from the main Jenkins view Dashboard, click on Manage Jenkins and then on Plugins:

On this step, let's install OpenTelemetry plugin. Select Available pugins and type OpenTelemetry in the search bar. Select the plugin and depending on the Jenkins version, press Install button or Install without restart.

After the installation is sucesfully finished, scroll down and select Restart Jenkins when installation is complete and no jobs are running:

Configuration

Now let's go to Dynatrace UI and generate an API token

From Manage Jenkins view, navigate to System configuration:

Scroll to the OpenTelemetry section and in the OTLP endpoint sectrion specify your SaaS OTLP API endpoint, e.g. https://{your-environment-id}.live.dynatrace.com/api/v2/otlp, in the Authentication section, select Header Authetication and type in the filed Header Name: Authorization:

For the Header Value, create a new secret tocken by seceting Add > Jenkins. You’ll need to create a secret text with the following fields:

For Kind, choose Secret text.
The Secret field should be in the follwoing format: Authorization: Api-Token dt.....
Optional field ID can be left blank.
Optionally add a description such as Dynatrace API ingest token.

Then scroll down to Advanced configuration and expand it. In the Configuration properties filed paste: otel.exporter.otlp.protocol=http/protobuf, as gRPC is not yet supported:

Additionally you can configure visualisation backend to show the link between build and trace in Dynatrace:

Receiving Jenkins data in Dynatrace

Please make sure that OTLP Endpoint set up earlier is accesible from Jenkins location. Let's create the first test pipeline and verify the monitoring data can be observed.

Click on New Item and create the Pipeline job:

Then scroll down to the Pipeline configuration and select the Github+Maven sample or paste the bellow snippet:

pipeline { agent any tools { // Install the Maven version configured as "M3" and add it to the path. maven "M3" } stages { stage('Build') { steps { // Get some code from a GitHub repository git 'https://github.com/jglick/simple-maven-project-with-tests.git' // Run Maven on a Unix agent. sh "mvn -Dmaven.test.failure.ignore=true clean package" // To run Maven on a Windows agent, use // bat "mvn -Dmaven.test.failure.ignore=true clean package" } post { // If Maven was able to run the tests, even if some of the test // failed, record the test results and archive the jar file. success { junit '**/target/surefire-reports/TEST-*.xml' archiveArtifacts 'target/*.jar' } } } } }

Note: the prerequisite is to install Maven, which requires defining Maven instance in the Mange Jenkins > Tools:

After finishing the configuration of the pipeline job you can click on the Build Now button which then trigger execution of the pipeline:

As it can be seen in the screenshot above, under OpenTelemetry appeared a link navigating to the particular trace in Dynatrace UI. In the trace view you could see all logs from the Jenkins build by clicking on View logs for this trace or click on the particular trace sub-node/span and open Logs tab:

Additionally, we encourage you to watch this demo session Dynatrace Tips & Tricks #18 - Jenkins Pipeline Analytics with OpenTelemetry.

Re: Jenkins Pipeline Analytics using OpenTelemetry and Dynatrace

Mizső — Wed, 02 Oct 2024 15:23:00 GMT

Hi @anton_konikov ,

Can we use this solution on Managed environment with OTLP traces v1 api?

Thanks in advance.

Best reagrads,

Mizső

Re: Jenkins Pipeline Analytics using OpenTelemetry and Dynatrace

Julius_Loman — Thu, 03 Oct 2024 07:16:09 GMT

@Mizső yes, you can.

Re: Jenkins Pipeline Analytics using OpenTelemetry and Dynatrace

Mizső — Thu, 03 Oct 2024 08:02:07 GMT

Hi @Julius_Loman ,

Thanks for the confirmation. 😉

Best regards,

Mizső

Re: Jenkins Pipeline Analytics using OpenTelemetry and Dynatrace

yniasreynders96 — Mon, 03 Feb 2025 09:28:02 GMT

For those who get traces but no logs, we needed to add the following parameters to get the build logs.

otel.exporter.otlp.protocol=http/protobuf otel.instrumentation.jenkins.remote.span.enabled=true otel.instrumentation.jenkins.agent.enabled=true otel.logs.exporter=otlp otel.logs.mirror_to_disk=true

Re: Jenkins Pipeline Analytics using OpenTelemetry and Dynatrace

hnimawat — Mon, 17 Feb 2025 17:16:30 GMT

Hi @Mizső

I am using Dynatrace Managed and did all the steps as mentioned here but still where I click on "openTelemetry" in Jenkins, it redirects me to our managed environment and shows "Trace Not Found".

Re: Jenkins Pipeline Analytics using OpenTelemetry and Dynatrace

Mizső — Tue, 18 Feb 2025 05:00:48 GMT

Hi @hnimawat,

We have not implemeted this soultion yet at our client. Sorry but I do not have further information about it.

Best regards,

János

Re: Jenkins Pipeline Analytics using OpenTelemetry and Dynatrace

anton_konikov — Tue, 18 Feb 2025 08:56:08 GMT

Hi @hnimawat,

Embedded OpenTelemetry knows the trace ID and substitutes it into the URL to your Managed. However, the trace was not exported to Dynatrace server.

This can happen e.g. when OpenTelemetry doesn't trust your certificate of the endpoint, but the real issue you could see in the Jenkins logs.

The preferable solution would be to add this certificate for Jenkins/OTLP or alternatively, you can use a Dynatrace OTLP collector as the middle point of the architecture to aggregate all OTEL data. Then add the certificate from your Managed to this collector or add an option to skip TLS cert verify (as Dynatrace we recommend avoiding using this option).