# Architecting an LLM-as-a-Judge Evaluation Framework using Dynatrace MCP & OpenTelemetry Traces (Dynatrace Managed)

**URL:** <https://community.dynatrace.com/t/architecting-an-llm-as-a-judge-evaluation-framework-using-dynatrace-mcp-opentelemetry-traces-dynatrace-managed/56511>\
**Category:** AI\
**Tags:** openllmetry, dynatrace-managed, agentic-ai, mcp-server, ai-observability, llms, prompt\
**Created:** [July 20, 2026, 3:48pm UTC](https://community.dynatrace.com/t/architecting-an-llm-as-a-judge-evaluation-framework-using-dynatrace-mcp-opentelemetry-traces-dynatrace-managed/56511 "2026-07-20T15:48:27Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Aboud1](https://avatars.discourse-cdn.com/v4/letter/a/47e85d/32.png) [@Aboud1](https://community.dynatrace.com/u/Aboud1)\
**Post date:** [July 20, 2026, 3:48pm UTC](https://community.dynatrace.com/t/architecting-an-llm-as-a-judge-evaluation-framework-using-dynatrace-mcp-opentelemetry-traces-dynatrace-managed/56511/1 "2026-07-20T15:48:27Z")

</div>

Hi everyone,

I’m designing an automated **LLM-as-a-Judge evaluation framework** for our production chatbot app, and I’d love to gather some architectural insights or approaches from the community.

**The Current Architecture:**

1. Our production Chatbot interacts with customers regarding tariffs/bundles. It uses a **RAG system** powered by **Azure OpenAI** to fetch data from our databases/knowledge bases.
2. The chatbot application is fully instrumented with **OpenTelemetry (OTel)**, capturing and sending runtime metrics to **Dynatrace**. Specifically, every trace contains the **User Prompt (Input)**, the **Model Response (Output)**, and the **Retrieved Context/References** utilized by the RAG system.
3. On the execution side, we have built a custom **MCP Client/Server environment** connected to our own **Azure OpenAI** &nbsp;instance. This MCP layer can successfully query and interact with Dynatrace.

**The Goal:** We want to leverage our **MCP SRE Agent** to fetch those OTel traces from Dynatrace, extract the (Input, Output, and Retrieved References), and feed them into our evaluator LLM to calculate and enforce advanced generative AI metrics:

- **Faithfulness / Hallucination Detection:** Evaluating if the chatbot’s output is strictly grounded in the retrieved RAG references, or if it invented fake package details or prices.
- **Answer Relevance:** Rating how well the chatbot’s output addresses the user’s initial prompt.
- **Context Precision:** Judging whether the RAG system retrieved relevant references for that specific user intent.
- **Guardrails Audit:** Post-runtime auditing to ensure no sensitive subscriber data or internal boundaries were breached.

**Where I need your thoughts:** Since this requires custom experimentation and engineering rather than an out-of-the-box feature:

- What is the best approach to structure the evaluator prompts for the Judge LLM when passing the (Input + Output + RAG References) fetched from OTel data?
- Are there any open-source evaluation schemas (like Ragas or DeepEval methodologies) that you successfully adapted into a custom MCP/Tool-calling pipeline?

Looking forward to your approaches and experiences! Let’s discuss. 🚀
