Hi everyone,
I’m designing an automated LLM-as-a-Judge evaluation framework for our production chatbot app, and I'd love to gather some architectural insights or approaches from the community.
The Current Architecture:
Our production Chatbot interacts with customers regarding tariffs/bundles. It uses a RAG system powered by Azure OpenAI to fetch data from our databases/knowledge bases.
The chatbot application is fully instrumented with OpenTelemetry (OTel), capturing and sending runtime metrics to Dynatrace. Specifically, every trace contains the User Prompt (Input), the Model Response (Output), and the Retrieved Context/References utilized by the RAG system.
On the execution side, we have built a custom MCP Client/Server environment connected to our own Azure OpenAI instance. This MCP layer can successfully query and interact with Dynatrace.
The Goal: We want to leverage our MCP SRE Agent to fetch those OTel traces from Dynatrace, extract the (Input, Output, and Retrieved References), and feed them into our evaluator LLM to calculate and enforce advanced generative AI metrics:
Faithfulness / Hallucination Detection: Evaluating if the chatbot’s output is strictly grounded in the retrieved RAG references, or if it invented fake package details or prices.
Answer Relevance: Rating how well the chatbot's output addresses the user's initial prompt.
Context Precision: Judging whether the RAG system retrieved relevant references for that specific user intent.
Guardrails Audit: Post-runtime auditing to ensure no sensitive subscriber data or internal boundaries were breached.
Where I need your thoughts: Since this requires custom experimentation and engineering rather than an out-of-the-box feature:
What is the best approach to structure the evaluator prompts for the Judge LLM when passing the (Input + Output + RAG References) fetched from OTel data?
Are there any open-source evaluation schemas (like Ragas or DeepEval methodologies) that you successfully adapted into a custom MCP/Tool-calling pipeline?
Looking forward to your approaches and experiences! Let's discuss. 🚀
