AI Observability

AI observability is a modern, complete approach to understanding how your AI applications behave, how data flows, and how performance changes over time. At Dynatrace, we provide advanced model observability with predictive capabilities for efficient cost management, end-to-end traceability in RAG pipelines and agentic frameworks, compliance and governance to adhere to laws like the EU AI Act, and LLM safeguards such as PII leakage prevention, language toxicity assessment, and hallucination detection. This ensures service health and performance, service quality, compliance, and governance. Service Health & Performance Monitor standard metrics like response times, error rates, and cost signals, and chart them based on different AI models. Leverage the predictive capabilities of Dynatrace Davis AI® to detect changes in usage behavior and predict cost changes to help team members understand model performance and optimization opportunities. Key Features: Cost Prediction Token Consumption Request Duration Service Health Open Problems Service Quality & Guardrails Monitor safety, privacy, and truthfulness safeguards for your generative AI applications with Dynatrace. Detect toxic language, track Personally Identifiable Information (PII) leakage, identify attempts at LLM misuse such as malicious prompt injection, and monitor model hallucinations. Key Features Toxicity PII Leakage Denied Topics Hallucinations End-to-End Tracing & Debugging Map dependencies between multiple large language models that work in concert in your RAG pipelines or agentic frameworks to provide end-to-end observability of the entire system. Track every step - from request to response - and gain full visibility into your pipeline's performance. Key Features RAG Pipeline Tracing Agentic Framework Tracing Prompt Data Lineage Data Compliance & Governance Dynatrace automatically tracks every input and output with no sampling to provide an audit trail of accurate monitoring and observability for your GenAI invocations. This helps ensure that applications comply with applicable laws such as the EU AI act and the US Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence. Key Features Comprehensive Audit Trailing Real-time Data Access Long-term Data Retention ... View more

By now, everyone is aware of generative AI fueled by large language models (LLMs) and generative pre-trained transformers (GPTs). The next level of innovation is agentic AI and the autonomous AI agents that drive it. Using Model Context Protocol (MCP) to facilitate agent-to-agent communication, these systems are revolutionizing how enterprises automate tasks and orchestrate complex workflows. Powered by LLMs, vector databases, retrieval augmented generation (RAG) pipelines and additional tools, these AI agents are expanding extensively, giving rise to multi-agent systems, cross-agent protocols, and context-sharing standards. But these autonomous agents also introduce new challenges in monitoring, debugging, and security. Go to the Dynatrace BLOG POST where we’ll examine in detail the fundamentals of AI agents, models, and the emerging standards that help them communicate, like Agent2Agent (A2A) and Model Context Protocol (MCP). Key takeaways: Autonomous AI agents are the backbone of agentic AI. These services combine to deliver adaptable automated tasks. AI agents depend on LLMs and orchestration logic. These technologies maintain the agent’s state, session memory, context, and reasoning strategies. Agents depend on protocols, such as A2A and MCP, to effectively communicate. Models and agents need these protocols to manage multi-agent communication. - - - Links: Understand the latest AI revolution: Agentic AI, Model Context Protocol (MCP) and all that jazz blog post Dynatrace MCP server on GitHub  ... View more

We're thrilled to announce that the OpenLLMetry semantic conventions have officially become part of OpenTelemetry! This milestone is a significant step in standardizing the observability of large language model (LLM) based applications. What does this mean for you? By integrating our semantic conventions with OpenTelemetry, you'll benefit from: Enhanced Observability: Gain deeper insights into your LLM applications with standardized metrics, traces, and logs. This helps in quickly identifying and resolving issues, improving overall system performance Consistency Across Platforms: Enjoy a unified approach to monitoring and troubleshooting, regardless of the platform or environment. This consistency simplifies the development and maintenance of LLM applications Improved Development Cycles: With better observability, you can iterate faster and enhance the quality of your LLM applications more efficiently. An official OpenTelemetry workgroup was founded to identify and define a standardized vocabulary and specification for the observability of LLM-based applications. This effort explores various LLM-based application APIs, LLM tasks, and LLM workflows for multistage reasoning. Join the discussion and help us shape the future of LLM observability! ... View more

Building an AI application doesn't have to break the bank. The top AI development tools are open-source, fostering an evolving ecosystem that democratizes AI for everyone. Here are the essential elements of this open-source AI toolkit: Frontend Development For creating stunning AI user interfaces, frameworks like NextJS and Streamlit are invaluable. Vercel also offers seamless deployment solutions. Embeddings and RAG Libraries Embedding models and RAG libraries such as Nomic, JinaAI, Cognito, and LLMAware enable developers to implement precise search and RAG functionalities. Backend and Model Integration Backend development is streamlined with frameworks like FastAPI, Langchain, and Netflix Metaflow. For model integration, tools like Ollama and Huggingface are excellent choices. Data Storage and Retrieval Efficient data storage and retrieval can be achieved using options like Postgres, Milvus, Weaviate, PGVector, and FAISS. Large-Language Models Open-source models like Llama, Mistral, Qwen, Phi, and Gemma offer competitive performance, serving as robust alternatives to proprietary models like GPT and Claude. What other tools would you recommend adding to the Open Source AI Toolkit? Let's discuss! ... View more

Dynatrace leverages OpenLLMetry to ingest telemetry data of your AI workloads. OpenLLMetry is a set of extensions built on top of OpenTelemetry that gives you complete observability over your LLM application. Simply install the OpenLLMetry SDK using the following command: pip install traceloop-sdk Instrument your GenAI application by adding the following code snippet to your app: from traceloop.sdk import Traceloop headers = { "Authorization": "Api-Token <YOUR_DT_API_TOKEN>" } Traceloop.init( app_name="<your-service>", api_endpoint="https://<YOUR_ENV>.live.dynatrace.com/api/v2/otlp", headers=headers )   Additional information, as well as support for different languages can be found in our documentation. ... View more