This Splunk Lantern article outlines the steps to monitor Gen AI applications with Splunk Observability Cloud, covering setup with OpenTelemetry, NVIDIA GPU metrics, Python instrumentation, and OpenLIT integration to monitor GenAI applications built with technologies like Python, LLMs (OpenAI's GPT-4o, Anthropic's Claude 3.5 Haiku, Meta’s Llama), NVIDIA GPUs, Langchain, and vector databases (Pinecone, Chroma) using Splunk Observability Cloud. It outlines a six-step process:
- Access Splunk Observability Cloud: Sign up for a free trial if needed.
- Deploy Splunk Distribution of OpenTelemetry Collector: Use a Helm chart to install the collector in Kubernetes.
- Capture NVIDIA GPU Metrics: Utilize the NVIDIA GPU Operator and Prometheus receiver in the OpenTelemetry Collector.
- Instrument Python Applications: Use the Splunk Distribution of OpenTelemetry Python agent for automatic instrumentation and enable Always On Profiling.
- Enhance with OpenLIT: Install and initialize OpenLIT to capture detailed trace data, including LLM calls and interactions with vector databases (with options to disable PII capture).
- Start Using the Data: Leverage the collected metrics and traces, including features like Tag Spotlight, to identify and resolve performance issues (example given: OpenAI rate limits).
The article emphasizes OpenTelemetry's role in GenAI observability and highlights how Splunk Observability Cloud facilitates monitoring these complex applications, providing insights into performance, cost, and potential bottlenecks. It also points to resources for help and further information on specific aspects of the process.
The article discusses the challenges faced due to differing observability tools and naming conventions, and how OpenTelemetry's standard naming schemas can streamline workflows and enhance interoperability.
Solomon Hykes, creator of Docker and CEO of Dagger, advocates for containerizing AI agents to manage complexity and enhance reusability. At Sourcegraph’s AI Tools Night, he demonstrated building an AI agent and a cURL clone using Dagger's container-based approach, emphasizing the benefits of standardization and debuggability.
OpenInference is a set of conventions and plugins that complements OpenTelemetry to enable tracing of AI applications, with native support from arize-phoenix and compatibility with other OpenTelemetry-compatible backends.
Arize Phoenix is an open-source observability library for AI experimentation, evaluation, and troubleshooting, built by Arize AI.
This article provides an overview of OpenTelemetry, an open-source observability framework, and guides on integrating it with Go applications. It covers key concepts like logs, metrics, and traces, and demonstrates setting up a reusable telemetry package using OpenTelemetry in Go.
OpenTelemetry, a Cloud Native Computing Foundation incubating project, helps software engineers collect and analyze data about system and application performance. Created from the merger of OpenTracing and OpenCensus in 2019, it addresses the challenges of observability in large-scale systems, especially with the rise of Kubernetes. The article discusses its rapid adoption, current challenges, and future innovations like profiling signals.
A Microsoft engineer demonstrates how WebAssembly modules can run alongside containers in Kubernetes environments, offering benefits like reduced size and faster cold start times for certain workloads.
The article discusses the future of observability in 2025, highlighting the significant role of OpenTelemetry and AI in improving observability and reducing costs.
Version 3.0 of the popular open-source monitoring system Prometheus has been released, with enhancements focused on a new user interface, OpenTelemetry support, and other new features aimed at improving user experience and streamlining workflows.
