This Splunk Lantern article outlines the steps to monitor Gen AI applications with Splunk Observability Cloud, covering setup with OpenTelemetry, NVIDIA GPU metrics, Python instrumentation, and OpenLIT integration to monitor GenAI applications built with technologies like Python, LLMs (OpenAI's GPT-4o, Anthropic's Claude 3.5 Haiku, Meta’s Llama), NVIDIA GPUs, Langchain, and vector databases (Pinecone, Chroma) using Splunk Observability Cloud. It outlines a six-step process:
- Access Splunk Observability Cloud: Sign up for a free trial if needed.
- Deploy Splunk Distribution of OpenTelemetry Collector: Use a Helm chart to install the collector in Kubernetes.
- Capture NVIDIA GPU Metrics: Utilize the NVIDIA GPU Operator and Prometheus receiver in the OpenTelemetry Collector.
- Instrument Python Applications: Use the Splunk Distribution of OpenTelemetry Python agent for automatic instrumentation and enable Always On Profiling.
- Enhance with OpenLIT: Install and initialize OpenLIT to capture detailed trace data, including LLM calls and interactions with vector databases (with options to disable PII capture).
- Start Using the Data: Leverage the collected metrics and traces, including features like Tag Spotlight, to identify and resolve performance issues (example given: OpenAI rate limits).
The article emphasizes OpenTelemetry's role in GenAI observability and highlights how Splunk Observability Cloud facilitates monitoring these complex applications, providing insights into performance, cost, and potential bottlenecks. It also points to resources for help and further information on specific aspects of the process.
This skill path by Bryce Yu guides users through the basics of managing databases on Kubernetes using KubeBlocks. It covers installation, deployment, upgrades, backup, observability, and auto-tuning of database clusters.
OpenTelemetry, a Cloud Native Computing Foundation incubating project, helps software engineers collect and analyze data about system and application performance. Created from the merger of OpenTracing and OpenCensus in 2019, it addresses the challenges of observability in large-scale systems, especially with the rise of Kubernetes. The article discusses its rapid adoption, current challenges, and future innovations like profiling signals.
Observe, Inc. launched Kubernetes Explorer, a new observability experience designed to simplify visualizing and troubleshooting in Kubernetes environments, providing DevOps teams and engineers with a comprehensive view of K8s health and performance.
This Splunk Lantern blog post highlights new articles on instrumenting LLMs with Splunk, leveraging Kubernetes for Splunk, and using Splunk Asset and Risk Intelligence.
