Google Cloud has announced native support for the OpenTelemetry Protocol (OTLP) in its Cloud Trace service, allowing developers to send trace data directly using OTLP and eliminating the need for vendor-specific exporters. This includes increased storage limits for attributes and spans.
The company's transition from fragmented observability tools to a unified system using OpenTelemetry and OneUptime dramatically improved incident response times, reducing MTTR from 41 to 9 minutes. By correlating logs, metrics, and traces through structured logging and intelligent sampling, they eliminated much of the noise and confusion that previously slowed root cause analysis. The shift also reduced the number of dashboards engineers needed to check per incident and significantly lowered the percentage of incidents with unknown causes.
Key practices included instrumenting once with OpenTelemetry, enforcing cardinality limits, and archiving raw data for future analysis. The move away from 100% trace capture and over-instrumentation helped manage data volume while maintaining visibility into anomalies. This transformation emphasized that effective observability isn't about collecting more data, but about designing correlated signals that support intentional diagnosis and reduce cognitive load.
Key practices included instrumenting once with OpenTelemetry, enforcing cardinality limits, and archiving raw data for future analysis. The move away from 100% trace capture and over-instrumentation helped manage data volume while maintaining visibility into anomalies. This transformation emphasized that effective observability isn't about collecting more data, but about designing correlated signals that support intentional diagnosis and reduce cognitive load.
A guide to building a robust logging system in Python, covering structured logging, log levels, handlers, formatters, filters, and integrating logging with modern observability practices.
This article compares three telemetry pipeline solutions – Cribl, Edge Delta, and DIY OpenTelemetry – based on scalability, performance, data management, intelligence, and cost. It details the strengths and weaknesses of each approach to help organizations choose the best solution for their observability and security data needs.
Traceloop's observability tool for LLM applications is now generally available. The company also announced a $6.1 million seed funding round. The platform extends OpenTelemetry to provide better observability for LLM applications, offering insights into model behavior and facilitating experimentation.
Edge Delta announces its new MCP Server, an open standard for streamlining communication between AI models and external data sources. It enables intelligent telemetry data analysis, adaptive pipelines, and effortless cross-tool orchestration directly within your IDE.
Edge Delta’s MCP Server acts as a bridge between developer tools and the Edge Delta platform, enabling generative AI to be integrated into observability workflows. Key benefits include:
* **Instant Root Cause Analysis:** Quickly identify the causes of errors using logs, metrics, and probable root causes.
* **Adaptive Pipelines:** AI-driven suggestions for optimizing telemetry pipeline configurations.
* **Effortless Orchestration:** Seamless integration of Edge Delta anomalies with other tools like Slack and AWS KB.
The server is built on Go and requires minimal authentication (Org ID + API Token). It can be easily integrated into IDEs with a simple configuration. The author anticipates that, despite current limitations like context window size and latency, this technology represents a significant step forward, similar to the impact of early algorithmic breakthroughs.
Edge Delta’s MCP Server acts as a bridge between developer tools and the Edge Delta platform, enabling generative AI to be integrated into observability workflows. Key benefits include:
* **Instant Root Cause Analysis:** Quickly identify the causes of errors using logs, metrics, and probable root causes.
* **Adaptive Pipelines:** AI-driven suggestions for optimizing telemetry pipeline configurations.
* **Effortless Orchestration:** Seamless integration of Edge Delta anomalies with other tools like Slack and AWS KB.
The server is built on Go and requires minimal authentication (Org ID + API Token). It can be easily integrated into IDEs with a simple configuration. The author anticipates that, despite current limitations like context window size and latency, this technology represents a significant step forward, similar to the impact of early algorithmic breakthroughs.
This Splunk Lantern article outlines the steps to monitor Gen AI applications with Splunk Observability Cloud, covering setup with OpenTelemetry, NVIDIA GPU metrics, Python instrumentation, and OpenLIT integration to monitor GenAI applications built with technologies like Python, LLMs (OpenAI's GPT-4o, Anthropic's Claude 3.5 Haiku, Meta’s Llama), NVIDIA GPUs, Langchain, and vector databases (Pinecone, Chroma) using Splunk Observability Cloud. It outlines a six-step process:
1. **Access Splunk Observability Cloud:** Sign up for a free trial if needed.
2. **Deploy Splunk Distribution of OpenTelemetry Collector:** Use a Helm chart to install the collector in Kubernetes.
3. **Capture NVIDIA GPU Metrics:** Utilize the NVIDIA GPU Operator and Prometheus receiver in the OpenTelemetry Collector.
4. **Instrument Python Applications:** Use the Splunk Distribution of OpenTelemetry Python agent for automatic instrumentation and enable Always On Profiling.
5. **Enhance with OpenLIT:** Install and initialize OpenLIT to capture detailed trace data, including LLM calls and interactions with vector databases (with options to disable PII capture).
6. **Start Using the Data:** Leverage the collected metrics and traces, including features like Tag Spotlight, to identify and resolve performance issues (example given: OpenAI rate limits).
The article emphasizes OpenTelemetry's role in GenAI observability and highlights how Splunk Observability Cloud facilitates monitoring these complex applications, providing insights into performance, cost, and potential bottlenecks. It also points to resources for help and further information on specific aspects of the process.
1. **Access Splunk Observability Cloud:** Sign up for a free trial if needed.
2. **Deploy Splunk Distribution of OpenTelemetry Collector:** Use a Helm chart to install the collector in Kubernetes.
3. **Capture NVIDIA GPU Metrics:** Utilize the NVIDIA GPU Operator and Prometheus receiver in the OpenTelemetry Collector.
4. **Instrument Python Applications:** Use the Splunk Distribution of OpenTelemetry Python agent for automatic instrumentation and enable Always On Profiling.
5. **Enhance with OpenLIT:** Install and initialize OpenLIT to capture detailed trace data, including LLM calls and interactions with vector databases (with options to disable PII capture).
6. **Start Using the Data:** Leverage the collected metrics and traces, including features like Tag Spotlight, to identify and resolve performance issues (example given: OpenAI rate limits).
The article emphasizes OpenTelemetry's role in GenAI observability and highlights how Splunk Observability Cloud facilitates monitoring these complex applications, providing insights into performance, cost, and potential bottlenecks. It also points to resources for help and further information on specific aspects of the process.
The article discusses the challenges faced due to differing observability tools and naming conventions, and how OpenTelemetry's standard naming schemas can streamline workflows and enhance interoperability.
Solomon Hykes, creator of Docker and CEO of Dagger, advocates for containerizing AI agents to manage complexity and enhance reusability. At Sourcegraph’s AI Tools Night, he demonstrated building an AI agent and a cURL clone using Dagger's container-based approach, emphasizing the benefits of standardization and debuggability.
OpenInference is a set of conventions and plugins that complements OpenTelemetry to enable tracing of AI applications, with native support from arize-phoenix and compatibility with other OpenTelemetry-compatible backends.
