Automating root cause analysis at scale: Multi-signal correlation for cloud native incident responseSantosh Balaranganathan and colleagues at Atlassian describe their automated root cause analysis system that treats incident diagnosis as a correlation problem across three dimensions: signal type (metrics, logs, traces), time, and service topology. The pipeline scopes the search using OpenTelemetry-derived dependency graphs, detects anomalies independently per signal, temporally aligns co-occurring anomalies into bundles, traverses the graph to determine causal direction, and emits ranked hypotheses with human-readable narratives so responders can validate and act quickly.
- Sequence fingerprinting collapses repeated fault patterns (the same upstream timeout replaying every few seconds) into a single bundle with a replay count, preventing dozens of identical hypotheses from obscuring the signal.
- The team found statistical methods (MAD, percentile bands) work well enough for metrics anomaly detection and are far easier to debug than ML models; they reserve ML for log clustering and trace structural analysis.
- The system is being extended with LLM-based orchestration to make RCA iterative—an agent can request additional telemetry, refine hypotheses, and adapt its investigation strategy across multiple steps rather than running one-shot.
- A shared incident context anchors all signals, hypotheses, and actions per incident, feeding both a faulty-service pager that pages the right team early and an LLM-powered copilot that recommends mitigations (rollbacks, feature flag disablement) grounded in the actual diagnosis.
- Sequence fingerprinting collapses repeated fault patterns (the same upstream timeout replaying every few seconds) into a single bundle with a replay count, preventing dozens of identical hypotheses from obscuring the signal.
- The team found statistical methods (MAD, percentile bands) work well enough for metrics anomaly detection and are far easier to debug than ML models; they reserve ML for log clustering and trace structural analysis.
- The system is being extended with LLM-based orchestration to make RCA iterative—an agent can request additional telemetry, refine hypotheses, and adapt its investigation strategy across multiple steps rather than running one-shot.
- A shared incident context anchors all signals, hypotheses, and actions per incident, feeding both a faulty-service pager that pages the right team early and an LLM-powered copilot that recommends mitigations (rollbacks, feature flag disablement) grounded in the actual diagnosis.
Prove AI is developing an observability-first foundation designed for production generative AI systems. Their mission is to enable engineering teams to understand, diagnose, and remediate failures within complex AI pipelines, including LLM inference, retrieval processes, and agent orchestration.
The current release, v0.1, provides an opinionated observability pipeline specifically for generative AI workloads through:
- A containerized, OpenTelemetry-based telemetry pipeline.
- Preconfigured collection of traces, metrics, and logs tailored for AI systems.
- Instrumentation patterns for RAG pipelines, embeddings, LLM inference, and agent-based systems.
- Compatibility with standard backends like Prometheus.
The current release, v0.1, provides an opinionated observability pipeline specifically for generative AI workloads through:
- A containerized, OpenTelemetry-based telemetry pipeline.
- Preconfigured collection of traces, metrics, and logs tailored for AI systems.
- Instrumentation patterns for RAG pipelines, embeddings, LLM inference, and agent-based systems.
- Compatibility with standard backends like Prometheus.
The company's transition from fragmented observability tools to a unified system using OpenTelemetry and OneUptime dramatically improved incident response times, reducing MTTR from 41 to 9 minutes. By correlating logs, metrics, and traces through structured logging and intelligent sampling, they eliminated much of the noise and confusion that previously slowed root cause analysis. The shift also reduced the number of dashboards engineers needed to check per incident and significantly lowered the percentage of incidents with unknown causes.
Key practices included instrumenting once with OpenTelemetry, enforcing cardinality limits, and archiving raw data for future analysis. The move away from 100% trace capture and over-instrumentation helped manage data volume while maintaining visibility into anomalies. This transformation emphasized that effective observability isn't about collecting more data, but about designing correlated signals that support intentional diagnosis and reduce cognitive load.
Key practices included instrumenting once with OpenTelemetry, enforcing cardinality limits, and archiving raw data for future analysis. The move away from 100% trace capture and over-instrumentation helped manage data volume while maintaining visibility into anomalies. This transformation emphasized that effective observability isn't about collecting more data, but about designing correlated signals that support intentional diagnosis and reduce cognitive load.
A guide to building a robust logging system in Python, covering structured logging, log levels, handlers, formatters, filters, and integrating logging with modern observability practices.
This article compares three telemetry pipeline solutions – Cribl, Edge Delta, and DIY OpenTelemetry – based on scalability, performance, data management, intelligence, and cost. It details the strengths and weaknesses of each approach to help organizations choose the best solution for their observability and security data needs.
Traceloop's observability tool for LLM applications is now generally available. The company also announced a $6.1 million seed funding round. The platform extends OpenTelemetry to provide better observability for LLM applications, offering insights into model behavior and facilitating experimentation.
The article discusses the challenges faced due to differing observability tools and naming conventions, and how OpenTelemetry's standard naming schemas can streamline workflows and enhance interoperability.
Solomon Hykes, creator of Docker and CEO of Dagger, advocates for containerizing AI agents to manage complexity and enhance reusability. At Sourcegraph’s AI Tools Night, he demonstrated building an AI agent and a cURL clone using Dagger's container-based approach, emphasizing the benefits of standardization and debuggability.
Arize Phoenix is an open-source observability library for AI experimentation, evaluation, and troubleshooting, built by Arize AI.
This article provides an overview of OpenTelemetry, an open-source observability framework, and guides on integrating it with Go applications. It covers key concepts like logs, metrics, and traces, and demonstrates setting up a reusable telemetry package using OpenTelemetry in Go.
