Automating root cause analysis at scale: Multi-signal correlation for cloud native incident responseSantosh Balaranganathan and colleagues at Atlassian describe their automated root cause analysis system that treats incident diagnosis as a correlation problem across three dimensions: signal type (metrics, logs, traces), time, and service topology. The pipeline scopes the search using OpenTelemetry-derived dependency graphs, detects anomalies independently per signal, temporally aligns co-occurring anomalies into bundles, traverses the graph to determine causal direction, and emits ranked hypotheses with human-readable narratives so responders can validate and act quickly.
- Sequence fingerprinting collapses repeated fault patterns (the same upstream timeout replaying every few seconds) into a single bundle with a replay count, preventing dozens of identical hypotheses from obscuring the signal.
- The team found statistical methods (MAD, percentile bands) work well enough for metrics anomaly detection and are far easier to debug than ML models; they reserve ML for log clustering and trace structural analysis.
- The system is being extended with LLM-based orchestration to make RCA iterative—an agent can request additional telemetry, refine hypotheses, and adapt its investigation strategy across multiple steps rather than running one-shot.
- A shared incident context anchors all signals, hypotheses, and actions per incident, feeding both a faulty-service pager that pages the right team early and an LLM-powered copilot that recommends mitigations (rollbacks, feature flag disablement) grounded in the actual diagnosis.
- Sequence fingerprinting collapses repeated fault patterns (the same upstream timeout replaying every few seconds) into a single bundle with a replay count, preventing dozens of identical hypotheses from obscuring the signal.
- The team found statistical methods (MAD, percentile bands) work well enough for metrics anomaly detection and are far easier to debug than ML models; they reserve ML for log clustering and trace structural analysis.
- The system is being extended with LLM-based orchestration to make RCA iterative—an agent can request additional telemetry, refine hypotheses, and adapt its investigation strategy across multiple steps rather than running one-shot.
- A shared incident context anchors all signals, hypotheses, and actions per incident, feeding both a faulty-service pager that pages the right team early and an LLM-powered copilot that recommends mitigations (rollbacks, feature flag disablement) grounded in the actual diagnosis.
InfoQ writes:
>"Atlassian has outlined a new approach to automating root cause analysis for large-scale cloud-native incidents, using correlation across metrics, logs, distributed traces, and service topology to generate ranked hypotheses about where failures originate and how they propagate"
>"Atlassian has outlined a new approach to automating root cause analysis for large-scale cloud-native incidents, using correlation across metrics, logs, distributed traces, and service topology to generate ranked hypotheses about where failures originate and how they propagate"
Anurag Singh writes that providing Claude Code with read-only access to a SaaS application's server logs allowed the coding agent to identify and propose fixes for real performance issues. By observing error patterns, traces, and metrics directly within the environment rather than relying on manual bug reports, the agent was able to autonomously trace bugs back to specific lines of code across various files.
- The experiment highlights a shift toward AI agents joining the "on-call" workflow by inspecting live operational telemetry.
- To mitigate security risks, it is recommended using Model Context Protocol (MCP) servers to restrict an agent's tools to read-only actions.
- Major observability companies like Sentry and Datadog are already implementing similar features to automate root cause analysis and pull request generation.
- The experiment highlights a shift toward AI agents joining the "on-call" workflow by inspecting live operational telemetry.
- To mitigate security risks, it is recommended using Model Context Protocol (MCP) servers to restrict an agent's tools to read-only actions.
- Major observability companies like Sentry and Datadog are already implementing similar features to automate root cause analysis and pull request generation.
Over 160 production-ready skill documents (SKILL.md) for LLM coding agents like Claude Code, Cursor, and Codex.Each skill contains expert-level configs, CLI commands, troubleshooting guides, and ready-to-run scripts spanning six domains: DevOps (CI/CD, Kubernetes, observability), Security (vulnerability scanning, secrets, hardening), Infrastructure (AWS/Azure/GCP, networking, databases, local inference stacks), LLM engineering (agent evals, RAG infrastructure, inference scaling), Compliance (SOC2, HIPAA, GDPR, ISO 27001), and IT operations. Skills install in under a minute via the `npx skills add` CLI or a simple git clone, and the agent discovers, matches, and activates them on demand.
- Ships with notable niche skills absent from other repos: MCP server security, eBPF kernel-level observability, OpenTofu migration, agent-eval CI/CD gates, and multi-tenant LLM hosting with KEDA autoscaling.
- Ships with notable niche skills absent from other repos: MCP server security, eBPF kernel-level observability, OpenTofu migration, agent-eval CI/CD gates, and multi-tenant LLM hosting with KEDA autoscaling.
Charles Humble writes about a September 10 live webinar in which AWS OpenSearch will demo two new capabilities aimed at fixing threshold alerting that breaks down at scale: Piped Processing Language (PPL) for chaining multi-step alert conditions across logs, metrics, and traces in a familiar Unix pipeline style, and a unified Alert Manager for centralized routing, suppression, and escalation of alert rules. The session, led by senior PM Joshua Bright, targets SREs and platform engineers drowning in false positives and alert-rule sprawl as AI agent telemetry adds yet another high-volume signal to monitor.
- 77% of organizations already treat OpenSearch as a core or supporting piece of their AI infrastructure, per a Linux Foundation report
- Both PPL alerting and the unified Alert Manager ship under Apache 2.0 with no feature gating
- PPL syntax transfers directly across search, analytics, and alerting, so conditions previously too complex to maintain become shareable with teammates
- 77% of organizations already treat OpenSearch as a core or supporting piece of their AI infrastructure, per a Linux Foundation report
- Both PPL alerting and the unified Alert Manager ship under Apache 2.0 with no feature gating
- PPL syntax transfers directly across search, analytics, and alerting, so conditions previously too complex to maintain become shareable with teammates
This article introduces OpenObserve, an open-source observability platform designed to provide a cost-effective alternative to proprietary services like Datadog. By leveraging cloud object storage such as S3 for storing logs, metrics, and traces, it significantly reduces the financial burden of data retention. Built with Rust for performance, it aims to simplify the observability stack while maintaining high scalability and speed.
- Drastic reduction in costs compared to traditional SaaS monitoring tools
- Unified platform for handling logs, metrics, and traces
- High-performance architecture powered by Rust and object storage integration
- Drastic reduction in costs compared to traditional SaaS monitoring tools
- Unified platform for handling logs, metrics, and traces
- High-performance architecture powered by Rust and object storage integration
Google explores the transition from traditional deterministic automation to agentic AI within Site Reliability Engineering. As system complexity grows due to microservices, cloud scale, and increased code generation, Google is implementing SRE AI across the entire software development lifecycle to enhance reliability. The approach includes using agents for automated runbook improvement, advanced anomaly detection, incident management orchestration, and autonomous investigation utilizing observability data.
- Moving from deterministic automation to agentic AI models
- Integration of AI in reliability design and documentation
- Using anomaly detection rather than static thresholds for alerting
- Orchestrating incident response via communication monitoring and automated summaries
- Leveraging historical data through AI Insights for risk management
- Adhering to principles of transparency, security, and agent identity
- Moving from deterministic automation to agentic AI models
- Integration of AI in reliability design and documentation
- Using anomaly detection rather than static thresholds for alerting
- Orchestrating incident response via communication monitoring and automated summaries
- Leveraging historical data through AI Insights for risk management
- Adhering to principles of transparency, security, and agent identity
STCLab's SRE team shares their experience building an AI-driven investigation pipeline to automate the triage of Kubernetes alerts. By utilizing HolmesGPT, they implemented a ReAct pattern that allows LLMs to autonomously select tools like Prometheus, Loki, and kubectl based on specific context. The core finding was that high-quality markdown runbooks containing exclusion rules were more critical for successful investigations than the underlying AI model itself.
Key points:
* Implementation of HolmesGPT using the ReAct agent pattern for autonomous troubleshooting.
* Integration with Robusta to manage Slack routing, deduplication, and thread matching.
* The vital role of runbooks in narrowing search spaces and reducing wasted tool calls.
* Comparison between self-hosted models via KubeAI and managed API approaches.
* Significant reduction in manual triage time from 20 minutes to under two minutes per investigation.
Key points:
* Implementation of HolmesGPT using the ReAct agent pattern for autonomous troubleshooting.
* Integration with Robusta to manage Slack routing, deduplication, and thread matching.
* The vital role of runbooks in narrowing search spaces and reducing wasted tool calls.
* Comparison between self-hosted models via KubeAI and managed API approaches.
* Significant reduction in manual triage time from 20 minutes to under two minutes per investigation.
AWS has released the general availability of its DevOps Agent, a generative AI assistant designed to automate incident investigation and operational tasks. Built on Amazon Bedrock AgentCore, the tool integrates with observability platforms, code repositories, and CI/CD pipelines to autonomously triage issues and correlate telemetry data. New capabilities include support for investigating applications in Azure and on-premises environments, custom agent skills, and personalized reporting.
Key highlights:
* Autonomous incident investigation triggered by webhooks from sources like CloudWatch or PagerDuty.
* Integration with major tools including Datadog, Grafana, Splunk, GitHub, and GitLab.
* Reported performance improvements of up to 75% lower MTTR during preview.
* Pricing model based on cumulative time spent on operational tasks per second.
Key highlights:
* Autonomous incident investigation triggered by webhooks from sources like CloudWatch or PagerDuty.
* Integration with major tools including Datadog, Grafana, Splunk, GitHub, and GitLab.
* Reported performance improvements of up to 75% lower MTTR during preview.
* Pricing model based on cumulative time spent on operational tasks per second.
This article examines the development of Microsoft’s Azure SRE Agent, designed to mitigate operational toil in mission-critical environments. By utilizing an "agentic workflow" of specialized AI agents, Microsoft has integrated automation across the entire software development lifecycle. This human-AI partnership has autonomously resolved over 35,000 incidents and saved more than 50,000 developer hours, accelerating root cause analysis and mitigation while maintaining rigorous governance and human oversight.
