Automating root cause analysis at scale: Multi-signal correlation for cloud native incident responseSantosh Balaranganathan and colleagues at Atlassian describe their automated root cause analysis system that treats incident diagnosis as a correlation problem across three dimensions: signal type (metrics, logs, traces), time, and service topology. The pipeline scopes the search using OpenTelemetry-derived dependency graphs, detects anomalies independently per signal, temporally aligns co-occurring anomalies into bundles, traverses the graph to determine causal direction, and emits ranked hypotheses with human-readable narratives so responders can validate and act quickly.
- Sequence fingerprinting collapses repeated fault patterns (the same upstream timeout replaying every few seconds) into a single bundle with a replay count, preventing dozens of identical hypotheses from obscuring the signal.
- The team found statistical methods (MAD, percentile bands) work well enough for metrics anomaly detection and are far easier to debug than ML models; they reserve ML for log clustering and trace structural analysis.
- The system is being extended with LLM-based orchestration to make RCA iterative—an agent can request additional telemetry, refine hypotheses, and adapt its investigation strategy across multiple steps rather than running one-shot.
- A shared incident context anchors all signals, hypotheses, and actions per incident, feeding both a faulty-service pager that pages the right team early and an LLM-powered copilot that recommends mitigations (rollbacks, feature flag disablement) grounded in the actual diagnosis.
- Sequence fingerprinting collapses repeated fault patterns (the same upstream timeout replaying every few seconds) into a single bundle with a replay count, preventing dozens of identical hypotheses from obscuring the signal.
- The team found statistical methods (MAD, percentile bands) work well enough for metrics anomaly detection and are far easier to debug than ML models; they reserve ML for log clustering and trace structural analysis.
- The system is being extended with LLM-based orchestration to make RCA iterative—an agent can request additional telemetry, refine hypotheses, and adapt its investigation strategy across multiple steps rather than running one-shot.
- A shared incident context anchors all signals, hypotheses, and actions per incident, feeding both a faulty-service pager that pages the right team early and an LLM-powered copilot that recommends mitigations (rollbacks, feature flag disablement) grounded in the actual diagnosis.
InfoQ writes:
>"Atlassian has outlined a new approach to automating root cause analysis for large-scale cloud-native incidents, using correlation across metrics, logs, distributed traces, and service topology to generate ranked hypotheses about where failures originate and how they propagate"
>"Atlassian has outlined a new approach to automating root cause analysis for large-scale cloud-native incidents, using correlation across metrics, logs, distributed traces, and service topology to generate ranked hypotheses about where failures originate and how they propagate"
Wes Steyn writes that making an AI agent production-ready requires addressing observability, governance, deployment, and evaluation through a shared agent factory pattern. By defining the agent once in a single factory, developers can deploy the same core logic across different hosts—such as interactive consoles, hosted services on Foundry, or automated eval runners—while applying specific security constraints like disabling shell access for cloud environments.
- Observability is achieved via OpenTelemetry to track model turns, tool calls, and token usage.
- Microsoft Purview can be integrated via middleware to screen prompts and responses against organizational policies.
- Hosted agents in Foundry automatically handle telemetry configuration through environment variables.
- Local evaluations use simple functions for quick checks, while hosted evals provide model-graded quality scores like relevance and coherence.
- Observability is achieved via OpenTelemetry to track model turns, tool calls, and token usage.
- Microsoft Purview can be integrated via middleware to screen prompts and responses against organizational policies.
- Hosted agents in Foundry automatically handle telemetry configuration through environment variables.
- Local evaluations use simple functions for quick checks, while hosted evals provide model-graded quality scores like relevance and coherence.
Charles Humble writes about a September 10 live webinar in which AWS OpenSearch will demo two new capabilities aimed at fixing threshold alerting that breaks down at scale: Piped Processing Language (PPL) for chaining multi-step alert conditions across logs, metrics, and traces in a familiar Unix pipeline style, and a unified Alert Manager for centralized routing, suppression, and escalation of alert rules. The session, led by senior PM Joshua Bright, targets SREs and platform engineers drowning in false positives and alert-rule sprawl as AI agent telemetry adds yet another high-volume signal to monitor.
- 77% of organizations already treat OpenSearch as a core or supporting piece of their AI infrastructure, per a Linux Foundation report
- Both PPL alerting and the unified Alert Manager ship under Apache 2.0 with no feature gating
- PPL syntax transfers directly across search, analytics, and alerting, so conditions previously too complex to maintain become shareable with teammates
- 77% of organizations already treat OpenSearch as a core or supporting piece of their AI infrastructure, per a Linux Foundation report
- Both PPL alerting and the unified Alert Manager ship under Apache 2.0 with no feature gating
- PPL syntax transfers directly across search, analytics, and alerting, so conditions previously too complex to maintain become shareable with teammates
Traceway is an OpenTelemetry-native observability platform that combines logs, traces, metrics, session replay, exceptions, AI tracing and on-call paging in one MIT-licensed tool that can be self-hosted with Docker in about 90 seconds or run embedded inside a Go app.
- Provides DuckDB telemetry backend for self-hosted mode and ClickHouse + PostgreSQL for standalone deployments
- Ships agent-first CLI and SKILL.md skills for Claude Code, Cursor and Codex to set up and query telemetry
- Symbolicator resolves minified production errors via memory-mapped .tw files with sub-millisecond lookups
- Provides DuckDB telemetry backend for self-hosted mode and ClickHouse + PostgreSQL for standalone deployments
- Ships agent-first CLI and SKILL.md skills for Claude Code, Cursor and Codex to set up and query telemetry
- Symbolicator resolves minified production errors via memory-mapped .tw files with sub-millisecond lookups
This article introduces OpenObserve, an open-source observability platform designed to provide a cost-effective alternative to proprietary services like Datadog. By leveraging cloud object storage such as S3 for storing logs, metrics, and traces, it significantly reduces the financial burden of data retention. Built with Rust for performance, it aims to simplify the observability stack while maintaining high scalability and speed.
- Drastic reduction in costs compared to traditional SaaS monitoring tools
- Unified platform for handling logs, metrics, and traces
- High-performance architecture powered by Rust and object storage integration
- Drastic reduction in costs compared to traditional SaaS monitoring tools
- Unified platform for handling logs, metrics, and traces
- High-performance architecture powered by Rust and object storage integration
Splunk .conf in Denver Sept 2026: The core theme focuses on "Agentic AI"—the transition toward autonomous workflows in security, observability, and operations to manage increasing data complexity and digital risk.
**Sessions**
* **Powering the Rise of the Agentic Enterprise:** Transforming security/ops with scalable, trust-based AI workflows.
* **See Inside Your Agentic AI Applications:** Tracing Python agent interactions to detect hallucinations and tool failures.
* **The Future is Here: What’s New in Splunk Security:** Transitioning to an "Agentic SOC" via automated orchestration.
* **The Platform for Your Agentic Enterprise:** Unlocking cross-domain insights through unified data and AI.
* **What's New in Platform:** Using a unified data fabric for predictive, agentic operations.
* **5 ML Algorithms in Under 5 Lines of Code:** Reducing alert fatigue using simple SPL/MLTK implementations.
* **Accelerate Digital Resilience (Cisco & Splunk):** Maximizing ROI through integrated ecosystem use cases.
* **AI Troubleshooting and Remediation Agents:** Using AI SRE agents to accelerate root cause analysis.
* **Accelerate SmartStore Searches with VAST Data:** Reducing storage costs via InfiniteCache integration.
* **The Cisco & Splunk Partner Playbook:** Strategies for scaling business in the AI era.
* **Accelerating Your Journey to Agentic Observability:** Navigating 4 steps toward agent-driven monitoring.
* **A Compromised Account Alerting System:** Automating detection of suspicious user behavior.
* **Advanced Threat Detection (DSDL & RBA):** Using machine learning for behavioral risk scoring.
* **Advancing Security Operations with ESCU Innovations:** Leveraging new high-fidelity threat intelligence content.
* **Agentic Data Ingestion (Anthropic's Claude):** Automating Splunk Add-On development using GenAI.
* **Agentic Detection Posture Management:** Using AI to rapidly identify and close detection gaps.
* **Agentic Incident Command:** Deploying coordinated AI agent teams for full incident lifecycles.
* **Hands-on Agent Attack Analysis:** Building anomaly detections from raw agent telemetry.
* **Build Your Own SOC AI Agent (Workshop):** Hands-on guide to building custom agents with guardrails.
* **Triage and Response at Scale:** Using Triage and SOP Agents for automated incident response.
* **Agents Are Easy, Trust Is Hard:** Implementing governance in cohesive agentic operations via Cisco Data Fabric.
* **AI Assistant V2 Unlocked (Workshop):** Hands-on with Agent Mode and organizational knowledge integration.
**Sessions**
* **Powering the Rise of the Agentic Enterprise:** Transforming security/ops with scalable, trust-based AI workflows.
* **See Inside Your Agentic AI Applications:** Tracing Python agent interactions to detect hallucinations and tool failures.
* **The Future is Here: What’s New in Splunk Security:** Transitioning to an "Agentic SOC" via automated orchestration.
* **The Platform for Your Agentic Enterprise:** Unlocking cross-domain insights through unified data and AI.
* **What's New in Platform:** Using a unified data fabric for predictive, agentic operations.
* **5 ML Algorithms in Under 5 Lines of Code:** Reducing alert fatigue using simple SPL/MLTK implementations.
* **Accelerate Digital Resilience (Cisco & Splunk):** Maximizing ROI through integrated ecosystem use cases.
* **AI Troubleshooting and Remediation Agents:** Using AI SRE agents to accelerate root cause analysis.
* **Accelerate SmartStore Searches with VAST Data:** Reducing storage costs via InfiniteCache integration.
* **The Cisco & Splunk Partner Playbook:** Strategies for scaling business in the AI era.
* **Accelerating Your Journey to Agentic Observability:** Navigating 4 steps toward agent-driven monitoring.
* **A Compromised Account Alerting System:** Automating detection of suspicious user behavior.
* **Advanced Threat Detection (DSDL & RBA):** Using machine learning for behavioral risk scoring.
* **Advancing Security Operations with ESCU Innovations:** Leveraging new high-fidelity threat intelligence content.
* **Agentic Data Ingestion (Anthropic's Claude):** Automating Splunk Add-On development using GenAI.
* **Agentic Detection Posture Management:** Using AI to rapidly identify and close detection gaps.
* **Agentic Incident Command:** Deploying coordinated AI agent teams for full incident lifecycles.
* **Hands-on Agent Attack Analysis:** Building anomaly detections from raw agent telemetry.
* **Build Your Own SOC AI Agent (Workshop):** Hands-on guide to building custom agents with guardrails.
* **Triage and Response at Scale:** Using Triage and SOP Agents for automated incident response.
* **Agents Are Easy, Trust Is Hard:** Implementing governance in cohesive agentic operations via Cisco Data Fabric.
* **AI Assistant V2 Unlocked (Workshop):** Hands-on with Agent Mode and organizational knowledge integration.
This article explores the evolution of Large Language Model (LLM) explainability, highlighting a shift from static benchmarks to dynamic evaluation frameworks designed to demystify "black-box" AI behaviors. It details key advancements such as SMILE-based local explanations for identifying influential input triggers, budget-friendly proxy models using open-source alternatives, and engineering tools like CometLLM that provide practical observability without requiring deep mathematical expertise. Ultimately, the piece emphasizes combining rigorous statistical analysis with accessible engineering solutions to build more trustworthy and transparent AI systems.
Netflix uses an internal system called Service Topology to maintain a live, queryable dependency graph for thousands of microservices. The platform merges three distinct data sources—eBPF network flow logs (for kernel-level visibility), IPC metrics from instrumented services (for application context), and aggregated distributed traces (for request paths)—to provide engineers with a unified view of runtime connections. This architecture helps teams quickly identify the blast radius of failures, understand upstream dependencies, and resolve incidents more efficiently by visualizing how various components interact in real-time.
- Data integration from eBPF logs, IPC metrics, and distributed traces ensures comprehensive coverage even for uninstrumented services.
- A three-stage aggregation pipeline resolves multi-hop paths into direct application-to-application edges to simplify troubleshooting.
- The processing architecture leverages Apache Pekko Streams across multi-region Kafka consumers.
- The system supports sub-second response times and provides historical time-window aggregations for incident correlation.
- Data integration from eBPF logs, IPC metrics, and distributed traces ensures comprehensive coverage even for uninstrumented services.
- A three-stage aggregation pipeline resolves multi-hop paths into direct application-to-application edges to simplify troubleshooting.
- The processing architecture leverages Apache Pekko Streams across multi-region Kafka consumers.
- The system supports sub-second response times and provides historical time-window aggregations for incident correlation.
This article explores how OpenTelemetry Semantic Conventions for Generative AI provide deep visibility into LLM-powered applications by standardizing the recording of model calls, tool invocations, and token exchanges. It provides a practical walkthrough on exporting telemetry from tools like VS Code Copilot and using the Aspire Dashboard to visualize traces, metrics, and chat-style conversations.
