Automating root cause analysis at scale: Multi-signal correlation for cloud native incident responseSantosh Balaranganathan and colleagues at Atlassian describe their automated root cause analysis system that treats incident diagnosis as a correlation problem across three dimensions: signal type (metrics, logs, traces), time, and service topology. The pipeline scopes the search using OpenTelemetry-derived dependency graphs, detects anomalies independently per signal, temporally aligns co-occurring anomalies into bundles, traverses the graph to determine causal direction, and emits ranked hypotheses with human-readable narratives so responders can validate and act quickly.
- Sequence fingerprinting collapses repeated fault patterns (the same upstream timeout replaying every few seconds) into a single bundle with a replay count, preventing dozens of identical hypotheses from obscuring the signal.
- The team found statistical methods (MAD, percentile bands) work well enough for metrics anomaly detection and are far easier to debug than ML models; they reserve ML for log clustering and trace structural analysis.
- The system is being extended with LLM-based orchestration to make RCA iterative—an agent can request additional telemetry, refine hypotheses, and adapt its investigation strategy across multiple steps rather than running one-shot.
- A shared incident context anchors all signals, hypotheses, and actions per incident, feeding both a faulty-service pager that pages the right team early and an LLM-powered copilot that recommends mitigations (rollbacks, feature flag disablement) grounded in the actual diagnosis.
- Sequence fingerprinting collapses repeated fault patterns (the same upstream timeout replaying every few seconds) into a single bundle with a replay count, preventing dozens of identical hypotheses from obscuring the signal.
- The team found statistical methods (MAD, percentile bands) work well enough for metrics anomaly detection and are far easier to debug than ML models; they reserve ML for log clustering and trace structural analysis.
- The system is being extended with LLM-based orchestration to make RCA iterative—an agent can request additional telemetry, refine hypotheses, and adapt its investigation strategy across multiple steps rather than running one-shot.
- A shared incident context anchors all signals, hypotheses, and actions per incident, feeding both a faulty-service pager that pages the right team early and an LLM-powered copilot that recommends mitigations (rollbacks, feature flag disablement) grounded in the actual diagnosis.
The article explores a real-world architectural shift where specialized data tools—Redis for caching/pub-sub, Elasticsearch for search, and Kafka for event streaming—were consolidated into a single database engine: PostgreSQL. The primary motivation was to reduce operational complexity, simplify the infrastructure stack, and minimize the cognitive load on developers by managing one unified system instead of several distributed ones.
Summary points:
- Consolidating specialized tools into PostgreSQL reduces overhead in deployment, monitoring, and data synchronization.
- Modern Postgres features like GIN indexes and Full Text Search can effectively substitute for Elasticsearch in many use cases.
- Utilizing Postgres's LISTEN/NOTIFY or simple table structures can replace lightweight pub-sub needs previously handled by Redis or Kafka.
Summary points:
- Consolidating specialized tools into PostgreSQL reduces overhead in deployment, monitoring, and data synchronization.
- Modern Postgres features like GIN indexes and Full Text Search can effectively substitute for Elasticsearch in many use cases.
- Utilizing Postgres's LISTEN/NOTIFY or simple table structures can replace lightweight pub-sub needs previously handled by Redis or Kafka.
Netflix uses an internal system called Service Topology to maintain a live, queryable dependency graph for thousands of microservices. The platform merges three distinct data sources—eBPF network flow logs (for kernel-level visibility), IPC metrics from instrumented services (for application context), and aggregated distributed traces (for request paths)—to provide engineers with a unified view of runtime connections. This architecture helps teams quickly identify the blast radius of failures, understand upstream dependencies, and resolve incidents more efficiently by visualizing how various components interact in real-time.
- Data integration from eBPF logs, IPC metrics, and distributed traces ensures comprehensive coverage even for uninstrumented services.
- A three-stage aggregation pipeline resolves multi-hop paths into direct application-to-application edges to simplify troubleshooting.
- The processing architecture leverages Apache Pekko Streams across multi-region Kafka consumers.
- The system supports sub-second response times and provides historical time-window aggregations for incident correlation.
- Data integration from eBPF logs, IPC metrics, and distributed traces ensures comprehensive coverage even for uninstrumented services.
- A three-stage aggregation pipeline resolves multi-hop paths into direct application-to-application edges to simplify troubleshooting.
- The processing architecture leverages Apache Pekko Streams across multi-region Kafka consumers.
- The system supports sub-second response times and provides historical time-window aggregations for incident correlation.
Sam Newman discusses the three golden rules of distributed computing and how they necessitate robust handling of timeouts, retries, and idempotency. He provides practical, data-driven strategies for implementing these principles, including using request IDs and server-side fingerprinting to create safe, resilient distributed systems.
Versioning strategies that prevent cascade failures across service boundaries. This article details the importance of schema evolution in microservices, the problems it can cause, and a four-pillar approach to managing it safely: Mandatory Versioning, Expand-and-Contract Migration, Consumer Impact Analysis, and Gradual Rollout with Circuit Breakers. It also includes AWS-specific implementation strategies and advanced patterns.
This article discusses how to choose a software architecture based on project size, domain features, runtime performance requirements, and flexibility needs. It emphasizes that there is no one-size-fits-all solution and advocates for selecting patterns as tools tailored to specific circumstances.
Build Agentic AI with NVIDIA NIM and NeMo. Explore optimized AI models, connect AI agents to data, and deploy anywhere with NVIDIA NIM microservices.
Hydrolix is a streaming data lake platform designed to handle large amounts of immutable log data at a lower cost than traditional solutions. The platform is particularly well-suited for observability data and offers real-time query performance on terabyte-scale data. Hydrolix uses an ANSI-compliant SQL interface, is schema-based and fully indexed, and is designed for high-cardinality data. It is purpose-built for log data and focuses on data that comes in once and never changes. Hydrolix is currently used by companies in industries like media, gaming, ad tech, and telecom security that require long-term retention of data. The company recently announced a $35 million Series B round, and its technology serves as the basis for Akamai's observability product TrafficPeak. The platform is designed to save costs for companies dealing with billions of transactions a day and terabytes of data, as it can store data for longer periods than traditional solutions like Splunk or Datadog, thereby reducing costs or increasing retention.
Red Hat’s Podman, a desktop tool for managing container pods, has been given extended duty as a platform for developers to build generative AI-based applications. Unlike many tools for building generative AI tools, this one was built specifically for developers, not data scientists.
