This article introduces OpenObserve, an open-source observability platform designed to provide a cost-effective alternative to proprietary services like Datadog. By leveraging cloud object storage such as S3 for storing logs, metrics, and traces, it significantly reduces the financial burden of data retention. Built with Rust for performance, it aims to simplify the observability stack while maintaining high scalability and speed.
- Drastic reduction in costs compared to traditional SaaS monitoring tools
- Unified platform for handling logs, metrics, and traces
- High-performance architecture powered by Rust and object storage integration
- Drastic reduction in costs compared to traditional SaaS monitoring tools
- Unified platform for handling logs, metrics, and traces
- High-performance architecture powered by Rust and object storage integration
Splunk .conf in Denver Sept 2026: The core theme focuses on "Agentic AI"—the transition toward autonomous workflows in security, observability, and operations to manage increasing data complexity and digital risk.
**Sessions**
* **Powering the Rise of the Agentic Enterprise:** Transforming security/ops with scalable, trust-based AI workflows.
* **See Inside Your Agentic AI Applications:** Tracing Python agent interactions to detect hallucinations and tool failures.
* **The Future is Here: What’s New in Splunk Security:** Transitioning to an "Agentic SOC" via automated orchestration.
* **The Platform for Your Agentic Enterprise:** Unlocking cross-domain insights through unified data and AI.
* **What's New in Platform:** Using a unified data fabric for predictive, agentic operations.
* **5 ML Algorithms in Under 5 Lines of Code:** Reducing alert fatigue using simple SPL/MLTK implementations.
* **Accelerate Digital Resilience (Cisco & Splunk):** Maximizing ROI through integrated ecosystem use cases.
* **AI Troubleshooting and Remediation Agents:** Using AI SRE agents to accelerate root cause analysis.
* **Accelerate SmartStore Searches with VAST Data:** Reducing storage costs via InfiniteCache integration.
* **The Cisco & Splunk Partner Playbook:** Strategies for scaling business in the AI era.
* **Accelerating Your Journey to Agentic Observability:** Navigating 4 steps toward agent-driven monitoring.
* **A Compromised Account Alerting System:** Automating detection of suspicious user behavior.
* **Advanced Threat Detection (DSDL & RBA):** Using machine learning for behavioral risk scoring.
* **Advancing Security Operations with ESCU Innovations:** Leveraging new high-fidelity threat intelligence content.
* **Agentic Data Ingestion (Anthropic's Claude):** Automating Splunk Add-On development using GenAI.
* **Agentic Detection Posture Management:** Using AI to rapidly identify and close detection gaps.
* **Agentic Incident Command:** Deploying coordinated AI agent teams for full incident lifecycles.
* **Hands-on Agent Attack Analysis:** Building anomaly detections from raw agent telemetry.
* **Build Your Own SOC AI Agent (Workshop):** Hands-on guide to building custom agents with guardrails.
* **Triage and Response at Scale:** Using Triage and SOP Agents for automated incident response.
* **Agents Are Easy, Trust Is Hard:** Implementing governance in cohesive agentic operations via Cisco Data Fabric.
* **AI Assistant V2 Unlocked (Workshop):** Hands-on with Agent Mode and organizational knowledge integration.
**Sessions**
* **Powering the Rise of the Agentic Enterprise:** Transforming security/ops with scalable, trust-based AI workflows.
* **See Inside Your Agentic AI Applications:** Tracing Python agent interactions to detect hallucinations and tool failures.
* **The Future is Here: What’s New in Splunk Security:** Transitioning to an "Agentic SOC" via automated orchestration.
* **The Platform for Your Agentic Enterprise:** Unlocking cross-domain insights through unified data and AI.
* **What's New in Platform:** Using a unified data fabric for predictive, agentic operations.
* **5 ML Algorithms in Under 5 Lines of Code:** Reducing alert fatigue using simple SPL/MLTK implementations.
* **Accelerate Digital Resilience (Cisco & Splunk):** Maximizing ROI through integrated ecosystem use cases.
* **AI Troubleshooting and Remediation Agents:** Using AI SRE agents to accelerate root cause analysis.
* **Accelerate SmartStore Searches with VAST Data:** Reducing storage costs via InfiniteCache integration.
* **The Cisco & Splunk Partner Playbook:** Strategies for scaling business in the AI era.
* **Accelerating Your Journey to Agentic Observability:** Navigating 4 steps toward agent-driven monitoring.
* **A Compromised Account Alerting System:** Automating detection of suspicious user behavior.
* **Advanced Threat Detection (DSDL & RBA):** Using machine learning for behavioral risk scoring.
* **Advancing Security Operations with ESCU Innovations:** Leveraging new high-fidelity threat intelligence content.
* **Agentic Data Ingestion (Anthropic's Claude):** Automating Splunk Add-On development using GenAI.
* **Agentic Detection Posture Management:** Using AI to rapidly identify and close detection gaps.
* **Agentic Incident Command:** Deploying coordinated AI agent teams for full incident lifecycles.
* **Hands-on Agent Attack Analysis:** Building anomaly detections from raw agent telemetry.
* **Build Your Own SOC AI Agent (Workshop):** Hands-on guide to building custom agents with guardrails.
* **Triage and Response at Scale:** Using Triage and SOP Agents for automated incident response.
* **Agents Are Easy, Trust Is Hard:** Implementing governance in cohesive agentic operations via Cisco Data Fabric.
* **AI Assistant V2 Unlocked (Workshop):** Hands-on with Agent Mode and organizational knowledge integration.
This article explores the evolution of Large Language Model (LLM) explainability, highlighting a shift from static benchmarks to dynamic evaluation frameworks designed to demystify "black-box" AI behaviors. It details key advancements such as SMILE-based local explanations for identifying influential input triggers, budget-friendly proxy models using open-source alternatives, and engineering tools like CometLLM that provide practical observability without requiring deep mathematical expertise. Ultimately, the piece emphasizes combining rigorous statistical analysis with accessible engineering solutions to build more trustworthy and transparent AI systems.
Netflix uses an internal system called Service Topology to maintain a live, queryable dependency graph for thousands of microservices. The platform merges three distinct data sources—eBPF network flow logs (for kernel-level visibility), IPC metrics from instrumented services (for application context), and aggregated distributed traces (for request paths)—to provide engineers with a unified view of runtime connections. This architecture helps teams quickly identify the blast radius of failures, understand upstream dependencies, and resolve incidents more efficiently by visualizing how various components interact in real-time.
- Data integration from eBPF logs, IPC metrics, and distributed traces ensures comprehensive coverage even for uninstrumented services.
- A three-stage aggregation pipeline resolves multi-hop paths into direct application-to-application edges to simplify troubleshooting.
- The processing architecture leverages Apache Pekko Streams across multi-region Kafka consumers.
- The system supports sub-second response times and provides historical time-window aggregations for incident correlation.
- Data integration from eBPF logs, IPC metrics, and distributed traces ensures comprehensive coverage even for uninstrumented services.
- A three-stage aggregation pipeline resolves multi-hop paths into direct application-to-application edges to simplify troubleshooting.
- The processing architecture leverages Apache Pekko Streams across multi-region Kafka consumers.
- The system supports sub-second response times and provides historical time-window aggregations for incident correlation.
This article explores how OpenTelemetry Semantic Conventions for Generative AI provide deep visibility into LLM-powered applications by standardizing the recording of model calls, tool invocations, and token exchanges. It provides a practical walkthrough on exporting telemetry from tools like VS Code Copilot and using the Aspire Dashboard to visualize traces, metrics, and chat-style conversations.
AWS has launched the public preview of OpenTelemetry (OTel) metrics support within Amazon CloudWatch, enabling developers to send metrics directly via the OTLP protocol. This update completes CloudWatch's support for logs, traces, and metrics using open standards.
- Support for high-cardinality metrics with up to 150 labels per metric.
- Integration of PromQL, allowing users to use Prometheus query language within the CloudWatch console and Managed Grafana.
- Automatic enrichment of ingested metrics with AWS resource metadata such as account ID, Region, and resource tags.
- Support for high-cardinality metrics with up to 150 labels per metric.
- Integration of PromQL, allowing users to use Prometheus query language within the CloudWatch console and Managed Grafana.
- Automatic enrichment of ingested metrics with AWS resource metadata such as account ID, Region, and resource tags.
At GrafanaCON 2026, Grafana Labs announced significant updates including the launch of Grafana 13 and a major architectural overhaul for Loki. The new Loki design moves away from replication-at-ingestion toward using Kafka as a durability layer to reduce data duplication and improve query performance. Additionally, the company introduced GCX, a new CLI tool in public preview designed to integrate observability data directly into agentic development environments like Claude Code and Cursor, allowing engineers to resolve production issues without leaving their coding tools.
:
- Loki rearchitected with Kafka to reduce storage overhead and improve query speed.
- Introduction of GCX CLI for seamless observability integration within AI coding agents.
- Launch of Grafana 13 featuring dynamic dashboards and expanded data source support.
- New AI Observability product in public preview for monitoring LLM applications.
:
- Loki rearchitected with Kafka to reduce storage overhead and improve query speed.
- Introduction of GCX CLI for seamless observability integration within AI coding agents.
- Launch of Grafana 13 featuring dynamic dashboards and expanded data source support.
- New AI Observability product in public preview for monitoring LLM applications.
STCLab's SRE team shares their experience building an AI-driven investigation pipeline to automate the triage of Kubernetes alerts. By utilizing HolmesGPT, they implemented a ReAct pattern that allows LLMs to autonomously select tools like Prometheus, Loki, and kubectl based on specific context. The core finding was that high-quality markdown runbooks containing exclusion rules were more critical for successful investigations than the underlying AI model itself.
Key points:
* Implementation of HolmesGPT using the ReAct agent pattern for autonomous troubleshooting.
* Integration with Robusta to manage Slack routing, deduplication, and thread matching.
* The vital role of runbooks in narrowing search spaces and reducing wasted tool calls.
* Comparison between self-hosted models via KubeAI and managed API approaches.
* Significant reduction in manual triage time from 20 minutes to under two minutes per investigation.
Key points:
* Implementation of HolmesGPT using the ReAct agent pattern for autonomous troubleshooting.
* Integration with Robusta to manage Slack routing, deduplication, and thread matching.
* The vital role of runbooks in narrowing search spaces and reducing wasted tool calls.
* Comparison between self-hosted models via KubeAI and managed API approaches.
* Significant reduction in manual triage time from 20 minutes to under two minutes per investigation.
Airbnb's observability engineering team has transitioned from a legacy StatsD and proprietary Veneur-based aggregation pipeline to a modern, open-source stack utilizing OpenTelemetry Protocol (OTLP), the OpenTelemetry Collector, and VictoriaMetrics' vmagent. The new system handles over 100 million samples per second in production while reducing costs by roughly an order of magnitude.
Key technical highlights include:
* Migration strategy using dual-emitting metrics to bridge legacy StatsD libraries with OTLP adoption.
* Performance improvements, including a reduction in JVM CPU time spent on metrics processing from 10% to under 1%.
* Use of vmagent for streaming aggregation and horizontal sharding to manage high-cardinality data.
* Implementation of a zero injection technique within the vmagent tier to solve Prometheus counter reset edge cases.
* A two-layer architecture consisting of stateless router pods and stateful aggregator pods.
Key technical highlights include:
* Migration strategy using dual-emitting metrics to bridge legacy StatsD libraries with OTLP adoption.
* Performance improvements, including a reduction in JVM CPU time spent on metrics processing from 10% to under 1%.
* Use of vmagent for streaming aggregation and horizontal sharding to manage high-cardinality data.
* Implementation of a zero injection technique within the vmagent tier to solve Prometheus counter reset edge cases.
* A two-layer architecture consisting of stateless router pods and stateful aggregator pods.
Prove AI is developing an observability-first foundation designed for production generative AI systems. Their mission is to enable engineering teams to understand, diagnose, and remediate failures within complex AI pipelines, including LLM inference, retrieval processes, and agent orchestration.
The current release, v0.1, provides an opinionated observability pipeline specifically for generative AI workloads through:
- A containerized, OpenTelemetry-based telemetry pipeline.
- Preconfigured collection of traces, metrics, and logs tailored for AI systems.
- Instrumentation patterns for RAG pipelines, embeddings, LLM inference, and agent-based systems.
- Compatibility with standard backends like Prometheus.
The current release, v0.1, provides an opinionated observability pipeline specifically for generative AI workloads through:
- A containerized, OpenTelemetry-based telemetry pipeline.
- Preconfigured collection of traces, metrics, and logs tailored for AI systems.
- Instrumentation patterns for RAG pipelines, embeddings, LLM inference, and agent-based systems.
- Compatibility with standard backends like Prometheus.
