Traceway is an OpenTelemetry-native observability platform that combines logs, traces, metrics, session replay, exceptions, AI tracing and on-call paging in one MIT-licensed tool that can be self-hosted with Docker in about 90 seconds or run embedded inside a Go app.
- Provides DuckDB telemetry backend for self-hosted mode and ClickHouse + PostgreSQL for standalone deployments
- Ships agent-first CLI and SKILL.md skills for Claude Code, Cursor and Codex to set up and query telemetry
- Symbolicator resolves minified production errors via memory-mapped .tw files with sub-millisecond lookups
- Provides DuckDB telemetry backend for self-hosted mode and ClickHouse + PostgreSQL for standalone deployments
- Ships agent-first CLI and SKILL.md skills for Claude Code, Cursor and Codex to set up and query telemetry
- Symbolicator resolves minified production errors via memory-mapped .tw files with sub-millisecond lookups
This article explores how OpenTelemetry Semantic Conventions for Generative AI provide deep visibility into LLM-powered applications by standardizing the recording of model calls, tool invocations, and token exchanges. It provides a practical walkthrough on exporting telemetry from tools like VS Code Copilot and using the Aspire Dashboard to visualize traces, metrics, and chat-style conversations.
AWS has launched the public preview of OpenTelemetry (OTel) metrics support within Amazon CloudWatch, enabling developers to send metrics directly via the OTLP protocol. This update completes CloudWatch's support for logs, traces, and metrics using open standards.
- Support for high-cardinality metrics with up to 150 labels per metric.
- Integration of PromQL, allowing users to use Prometheus query language within the CloudWatch console and Managed Grafana.
- Automatic enrichment of ingested metrics with AWS resource metadata such as account ID, Region, and resource tags.
- Support for high-cardinality metrics with up to 150 labels per metric.
- Integration of PromQL, allowing users to use Prometheus query language within the CloudWatch console and Managed Grafana.
- Automatic enrichment of ingested metrics with AWS resource metadata such as account ID, Region, and resource tags.
Airbnb's observability engineering team has transitioned from a legacy StatsD and proprietary Veneur-based aggregation pipeline to a modern, open-source stack utilizing OpenTelemetry Protocol (OTLP), the OpenTelemetry Collector, and VictoriaMetrics' vmagent. The new system handles over 100 million samples per second in production while reducing costs by roughly an order of magnitude.
Key technical highlights include:
* Migration strategy using dual-emitting metrics to bridge legacy StatsD libraries with OTLP adoption.
* Performance improvements, including a reduction in JVM CPU time spent on metrics processing from 10% to under 1%.
* Use of vmagent for streaming aggregation and horizontal sharding to manage high-cardinality data.
* Implementation of a zero injection technique within the vmagent tier to solve Prometheus counter reset edge cases.
* A two-layer architecture consisting of stateless router pods and stateful aggregator pods.
Key technical highlights include:
* Migration strategy using dual-emitting metrics to bridge legacy StatsD libraries with OTLP adoption.
* Performance improvements, including a reduction in JVM CPU time spent on metrics processing from 10% to under 1%.
* Use of vmagent for streaming aggregation and horizontal sharding to manage high-cardinality data.
* Implementation of a zero injection technique within the vmagent tier to solve Prometheus counter reset edge cases.
* A two-layer architecture consisting of stateless router pods and stateful aggregator pods.
Prove AI is developing an observability-first foundation designed for production generative AI systems. Their mission is to enable engineering teams to understand, diagnose, and remediate failures within complex AI pipelines, including LLM inference, retrieval processes, and agent orchestration.
The current release, v0.1, provides an opinionated observability pipeline specifically for generative AI workloads through:
- A containerized, OpenTelemetry-based telemetry pipeline.
- Preconfigured collection of traces, metrics, and logs tailored for AI systems.
- Instrumentation patterns for RAG pipelines, embeddings, LLM inference, and agent-based systems.
- Compatibility with standard backends like Prometheus.
The current release, v0.1, provides an opinionated observability pipeline specifically for generative AI workloads through:
- A containerized, OpenTelemetry-based telemetry pipeline.
- Preconfigured collection of traces, metrics, and logs tailored for AI systems.
- Instrumentation patterns for RAG pipelines, embeddings, LLM inference, and agent-based systems.
- Compatibility with standard backends like Prometheus.
"Prove AI is a self-hosted solution designed to accelerate GenAI performance monitoring. It allows AI engineers to capture, customize, and monitor GenAI metrics on their own terms, without vendor lock-in. Built on OpenTelemetry, Prove AI connects to existing OpenTelemetry pipelines and surfaces meaningful metrics quickly.
Key features include a unified web-based interface for consolidating performance metrics like token throughput, latency distributions, and service health. It enables faster debugging, improved time-to-metric, and better measurement of GenAI ROI. The platform is open-source, free to deploy, and offers full control over telemetry data."
Key features include a unified web-based interface for consolidating performance metrics like token throughput, latency distributions, and service health. It enables faster debugging, improved time-to-metric, and better measurement of GenAI ROI. The platform is open-source, free to deploy, and offers full control over telemetry data."
Distributed tracing is crucial for modern observability, offering richer context than logs. However, the volume of tracing data can be overwhelming. Sampling addresses this by selectively retaining data, with two main approaches: head sampling (deciding upfront) and tail sampling (deciding after collecting all spans). Head sampling is simpler but can miss localized issues. Tail sampling, while more accurate, is complex to implement at scale, requiring buffering, stateful processing, and potentially impacting system resilience. Furthermore, sampling inherently affects the accuracy of RED metrics (request rate, error rate, duration), necessitating metric materialization *before* sampling.
This article details building end-to-end observability for LLM applications using FastAPI and OpenTelemetry. It emphasizes a code-first approach, manually designing traces, spans, and semantic attributes to capture the full lifecycle of LLM-powered requests. The guide advocates for a structured approach to tracing RAG workflows, focusing on clear span boundaries, safe metadata capture (hashing prompts/responses), token usage tracking, and integration with observability backends like Jaeger, Grafana Tempo, or specialized LLM platforms. It highlights the importance of understanding LLM behavior beyond traditional infrastructure metrics.
Agoda engineers developed API Agent, a system with zero code and zero deployments that enables a single Model Context Protocol (MCP) server to connect to internal REST or GraphQL APIs. The system is designed to reduce the operational overhead of managing multiple APIs with distinct schemas and authentication methods, allowing teams to query services through AI assistants without building individual MCP servers for each API.
Google Cloud has announced native support for the OpenTelemetry Protocol (OTLP) in its Cloud Trace service, allowing developers to send trace data directly using OTLP and eliminating the need for vendor-specific exporters. This includes increased storage limits for attributes and spans.
