Airbnb's observability engineering team has transitioned from a legacy StatsD and proprietary Veneur-based aggregation pipeline to a modern, open-source stack utilizing OpenTelemetry Protocol (OTLP), the OpenTelemetry Collector, and VictoriaMetrics' vmagent. The new system handles over 100 million samples per second in production while reducing costs by roughly an order of magnitude.
Key technical highlights include:
* Migration strategy using dual-emitting metrics to bridge legacy StatsD libraries with OTLP adoption.
* Performance improvements, including a reduction in JVM CPU time spent on metrics processing from 10% to under 1%.
* Use of vmagent for streaming aggregation and horizontal sharding to manage high-cardinality data.
* Implementation of a zero injection technique within the vmagent tier to solve Prometheus counter reset edge cases.
* A two-layer architecture consisting of stateless router pods and stateful aggregator pods.
Key technical highlights include:
* Migration strategy using dual-emitting metrics to bridge legacy StatsD libraries with OTLP adoption.
* Performance improvements, including a reduction in JVM CPU time spent on metrics processing from 10% to under 1%.
* Use of vmagent for streaming aggregation and horizontal sharding to manage high-cardinality data.
* Implementation of a zero injection technique within the vmagent tier to solve Prometheus counter reset edge cases.
* A two-layer architecture consisting of stateless router pods and stateful aggregator pods.
Google Cloud has announced native support for the OpenTelemetry Protocol (OTLP) in its Cloud Trace service, allowing developers to send trace data directly using OTLP and eliminating the need for vendor-specific exporters. This includes increased storage limits for attributes and spans.
Jaeger, a leading open-source distributed tracing platform, releases its version 2, which aligns with the OpenTelemetry framework, bringing improved performance, streamlined architecture, and native support for OTLP. The update includes advanced sampling techniques, expanded ecosystem access, and a more flexible storage implementation.
