AWS has launched the public preview of OpenTelemetry (OTel) metrics support within Amazon CloudWatch, enabling developers to send metrics directly via the OTLP protocol. This update completes CloudWatch's support for logs, traces, and metrics using open standards.
- Support for high-cardinality metrics with up to 150 labels per metric.
- Integration of PromQL, allowing users to use Prometheus query language within the CloudWatch console and Managed Grafana.
- Automatic enrichment of ingested metrics with AWS resource metadata such as account ID, Region, and resource tags.
- Support for high-cardinality metrics with up to 150 labels per metric.
- Integration of PromQL, allowing users to use Prometheus query language within the CloudWatch console and Managed Grafana.
- Automatic enrichment of ingested metrics with AWS resource metadata such as account ID, Region, and resource tags.
Airbnb's observability engineering team has transitioned from a legacy StatsD and proprietary Veneur-based aggregation pipeline to a modern, open-source stack utilizing OpenTelemetry Protocol (OTLP), the OpenTelemetry Collector, and VictoriaMetrics' vmagent. The new system handles over 100 million samples per second in production while reducing costs by roughly an order of magnitude.
Key technical highlights include:
* Migration strategy using dual-emitting metrics to bridge legacy StatsD libraries with OTLP adoption.
* Performance improvements, including a reduction in JVM CPU time spent on metrics processing from 10% to under 1%.
* Use of vmagent for streaming aggregation and horizontal sharding to manage high-cardinality data.
* Implementation of a zero injection technique within the vmagent tier to solve Prometheus counter reset edge cases.
* A two-layer architecture consisting of stateless router pods and stateful aggregator pods.
Key technical highlights include:
* Migration strategy using dual-emitting metrics to bridge legacy StatsD libraries with OTLP adoption.
* Performance improvements, including a reduction in JVM CPU time spent on metrics processing from 10% to under 1%.
* Use of vmagent for streaming aggregation and horizontal sharding to manage high-cardinality data.
* Implementation of a zero injection technique within the vmagent tier to solve Prometheus counter reset edge cases.
* A two-layer architecture consisting of stateless router pods and stateful aggregator pods.
Cloudflare, an internet infrastructure and security company, has upgraded its logging pipeline by migrating from syslog-ng to OpenTelemetry Collector. This change affects one of Cloudflare's largest data pipelines, processing millions of log events per second.
**Key Points**
* **Motivations:** Language compatibility, easier integration, enhanced metrics, and unified telemetry infrastructure.
* **Custom Components:** Custom exporter, modified file exporter, processors, and rate-limiters.
* **Migration and Future Plans:** Careful rollout, monitoring, and plans for more sophisticated log sampling techniques and open-source contributions.
* **Other Adopters:** Google, Splunk, Shopify, and GitHub are also adopting OpenTelemetry for various use cases.
**Key Points**
* **Motivations:** Language compatibility, easier integration, enhanced metrics, and unified telemetry infrastructure.
* **Custom Components:** Custom exporter, modified file exporter, processors, and rate-limiters.
* **Migration and Future Plans:** Careful rollout, monitoring, and plans for more sophisticated log sampling techniques and open-source contributions.
* **Other Adopters:** Google, Splunk, Shopify, and GitHub are also adopting OpenTelemetry for various use cases.
OpenTelemetry offers a standardized process for observability, but its functionality is a work in progress. Its usefulness depends on the observability tools and platforms used in conjunction with OpenTelemetry.
