Netflix uses an internal system called Service Topology to maintain a live, queryable dependency graph for thousands of microservices. The platform merges three distinct data sources—eBPF network flow logs (for kernel-level visibility), IPC metrics from instrumented services (for application context), and aggregated distributed traces (for request paths)—to provide engineers with a unified view of runtime connections. This architecture helps teams quickly identify the blast radius of failures, understand upstream dependencies, and resolve incidents more efficiently by visualizing how various components interact in real-time.
- Data integration from eBPF logs, IPC metrics, and distributed traces ensures comprehensive coverage even for uninstrumented services.
- A three-stage aggregation pipeline resolves multi-hop paths into direct application-to-application edges to simplify troubleshooting.
- The processing architecture leverages Apache Pekko Streams across multi-region Kafka consumers.
- The system supports sub-second response times and provides historical time-window aggregations for incident correlation.
- Data integration from eBPF logs, IPC metrics, and distributed traces ensures comprehensive coverage even for uninstrumented services.
- A three-stage aggregation pipeline resolves multi-hop paths into direct application-to-application edges to simplify troubleshooting.
- The processing architecture leverages Apache Pekko Streams across multi-region Kafka consumers.
- The system supports sub-second response times and provides historical time-window aggregations for incident correlation.
This article details 12 command-line tools powered by eBPF for debugging and monitoring Linux systems, offering insights into kernel-level behavior with minimal overhead.
1. **bpftrace:** A high-level scripting language for on-demand kernel debugging.
2. **execsnoop:** Catches fleeting processes.
3. **opensnoop:** Identifies files processes are trying to open and any permission issues.
4. **tcplife:** Traces the duration of TCP sessions.
5. **gethostlatency:** Diagnoses DNS resolution delays.
6. **biotop:** Displays real-time disk I/O activity by process.
7. **biolatency:** Shows disk I/O latency histograms.
8. **tcpretrans:** Detects TCP retransmissions, indicating network issues.
9. **tcpconnect:** Traces outbound TCP connections.
10. **tcpaccept:** Traces inbound TCP connections.
11. **biosnoop:** Detailed view of block I/O operations.
12. **bpftool:** Inspects and debugs already running eBPF programs.
1. **bpftrace:** A high-level scripting language for on-demand kernel debugging.
2. **execsnoop:** Catches fleeting processes.
3. **opensnoop:** Identifies files processes are trying to open and any permission issues.
4. **tcplife:** Traces the duration of TCP sessions.
5. **gethostlatency:** Diagnoses DNS resolution delays.
6. **biotop:** Displays real-time disk I/O activity by process.
7. **biolatency:** Shows disk I/O latency histograms.
8. **tcpretrans:** Detects TCP retransmissions, indicating network issues.
9. **tcpconnect:** Traces outbound TCP connections.
10. **tcpaccept:** Traces inbound TCP connections.
11. **biosnoop:** Detailed view of block I/O operations.
12. **bpftool:** Inspects and debugs already running eBPF programs.
With the addition of profiling to OpenTelemetry, we expect continuous production profiling to hit the mainstream.
