Tansu, an open-source, Apache Kafka-compatible messaging broker, challenges traditional approaches by prioritizing statelessness. Instead of replicating data like Kafka, Tansu delegates durability to external storage, allowing for brokers that are lightweight ("cattle," not "pets") and scale rapidly. It supports various storage backends like S3, SQLite, and Postgres, with a particular emphasis on Postgres integration for streamlined data pipelines. Tansu also offers broker-side schema validation and the ability to directly write validated data to open table formats like Iceberg, Delta Lake, or Parquet. The project is written in Rust and seeks contributors.
An article about implementing a durable, distributed, and highly available log using S3, covering topics such as log interfaces, conditional writes, checksums, and failover recovery.
The article discusses strategies for reducing latency and costs in distributed systems by using zone-aware routing techniques. It emphasizes the importance of optimizing network traffic and resource distribution across multiple availability zones to maintain high availability and performance while minimizing data transfer costs.
