Databricks co-founder and CTO Matei Zaharia has been honored with the 2026 ACM Prize in Computing, recognizing his massive impact on big data through the creation of Spark. Zaharia, an associate professor at UC Berkeley, argues that Artificial General Intelligence (AGI) is already a reality, though it should not be judged by human standards. He warns of the security implications of AI agents that mimic human behavior and expresses optimism about AI's ability to transform research and engineering. By automating complex tasks like molecular simulation and data compilation, Zaharia believes AI will become a universal tool for understanding information.
Apache Spark 4.0 marks a major milestone with advancements in SQL language enhancements, Spark Connect, reliability, Python capabilities, and structured streaming. It's designed to be more powerful, ANSI-compliant, and user-friendly while maintaining compatibility.
This article details how to accelerate deep learning and LLM inference using Apache Spark, focusing on distributed inference strategies. It covers basic deployment with `predict_batch_udf`, advanced deployment with inference servers like NVIDIA Triton and vLLM, and deployment on cloud platforms like Databricks and Dataproc. It also provides guidance on resource management and configuration for optimal performance.
$ bin/spark-shell --packages com.databricks:spark-avro_2.11:3.1.0
