Ayush Pande writes about running the Cactus Compute Needle 3, a 29MB edge model, on a Raspberry Pi 5 to perform local smart home automation without a dedicated AI HAT. The model uses a unique sliceable ladder architecture that allows it to run on 2 to 20 layers, focusing on machine-level tasks like tool calling and data conversion rather than general chat. He demonstrates how to integrate it with Home Assistant via Python to control a smart plug, achieving impressive decode and prefill speeds.
- The model requires an ARMv7 or newer architecture, meaning it does not run on first-gen Raspberry Pi Zero devices.
- The author achieved a decode rate of 280.2 tokens/second and a prefill rate of 996.4 tokens/second on an 8GB Raspberry Pi 5.
- While designed for specialized tasks, the tool-calling framework is flexible enough to be extended for document analysis or Docker container management.
- The model requires an ARMv7 or newer architecture, meaning it does not run on first-gen Raspberry Pi Zero devices.
- The author achieved a decode rate of 280.2 tokens/second and a prefill rate of 996.4 tokens/second on an 8GB Raspberry Pi 5.
- While designed for specialized tasks, the tool-calling framework is flexible enough to be extended for document analysis or Docker container management.
