This repository provides access to Apple's built-in large language models via Node.js and Python packages, requiring only macOS 26+ on Apple Silicon and the Xcode Command Line Tools. It offers two tiers of interaction: an on-device tier using a small sparse model (AFM 3 Core Advanced) that ensures privacy and guaranteed JSON output through constrained decoding, and a cloud tier via Apple's Private Cloud Compute which provides more powerful reasoning capabilities but sends prompts off the machine.
- The library includes both npm (`apple-llm`) and PyPI (`apple-llm`) packages.
- On-device models are noted to be poor at code generation and long reasoning tasks.
- Structured JSON output is guaranteed on-device via a specific "GenerationSchema" dialect used by Apple's decoder.
- The cloud tier uses Shortcuts as an intermediary because the direct Private Cloud Compute API requires restricted developer entitlements.
- Users can utilize built-in tools like OCR, barcode reading, and Spotlight semantic search for local RAG (macOS 27+).
- The library includes both npm (`apple-llm`) and PyPI (`apple-llm`) packages.
- On-device models are noted to be poor at code generation and long reasoning tasks.
- Structured JSON output is guaranteed on-device via a specific "GenerationSchema" dialect used by Apple's decoder.
- The cloud tier uses Shortcuts as an intermediary because the direct Private Cloud Compute API requires restricted developer entitlements.
- Users can utilize built-in tools like OCR, barcode reading, and Spotlight semantic search for local RAG (macOS 27+).
TokenWatt is a transparent, OpenAI-compatible proxy designed to measure the actual electricity cost of running local Large Language Model (LLM) inference on Apple Silicon hardware. By sitting in front of local inference servers and utilizing Apple's IOReport via SoC rail energy measurements, it provides real-time pricing for requests based on user-defined utility rates without requiring sudo privileges. The tool allows users to compare the cost-efficiency of local execution versus cloud API providers, particularly highlighting the economic advantages of running high-context agentic loops locally where context re-processing is essentially free (limited only by power).
- Uses Apple's IOReport for sudoless energy measurement on macOS/Apple Silicon.
- Provides an OpenAI-compatible interface that forwards requests byte-for-byte to backends like LM Studio or MLX.
- Supports dynamic model discovery so routing updates automatically when models are loaded into memory.
- Offers a calibration feature to replace estimates (±15–30%) with highly accurate measurements via smart plugs.
- Uses Apple's IOReport for sudoless energy measurement on macOS/Apple Silicon.
- Provides an OpenAI-compatible interface that forwards requests byte-for-byte to backends like LM Studio or MLX.
- Supports dynamic model discovery so routing updates automatically when models are loaded into memory.
- Offers a calibration feature to replace estimates (±15–30%) with highly accurate measurements via smart plugs.
>"I Measured Every Watt on Apple Silicon Five models, sustained generation, real wall-socket energy at $0.31/kWh — and the surprise the RTX-3090 numbers predicted, only bigger."
Justin Stewart writes about how the energy cost of running local Large Language Models (LLMs) on Apple Silicon depends more on throughput than parameter count. Using an M3 Ultra Mac Studio, he demonstrates that large Mixture-of-Experts (MoE) models can be significantly cheaper to operate per token than smaller dense models because they only activate a fraction of their parameters during generation. Ultimately, the study reveals that efficiency is driven by how much data must be moved from memory for every token produced.
* The measurements were calibrated against actual wall power using a Shelly Plug US Gen4 meter.
* A custom tool called TokenWatt was used to measure marginal energy consumption via Apple’s IOReport interface.
* In real-world "lumpy" traffic scenarios, the cost of dense models compared to MoE models actually widens even further.
Justin Stewart writes about how the energy cost of running local Large Language Models (LLMs) on Apple Silicon depends more on throughput than parameter count. Using an M3 Ultra Mac Studio, he demonstrates that large Mixture-of-Experts (MoE) models can be significantly cheaper to operate per token than smaller dense models because they only activate a fraction of their parameters during generation. Ultimately, the study reveals that efficiency is driven by how much data must be moved from memory for every token produced.
* The measurements were calibrated against actual wall power using a Shelly Plug US Gen4 meter.
* A custom tool called TokenWatt was used to measure marginal energy consumption via Apple’s IOReport interface.
* In real-world "lumpy" traffic scenarios, the cost of dense models compared to MoE models actually widens even further.
This Gist details a solution for reclaiming control over a macOS system when using older versions of Bash and the sandboxed App Store version of WireGuard cause issues. The core problem is that macOS's system Bash is outdated (version 3.2), which breaks modern scripts. Additionally, the App Store WireGuard client stores configurations in the Keychain, making them inaccessible to the command-line interface.
The solution involves installing a modern Bash and WireGuard tools via Homebrew, fixing terminal issues with terminfo, configuring the root environment's PATH, and rescuing WireGuard configurations from the Keychain. By following these steps, users can establish a functional WireGuard CLI setup on their macOS systems, bypassing the limitations imposed by the system's default configuration.
The solution involves installing a modern Bash and WireGuard tools via Homebrew, fixing terminal issues with terminfo, configuring the root environment's PATH, and rescuing WireGuard configurations from the Keychain. By following these steps, users can establish a functional WireGuard CLI setup on their macOS systems, bypassing the limitations imposed by the system's default configuration.
M5 delivers over 4x the peak GPU compute performance for AI compared to M4, featuring a next-generation GPU with a Neural Accelerator in each core, a more powerful CPU, a faster Neural Engine, and higher unified memory bandwidth.
An Apple study shows that large language models (LLMs) can improve performance by using a checklist-based reinforcement learning scheme, similar to a simple productivity trick of checking one's work.
A retrospective on the MOS Technology 6502 microprocessor, celebrating its 50th anniversary. The article discusses its history, impact on early home computers and consoles, and its continued (though niche) presence in modern electronics.
Judge Yvonne Gonzalez Rogers is highly critical of Apple's actions following the initial ruling in the Epic Games case, finding that Apple deliberately defied court orders and engaged in anticompetitive behavior to protect its App Store revenue.
>"The Court enjoins Apple from implementing its new anticompetitive acts to avoid compliance with the Injunction. Effective immediately Apple will no longer impede developers’ ability to communicate with users nor will they levy or impose a new commission on off-app purchases.”
>"The Court enjoins Apple from implementing its new anticompetitive acts to avoid compliance with the Injunction. Effective immediately Apple will no longer impede developers’ ability to communicate with users nor will they levy or impose a new commission on off-app purchases.”
The Vivarium Project was an Apple-sponsored research program led by Alan Kay and Ann Marion, focused on modeling animal minds to create new programming languages and environments, especially for children. It explored agent-based systems and end-user programming.
