This repository provides access to Apple's built-in large language models via Node.js and Python packages, requiring only macOS 26+ on Apple Silicon and the Xcode Command Line Tools. It offers two tiers of interaction: an on-device tier using a small sparse model (AFM 3 Core Advanced) that ensures privacy and guaranteed JSON output through constrained decoding, and a cloud tier via Apple's Private Cloud Compute which provides more powerful reasoning capabilities but sends prompts off the machine.
- The library includes both npm (`apple-llm`) and PyPI (`apple-llm`) packages.
- On-device models are noted to be poor at code generation and long reasoning tasks.
- Structured JSON output is guaranteed on-device via a specific "GenerationSchema" dialect used by Apple's decoder.
- The cloud tier uses Shortcuts as an intermediary because the direct Private Cloud Compute API requires restricted developer entitlements.
- Users can utilize built-in tools like OCR, barcode reading, and Spotlight semantic search for local RAG (macOS 27+).
- The library includes both npm (`apple-llm`) and PyPI (`apple-llm`) packages.
- On-device models are noted to be poor at code generation and long reasoning tasks.
- Structured JSON output is guaranteed on-device via a specific "GenerationSchema" dialect used by Apple's decoder.
- The cloud tier uses Shortcuts as an intermediary because the direct Private Cloud Compute API requires restricted developer entitlements.
- Users can utilize built-in tools like OCR, barcode reading, and Spotlight semantic search for local RAG (macOS 27+).
Muhammed Nazeem writes about a browser extension that provides a sidebar and web interface to interact with local large language models while browsing. It supports various providers like Ollama or OpenAI-compatible APIs, allowing users to chat with files such as PDFs through the sidebar.
- Includes vision model support and internet search capabilities.
- Features beta versions of tab mention, MCP client support, and memory functionality.
- Compatible with Gemini Nano (beta) via Chrome.
- Includes vision model support and internet search capabilities.
- Features beta versions of tab mention, MCP client support, and memory functionality.
- Compatible with Gemini Nano (beta) via Chrome.
This article details how to set up a local AI assistant within a Linux terminal using Ollama and Llama 3.2. It explains the installation process, necessary shell configurations, and practical applications for troubleshooting and understanding system logs and processes. The author demonstrates how to use the AI to explain command outputs, interpret journal logs, and gain insights into disk usage and running processes, improving efficiency and understanding for both beginners and advanced Linux users. It also discusses the benefits and limitations of this approach.
