noonghunna writes a single-card RTX 3090 (24 GB) guide for local LLM inference, mapping which models and context lengths fit on one card, what can't be done, and the pitfalls that cause mid-session OOM crashes.
- A hardware cliff ("Cliff 2b") at ~21–26K accumulated tokens makes all single-card vLLM configs unsafe for agent-style workloads that retain context across turns.
- Qwen3.8-27B has an incubating single-card llama.cpp path at 262K context with vision (q4_0 KV + F16 mmproj), but it sits below the project's serving-grade KV floor.
- The 2026-08-12 retirement of all llama.cpp single-card slugs removed 200K context and ~60 TPS support for Qwen3.6-27B, leaving only a 32K, no-vision vLLM path at ~32 TPS.
- A hardware cliff ("Cliff 2b") at ~21–26K accumulated tokens makes all single-card vLLM configs unsafe for agent-style workloads that retain context across turns.
- Qwen3.8-27B has an incubating single-card llama.cpp path at 262K context with vision (q4_0 KV + F16 mmproj), but it sits below the project's serving-grade KV floor.
- The 2026-08-12 retirement of all llama.cpp single-card slugs removed 200K context and ~60 TPS support for Qwen3.6-27B, leaving only a 32K, no-vision vLLM path at ~32 TPS.
Simon Willison reviews Ornith-1.0, a new series of open-weights models from DeepReinforce designed for agentic coding. Built on Gemma 4 and Qwen 3.5, these models range from 9B to 397B parameters and demonstrate high proficiency in coding benchmarks and tool-use tasks.
- Available in 9B Dense, 31B Dense, 35B MoE, and 397B MoE variants
- Capable of navigating codebases and executing complex tool calls
- Built on Apache 2.0 licensed base models
- Available in 9B Dense, 31B Dense, 35B MoE, and 397B MoE variants
- Capable of navigating codebases and executing complex tool calls
- Built on Apache 2.0 licensed base models
An exploration of high-performing small language models with under 7 billion parameters that can run locally on consumer hardware like laptops and smartphones. The article explains how advancements in training data quality, model distillation from larger frontier models, and architectural improvements like Mixture-of-Experts have enabled these compact models to compete with much larger versions on reasoning benchmarks. It provides a curated guide of top available models on Hugging Face, detailing their specific strengths, benchmark performance, and providing Python code for implementation.
Key models covered:
- Qwen3.5-4B for multilingual tasks and long context windows
- Microsoft Phi-4-mini-instruct for reasoning-heavy English workloads
- Google Gemma 3 4B IT for coding and mathematics
- Google Gemma 3n E4B for efficient mobile and on-device deployment
- Meta Llama 3.2 3B Instruct for tool calling and community support
- SmolLM3-3B for research transparency and open-source projects
- DeepSeek-R1-Distill-Qwen-1.5B for lightweight reasoning on edge devices
- Qwen3-0.6B for ultra-constrained hardware and text classification
Key models covered:
- Qwen3.5-4B for multilingual tasks and long context windows
- Microsoft Phi-4-mini-instruct for reasoning-heavy English workloads
- Google Gemma 3 4B IT for coding and mathematics
- Google Gemma 3n E4B for efficient mobile and on-device deployment
- Meta Llama 3.2 3B Instruct for tool calling and community support
- SmolLM3-3B for research transparency and open-source projects
- DeepSeek-R1-Distill-Qwen-1.5B for lightweight reasoning on edge devices
- Qwen3-0.6B for ultra-constrained hardware and text classification
Needle is an experimental 26m parameter Simple Attention Network distilled from Gemini 3.1, designed to redefine tiny AI for consumer devices like phones, watches, and glasses. It specializes in single-shot function calling, outperforming larger models such as FunctionGemma-270m and Qwen-0.6B in that specific domain. The project provides fully open weights, dataset generation methods, and tools for local finetuning on consumer hardware.
* 26m parameter architecture optimized for extremely small devices.
* High performance prefill and decode speeds when running on Cactus.
* Pretrained on 200B tokens and post-trained for function calling.
* Support for local finetuning via a web UI playground or CLI.
* 26m parameter architecture optimized for extremely small devices.
* High performance prefill and decode speeds when running on Cactus.
* Pretrained on 200B tokens and post-trained for function calling.
* Support for local finetuning via a web UI playground or CLI.
Google DeepMind has released four new open-source, vision-capable LLMs under the Apache 2.0 license – Gemma 4, with sizes ranging from 2B to 31B parameters, and a 26B-A4B Mixture-of-Experts model. The models are notable for their intelligence-per-parameter ratio, with the smaller models (E2B and E4B) utilizing Per-Layer Embeddings to maximize efficiency.
The models support both vision and audio input, although audio functionality is not yet fully implemented in tools like LM Studio or Ollama. Testing with LM Studio showed varying results, with the 31B model experiencing output issues. The author also experimented with the models through the llm-gemini API, generating SVG images of a pelican riding a bicycle to assess their visual capabilities.
The models support both vision and audio input, although audio functionality is not yet fully implemented in tools like LM Studio or Ollama. Testing with LM Studio showed varying results, with the 31B model experiencing output issues. The author also experimented with the models through the llm-gemini API, generating SVG images of a pelican riding a bicycle to assess their visual capabilities.
This Hugging Face page details the Gemma 4 31B-it model, an open-weights multimodal model created by Google DeepMind. Gemma 4 can process both text and image inputs, generating text outputs, with smaller models also supporting audio. It comes in various sizes (E2B, E4B, 26B A4B, and 31B) allowing for deployment on diverse hardware, from phones to servers.
The model boasts a context window of up to 256K tokens and supports over 140 languages. It utilizes dense and Mixture-of-Experts (MoE) architectures, excelling in tasks like text generation, coding, and reasoning. The page provides details on model data, training, ethics, usage, limitations, and best practices, along with code snippets for getting started with Transformers.
The model boasts a context window of up to 256K tokens and supports over 140 languages. It utilizes dense and Mixture-of-Experts (MoE) architectures, excelling in tasks like text generation, coding, and reasoning. The page provides details on model data, training, ethics, usage, limitations, and best practices, along with code snippets for getting started with Transformers.
This article explains how to implement function calling with Google’s Gemma 3 27B model. It covers the concept of function calling, the step‑by‑step workflow, and provides a practical example using a Python `convert` function to turn $200,000 into EUR. The post walks through prompting Gemma, parsing its `tool_code` output, executing the function with `eval`, and returning a friendly final response. It also demonstrates how to set up the Google‑GenAI SDK, create a chat session, and extract tool calls. The discussion highlights Gemma’s multilingual, multimodal, and agentic capabilities, making it suitable for real‑world AI assistants that need to interact with external APIs and tools.
This article details research into finding the optimal architecture for small language models (70M parameters), exploring depth-width tradeoffs, comparing different architectures, and introducing Dhara-70M, a diffusion model offering 3.8x faster throughput with improved factuality.
This discussion details performance benchmarks of llama.cpp on an NVIDIA DGX Spark, including tests for various models (gpt-oss-20b, gpt-oss-120b, Qwen3, Qwen2.5, Gemma, GLM) with different context depths and batch sizes.
Extracting structured information effectively and accurately from long unstructured text with LangExtract and LLMs. This article explores Google’s LangExtract framework and its open-source LLM, Gemma 3, demonstrating how to parse an insurance policy to surface details like exclusions.
