An open-source command-line tool designed to identify the optimal local Large Language Model specifically suited for a user's existing or planned hardware. It automatically detects GPU, CPU, and RAM capacity to rank HuggingFace models using real performance benchmarks instead of relying on parameter size alone.
* Hardware auto-detection for NVIDIA, AMD, Apple Silicon, and CPUs
* Intelligent ranking based on benchmark evidence and recency awareness
* Capability to simulate different GPUs for hardware upgrade planning
* Support for GGUF, AWQ, and GPTQ model formats
* Streamlined workflows including one-command chat sessions and Python code snippet generation
* Hardware auto-detection for NVIDIA, AMD, Apple Silicon, and CPUs
* Intelligent ranking based on benchmark evidence and recency awareness
* Capability to simulate different GPUs for hardware upgrade planning
* Support for GGUF, AWQ, and GPTQ model formats
* Streamlined workflows including one-command chat sessions and Python code snippet generation
AirLLM is an open-source library that allows large language models to run on consumer hardware using layer-wise inference. By loading layers sequentially, it enables 70B parameter models to operate on as little as 4GB of VRAM. Optimized for research and batch processing, it features block-wise quantization for up to 3x faster performance on Linux and Apple Silicon.
This guide explains how to use tool calling with local LLMs, including examples with mathematical, story, Python code, and terminal functions, using llama.cpp, llama-server, and OpenAI endpoints.
oLLM is a Python library for running large-context Transformers on NVIDIA GPUs by offloading weights and KV-cache to SSDs. It supports models like Llama-3, GPT-OSS-20B, and Qwen3-Next-80B, enabling up to 100K tokens of context on 8-10 GB GPUs without quantization.
This document details how to run Qwen models locally using the Text Generation Web UI (oobabooga), covering installation, setup, and launching the web interface.
