Unify your existing devices into one powerful GPU: iPhone, iPad, Android, Mac, NVIDIA, Raspberry Pi, pretty much any device!
A USB stick equipped with a Raspberry Pi Zero W runs a large language model using llama.cpp. The project involves porting the model to an ARMv6 architecture and setting up the device as a composite that presents a filesystem to the host, allowing users to interact with the LLM by creating text files that are automatically filled with generated content.
This PR implements the StreamingLLM technique for model loaders, focusing on handling context length and optimizing chat generation speed.
A collection of lightweight AI-powered tools built with LLaMA.cpp and small language models.
A guide on how to download, convert, quantize, and use Llama 3.1 8B model with llama.cpp on a Mac.
A step-by-step guide on building llamafiles from Llama 3.2 GGUFs, including scripting and Dockerization.
Tutorial on enforcing JSON output with Llama.cpp or the Gemini’s API for structured data generation from LLMs.
Large Model Proxy is designed to make it easy to run multiple resource-heavy Large Models (LM) on the same machine with limited amount of VRAM/other resources.
This page provides information about LLooM, a tool that uses raw LLM logits to weave threads in a probabilistic way. It includes instructions on how to use LLooM with various environments, such as vLLM, llama.cpp, and OpenAI. The README also explains the parameters and configurations for LLooM.
An explanation of the quant names used in the llama.cpp implementation, as well as information on the different types of quant schemes available.
