SemanticScuttle - klotz.me

Tags: python* + llm* + inference*

0 bookmark(s) - Sort by: Date ↓ / Title /

1. Meet oLLM: A Lightweight Python Library that brings 100K-Context LLM Inference to 8 GB Consumer GPUs via SSD Offload—No Quantization Required

   oLLM is a Python library for running large-context Transformers on NVIDIA GPUs by offloading weights and KV-cache to SSDs. It supports models like Llama-3, GPT-OSS-20B, and Qwen3-Next-80B, enabling up to 100K tokens of context on 8-10 GB GPUs without quantization.

   2025-09-30 Tags: ollm, llm, inference, python, huggingface, pytorch, llama-3, gpt-oss, qwen3-next by klotz
2. Text Generation Web UI

   This document details how to run Qwen models locally using the Text Generation Web UI (oobabooga), covering installation, setup, and launching the web interface.

   2025-04-08 Tags: alibaba, qwen, text generation web ui, oobabooga, llm, inference, llama.cpp, transformers, quantization, python by klotz
3. shawwn/llama: Inference code for LLaMA models

   2023-06-05 Tags: llm, llama, github, inference, python by klotz

Top of the page

First / Previous / Next / Last / Page 1 of 0