SemanticScuttle - klotz.me

klotz: gguf* + llama.cpp*

0 bookmark(s) - Sort by: Date ↓ / Title / - Bookmarks from other users for this tag

1. Server approved! 4xH100 (320gb vram). Looking for advice

   A user is seeking advice on deploying a new server with 4x H100 GPUs (320GB VRAM) for on-premise AI workloads. They are considering a Kubernetes-based deployment with RKE2, Nvidia GPU Operator, and tools like vLLM, llama.cpp, and Litellm. They are also exploring the option of GPU pass-through with a hypervisor. The post details their current infrastructure and asks for potential gotchas or best practices.

   2025-04-28 Tags: h100, kubernetes, vllm, llama.cpp, gpu, ai, deployment, rke2, litellm, quantization, sxm, fp8, awq, gguf, production engineering, inference engineering, scale, reddit, localllama by klotz
2. TIL: Building llamafiles from Llama 3.2 GGUFs

   A step-by-step guide on building llamafiles from Llama 3.2 GGUFs, including scripting and Dockerization.

   2024-09-28 Tags: llamafile, llama.cpp, llm, llama 3.2, gguf, model quantization, docker, mozilla-ocho by klotz
3. GoogleCloudPlatform/localllm: Run LLMs locally on Cloud Workstations

   - create a custom base image for a Cloud Workstation environment using a Dockerfile
   . Uses:
   
   Quantized models from

   2024-02-08 Tags: llm, google, locallama, github, foss, gguf, huggingface, llama.cpp by klotz
4. Democratizing LLMs: 4-bit Quantization for Optimal LLM Inference

   A deep dive into model quantization with GGUF and llama.cpp and model evaluation with LlamaIndex

   2024-01-15 Tags: llm, gguf, georgi gerganov, llama.cpp, llamaindex, huggingface, rag by klotz

Top of the page

First / Previous / Next / Last / Page 1 of 0