klotz: llama cpp* + self-hosted* + github*

0 bookmark(s) - Sort by: Date ↓ / Title / - Bookmarks from other users for this tag

  1. obtain the original LLaMA model weights and place them in ./models

    ls ./models 65B 30B 13B 7B tokenizer_checklist.chk tokenizer.model

    install Python dependencies

    python3 -m pip install -r requirements.txt

    convert the 7B model to ggml FP16 format

    python3 convert.py models/7B/

    quantize the model to 4-bits (using q4_0 method)

    ./quantize ./models/7B/ggml-model-f16.bin ./models/7B/ggml-model-q4_0.bin q4_0

    run the inference

    ./main -m ./models/7B/ggml-model-q4_0.bin -n 128

    2023-06-05 Tags: , , , , by klotz

Top of the page

First / Previous / Next / Last / Page 1 of 0 SemanticScuttle - klotz.me: Tags: llama cpp + self-hosted + github

About - Propulsed by SemanticScuttle