Persys is a locally-run device designed to function as a second brain. The repository includes the backend server (Linux only) and the Electron-based desktop application for accessing the server.
llm-tool provides a command-line utility for running large language models locally. It includes scripts for pulling models from the internet, starting them, and managing them using various commands such as 'run', 'ps', 'kill', 'rm', and 'pull'. Additionally, it offers a Python script named 'querylocal.py' for querying these models. The repository also come
Compare the performance of different LLM that can be deployed locally on consumer hardware. The expected good response and scores are generated by GPT-4.
obtain the original LLaMA model weights and place them in ./models
ls ./models 65B 30B 13B 7B tokenizer_checklist.chk tokenizer.model
install Python dependencies
python3 -m pip install -r requirements.txt
convert the 7B model to ggml FP16 format
python3 convert.py models/7B/
quantize the model to 4-bits (using q4_0 method)
./quantize ./models/7B/ggml-model-f16.bin ./models/7B/ggml-model-q4_0.bin q4_0
run the inference
./main -m ./models/7B/ggml-model-q4_0.bin -n 128
