Serdar Yegulalp writes that he tested PrismML'''s Bonsai 27B 1-bit quantized model locally on an RTX 5060 with LM Studio, finding it compact enough for a smartphone but slower than smaller models due to its 27 billion parameters. Despite modest token-per-second speeds, the model delivers strong reasoning and coding quality when thinking is enabled, making it a good trade-off for users prioritizing small footprint and large context over speed.
- 1-bit quantized version is 3.9 GB versus 54 GB original
- Maximum context window is 262,144 tokens; test used 32,767
- Supports speculative decoding and flash attention with open Apache 2.0 weights
- Speculative decoding not usable on 8GB VRAM due to draft model requirement
- Tokenization of code examples took up to two minutes, max output ~40 tokens/sec
- 1-bit quantized version is 3.9 GB versus 54 GB original
- Maximum context window is 262,144 tokens; test used 32,767
- Supports speculative decoding and flash attention with open Apache 2.0 weights
- Speculative decoding not usable on 8GB VRAM due to draft model requirement
- Tokenization of code examples took up to two minutes, max output ~40 tokens/sec
