Adam Conway writes that Qwen 3.8 27B, a 17 GB open-weights model running on a single Lenovo ThinkStation PGX, completed a commercial app's license reverse-engineering task in roughly 30 minutes'', statically disassembling arm64 code, recovering a deliberately obscured RSA public key, self-correcting an initial hash mismatch, and producing a working authentication bypass entirely offline.
- The model detected the jailbreak prompt, correctly identified the real developer from the signing certificate, then proceeded to audit the scheme and ultimately built the bypass on its own
- The target app's licensing was described as unusually thorough for its class: one-time online activation, offline signature check, hardware serial binding, embedded revocation list, and signed update path
- The model ran via the Pi harness using only standard Bash-based tools; no execution of the target app occurred until the final bypass demonstration
- Default max reasoning effort means even trivial requests consume hundreds to thousands of tokens, making the self-correction loop verbose but accurate
- Conway frames this as a shift in the threat model: a capable local model with no cloud oversight gives full control to whoever is at the keyboard
- The model detected the jailbreak prompt, correctly identified the real developer from the signing certificate, then proceeded to audit the scheme and ultimately built the bypass on its own
- The target app's licensing was described as unusually thorough for its class: one-time online activation, offline signature check, hardware serial binding, embedded revocation list, and signed update path
- The model ran via the Pi harness using only standard Bash-based tools; no execution of the target app occurred until the final bypass demonstration
- Default max reasoning effort means even trivial requests consume hundreds to thousands of tokens, making the self-correction loop verbose but accurate
- Conway frames this as a shift in the threat model: a capable local model with no cloud oversight gives full control to whoever is at the keyboard
- **Inference** – Platforms and engines for running models, plus user interfaces.
- **Models** – LLMs (general, coding, multimodal, image, audio), model providers, and specific model highlights.
- **RAG** – Retrieval-Augmented Generation tools.
- **Safeguards** – Safety and content filtering.
- **Agents & Tools** – Agent frameworks, Model Context Protocol, coding agents, computer/browser automation, memory management, and testing/evaluation.
- **Research, Training & Fine-tuning** – Security, sandboxing, and model development.
- **Hardware** – Local hardware options.
- **Tutorials** – Guides covering models, prompt/context engineering, inference, agents, and RAG.
- **Communities** – Places to connect and share knowledge.
- **Models** – LLMs (general, coding, multimodal, image, audio), model providers, and specific model highlights.
- **RAG** – Retrieval-Augmented Generation tools.
- **Safeguards** – Safety and content filtering.
- **Agents & Tools** – Agent frameworks, Model Context Protocol, coding agents, computer/browser automation, memory management, and testing/evaluation.
- **Research, Training & Fine-tuning** – Security, sandboxing, and model development.
- **Hardware** – Local hardware options.
- **Tutorials** – Guides covering models, prompt/context engineering, inference, agents, and RAG.
- **Communities** – Places to connect and share knowledge.
Anurag Singh replaced five Python scripts (backup, organizer, renamer, cleaner, watchdog) with a local LLM agent, which made errors the scripts didn't (wrong directories, skipped steps, false success reports).Each of the original scripts followed explicit rules through a scheduler; the agent instead added a longer inference chain (inspect, interpret, choose a tool, build a command, execute, review) to tasks that fixed logic already described completely, while also holding a loaded model in memory between runs.
- AutomationBench scores for frontier models remain well under 20%: GPT-5.6 Sol 18.1%, GPT-5.5 12.9%, Claude Opus 4.8 15.5%, Gemini 3.5 Flash 14.5%
- Granting an LLM system-level access creates a prompt-injection vector: a malicious file on disk could carry instructions the agent interprets as commands
- Singh's proposed fix: let the agent classify and route ambiguous requests, then hand off to a validator + fixed script for the actual filesystem action
- The five original scripts covered photo backup, extension-based Downloads sorting, file renaming, app-cache clearing, and a disk-threshold alert
- AutomationBench scores for frontier models remain well under 20%: GPT-5.6 Sol 18.1%, GPT-5.5 12.9%, Claude Opus 4.8 15.5%, Gemini 3.5 Flash 14.5%
- Granting an LLM system-level access creates a prompt-injection vector: a malicious file on disk could carry instructions the agent interprets as commands
- Singh's proposed fix: let the agent classify and route ambiguous requests, then hand off to a validator + fixed script for the actual filesystem action
- The five original scripts covered photo backup, extension-based Downloads sorting, file renaming, app-cache clearing, and a disk-threshold alert
Anurag Singh replaced his home lab cron scripts with Qwen3.5 9B using an agent harness with shell access. He expected contextual reasoning to be superior to rigid automation. The local model succeeded in identifying ballooned directories or judging if a container restart was needed, but it failed more often, sometimes stalling or silently skipping checks.He concluded that deterministic scripts remain the more dependable choice for routine tasks and pointed to n8n as a sensible middle ground when the friction is writing and maintaining code rather than the logic itself.
- A 9-billion-parameter local model needs several GB of RAM just to load weights, which is painful on a home server already running Docker, DNS, and other services.
- Singh's specific hardware ceiling: roughly 14B parameters on a 16 GB MacBook, maybe 32B on an M5 Pro, beyond which you need a dedicated rig.
- His suggested hybrid: let the local model read an error log and draft a short explanation, then have n8n relay that summary without granting the model permission to restart or modify anything.
The model's failure mode was not wrong commands but an inconsistent process—the same prompt and the same system state, yet different execution paths on successive runs.
- A 9-billion-parameter local model needs several GB of RAM just to load weights, which is painful on a home server already running Docker, DNS, and other services.
- Singh's specific hardware ceiling: roughly 14B parameters on a 16 GB MacBook, maybe 32B on an M5 Pro, beyond which you need a dedicated rig.
- His suggested hybrid: let the local model read an error log and draft a short explanation, then have n8n relay that summary without granting the model permission to restart or modify anything.
The model's failure mode was not wrong commands but an inconsistent process—the same prompt and the same system state, yet different execution paths on successive runs.
Qwen 3.8-27B Outperforms Meta’s Muse Glimmer in Local Inference.
Julian Horsey reports Alibaba's Qwen 3.8-27B, a 27B model derived from the 2.4T Qwen 3.8 Max, beats Meta's Muse Glimmer in local AI benchmarks, offering a resource-efficient option for Nvidia, AMD, and Apple Mac MLX deployments, with FP8 and NVFP4 quantization support for quality under VRAM limits.
- SG Lang paired with NVFP4 quantization exceeds 200 tokens/sec, outpacing VLLM and Llama.cpp alternatives.
- Four reasoning tiers (none, low, medium, X-high) trade token cost against output nuance; medium suits general tasks, X-high targets detailed analyses.
- Speculative decoding via multi-threaded processing (MTP) set to 3 further boosts generation speed.
- Over-aggressive quantization risks repeated reasoning loops, degrading coherence on limited-VRAM systems.
- Upcoming "thinking cap" fine-tunes and fused kernels are expected to cut token usage and raise throughput.
Julian Horsey reports Alibaba's Qwen 3.8-27B, a 27B model derived from the 2.4T Qwen 3.8 Max, beats Meta's Muse Glimmer in local AI benchmarks, offering a resource-efficient option for Nvidia, AMD, and Apple Mac MLX deployments, with FP8 and NVFP4 quantization support for quality under VRAM limits.
- SG Lang paired with NVFP4 quantization exceeds 200 tokens/sec, outpacing VLLM and Llama.cpp alternatives.
- Four reasoning tiers (none, low, medium, X-high) trade token cost against output nuance; medium suits general tasks, X-high targets detailed analyses.
- Speculative decoding via multi-threaded processing (MTP) set to 3 further boosts generation speed.
- Over-aggressive quantization risks repeated reasoning loops, degrading coherence on limited-VRAM systems.
- Upcoming "thinking cap" fine-tunes and fused kernels are expected to cut token usage and raise throughput.
Thomas Joos writes that choosing LLM deployment now means picking between cloud APIs, datacenter GPUs or local workstations, with technical limits set by memory and bandwidth rather than raw compute and software choice tied to load profile. Quantization shrinks models to fit consumer GPUs and unified-memory systems, while token generation is bandwidth bound so bandwidth per dollar matters more than TFLOPS. Economically, self-hosting an eight-H100 cluster costs about $237k per year over three years versus higher on-demand cloud pricing, but break-even needs 50-83% sustained utilization that teams typically reach only 40-65%, and the strongest case for local inference is sovereignty and resilience not pure cost.
- Mistral Large 3 ships as a 675B-parameter mixture-of-experts with 41B active per token under Apache 2.0
- Nvidia DGX Spark offers 128 GB unified memory at $4,699 list price since Feb 23 2026
- vLLM generates 44x the tokens per second of llama.cpp at 64 concurrent users on H200 with Llama 3.1 8B
- DevTk finds self-hosting breaks even only at several billion tokens per month versus budget hosted open models
- Martin Kuppinger says sovereignty is not an end in itself; security and resilience come first and exit path matters more than immediate rebuild
- Mistral Large 3 ships as a 675B-parameter mixture-of-experts with 41B active per token under Apache 2.0
- Nvidia DGX Spark offers 128 GB unified memory at $4,699 list price since Feb 23 2026
- vLLM generates 44x the tokens per second of llama.cpp at 64 concurrent users on H200 with Llama 3.1 8B
- DevTk finds self-hosting breaks even only at several billion tokens per month versus budget hosted open models
- Martin Kuppinger says sovereignty is not an end in itself; security and resilience come first and exit path matters more than immediate rebuild
Gilbert Pagayon writes that Alibaba'''s Qwen team released Qwen3.8-27B, a 27-billion-parameter open-weight model under Apache 2.0 that can run quantized versions on consumer hardware for coding, research, multimodal tasks and long agent work while keeping data local. The model uses a dense hybrid-attention architecture with flexible reasoning mode and multi-token prediction, and Alibaba'''s benchmarks show it outperforming Claude Opus 4.6 Max on some coding tests but lagging on others. Early hands-on reports praise its capability for its size, though performance varies with quantization, hardware and settings, making the Opus rivalry a signal rather than a verdict.
- Native context is 262,144 tokens, extendable to ~1 million tokens with configuration
- Vision encoder handles images and video natively; scores 91.1 on OmniDocBench 1.5
- Q4_K_M GGUF is ~17.8GB, fitting a 24GB GPU; full BF16 is ~54.7GB
- Multi-token prediction speed gains depend on workload and engine and can be slower in some cases
- Native context is 262,144 tokens, extendable to ~1 million tokens with configuration
- Vision encoder handles images and video natively; scores 91.1 on OmniDocBench 1.5
- Q4_K_M GGUF is ~17.8GB, fitting a 24GB GPU; full BF16 is ~54.7GB
- Multi-token prediction speed gains depend on workload and engine and can be slower in some cases
Michael Larabel writes that Meta Superintelligence Labs announced the release of Muse Glimmer, a 30-billion-parameter open model for always-on local agent workflows with weights released under Apache 2.0. The model is sized to run on a single consumer GPU and targets local coding agents, LLM-as-a-judge evaluation and similar uses, having been trained and evaluated for end-to-end agentic task completion, multi-step reasoning and optimized local deployment. Details are posted on research.meta.ai and the model is available on Hugging Face, with Ollama 0.32.7 already adding support.
-
-
Joe Rice-Jones writes that his Asus ROG Flow Z13 tablet with a Ryzen AI Max+ 395 APU and 128 GB unified memory now runs the same local LLMs as his desktop mini PCs via Lemonade, using a model router to automatically send prompts to different sized models for everyday use and agentic coding with Crush, making it his preferred device.
- Tablet is the Kojima Productions Ludens special edition ROG Flow Z13
- Radeon 8060S iGPU has 96 GB of memory allocated in Adrenalin software
- Model tiers include Qwen3.5-2B to gpt-oss-120b MXFP4 ~59 GB on disk
- 120B model loads in 27.6 s, first token in 0.86 s, ~52 tokens/s on iGPU
- Misrouted CPU-only run dropped to 0.44 tokens/s, over 100x slower
- Router uses Qwen3.5-0.8B classifier and logs which rule fired per response
- Crush coding required raising Lemonade context window from 4,096 to 32,768 tokens
- Tablet is the Kojima Productions Ludens special edition ROG Flow Z13
- Radeon 8060S iGPU has 96 GB of memory allocated in Adrenalin software
- Model tiers include Qwen3.5-2B to gpt-oss-120b MXFP4 ~59 GB on disk
- 120B model loads in 27.6 s, first token in 0.86 s, ~52 tokens/s on iGPU
- Misrouted CPU-only run dropped to 0.44 tokens/s, over 100x slower
- Router uses Qwen3.5-0.8B classifier and logs which rule fired per response
- Crush coding required raising Lemonade context window from 4,096 to 32,768 tokens
Qwen Team writes that Qwen3.8-27B is presented as the most capable generation in the Qwen open-model family so far, built on the Qwen3.5 foundation with substantial gains across coding, professional work, research and long-horizon agentic tasks. The model is a 27B-parameter dense causal language model with a vision encoder, native 262,144-token context extensible to 1,000,000 tokens, flexible thinking control with reasoning_effort and preserve_thinking, and Multi-Token Prediction for faster inference. The Hugging Face page hosts Unsloth's GGUF quantizations and provides install and run instructions for llama.cpp, Ollama, Unsloth Studio, LM Studio and other local apps.
