This article explores how temperature and seed values impact the reliability of agentic loops, which combine LLMs with an Observe-Reason-Act cycle. Low temperatures can lead to deterministic loops where agents get stuck, while high temperatures introduce reasoning drift and instability. Fixed seed values in production environments create reproducibility issues, essentially locking the agent into repeating failed reasoning paths. The piece advocates for dynamic adjustment of these parameters during retries, leveraging techniques like raising temperature or randomizing seeds to encourage exploration and escape failure modes, and highlights the benefits of cost-free tools for testing these adjustments.
This blog post explains the causes of nondeterminism in LLM inference, arguing that it's not simply due to floating-point non-associativity and concurrency, but rather a lack of batch invariance in kernels. It details how to achieve batch invariance in RMSNorm, matrix multiplication, and attention, and presents experimental results demonstrating deterministic completions and the benefits for on-policy RL.
