This article examines the architectural implications of choosing between stateless and stateful designs when building agentic systems. It evaluates how an agent's approach to managing memory impacts deployment, horizontal scaling, and client-side complexity.
- Stateless agents allow for easy horizontal scaling since no user memory is stored on a backend server, but they require the client to send the full conversation history with every request, leading to increased token usage as conversations grow.
- Stateful agents manage their own context through a database layer using session identifiers, which simplifies client interactions and supports complex workflows, though it introduces challenges in distributed scaling and data persistence.
- Stateless agents allow for easy horizontal scaling since no user memory is stored on a backend server, but they require the client to send the full conversation history with every request, leading to increased token usage as conversations grow.
- Stateful agents manage their own context through a database layer using session identifiers, which simplifies client interactions and supports complex workflows, though it introduces challenges in distributed scaling and data persistence.
This article details the steps to move a Large Language Model (LLM) from a prototype to a production-ready system, covering aspects like observability, evaluation, cost management, and scalability.
vLLM Production Stack provides a reference implementation on how to build an inference stack on top of vLLM, allowing for scalable, monitored, and performant LLM deployments using Kubernetes and Helm.
High-performance deployment of the vLLM serving engine, optimized for serving large language models at scale.
