SemanticScuttle - klotz.me

Tags: vllm* + llm* + production engineering*

0 bookmark(s) - Sort by: Date ↓ / Title /

1. Benchmarks show even an old Nvidia RTX 3090 is enough to serve LLMs to thousands

   A startup called Backprop has demonstrated that a single Nvidia RTX 3090 GPU, released in 2020, can handle serving a modest large language model (LLM) like Llama 3.1 8B to over 100 concurrent users with acceptable throughput. This suggests that expensive enterprise GPUs may not be necessary for scaling LLMs to a few thousand users.

   2024-08-24 Tags: nvidia, rtx 3090, llm, gpu, performance, benchmark, llama 3.1 8b, vllm, production engineering, backprop.co by klotz
2. vLLM: Serve LLMs at Scale

   High-performance deployment of the vLLM serving engine, optimized for serving large language models at scale.

   2024-08-16 Tags: vllm, llm, scalability, openai, api, production engineering by klotz

Top of the page

First / Previous / Next / Last / Page 1 of 0