SemanticScuttle - klotz.me

klotz: llm* + performance* + vllm*

0 bookmark(s) - Sort by: Date ↓ / Title / - Bookmarks from other users for this tag

1. vLLM Benchmark

   This repository contains scripts for benchmarking the performance of large language models (LLMs) served using vLLM.

   2024-08-24 Tags: vllm, benchmark, llm, performance, backprop.co by klotz
2. Benchmarks show even an old Nvidia RTX 3090 is enough to serve LLMs to thousands

   A startup called Backprop has demonstrated that a single Nvidia RTX 3090 GPU, released in 2020, can handle serving a modest large language model (LLM) like Llama 3.1 8B to over 100 concurrent users with acceptable throughput. This suggests that expensive enterprise GPUs may not be necessary for scaling LLMs to a few thousand users.

   2024-08-24 Tags: nvidia, rtx 3090, llm, gpu, performance, benchmark, llama 3.1 8b, vllm, production engineering, backprop.co by klotz

Top of the page

First / Previous / Next / Last / Page 1 of 0