Tags: gpu* + llm*

0 bookmark(s) - Sort by: Date ↓ / Title /

  1. The article discusses the competition Nvidia faces from Intel and AMD in the GPU market. While these competitors have introduced new accelerators that match or surpass Nvidia's offerings in terms of memory capacity, performance, and price, Nvidia maintains a strong advantage through its CUDA software ecosystem. CUDA has been a significant barrier for developers switching to alternative hardware due to the effort required to port and optimize existing code. However, both Intel and AMD have developed tools to ease this transition, like AMD's HIPIFY and Intel's SYCL. Despite these efforts, the article notes that the majority of developers now write higher-level code using frameworks like PyTorch, which can run on different hardware with varying levels of support and performance. This shift towards higher-level programming languages has reduced the impact of Nvidia's CUDA moat, though challenges still exist in ensuring compatibility and performance across different hardware platforms.

    2024-12-25 Tags: , , , , , by klotz
  2. The article discusses the challenges and strategies for load testing and infrastructure decisions when self-hosting Large Language Models (LLMs).

  3. Run:ai offers a platform to accelerate AI development, optimize GPU utilization, and manage AI workloads. It is designed for GPUs, offers CLI & GUI interfaces, and supports various AI tools & frameworks.

  4. This blog post provides a guide for optimizing LLM serving performance on Google Kubernetes Engine (GKE) by covering infrastructure decisions, model server optimizations, and best practices for maximizing GPU utilization. It includes recommendations for quantization, GPU selection (G2 vs A3), batching strategies, and leveraging model server features like PagedAttention.

    2024-08-25 Tags: , , , by klotz
  5. A startup called Backprop has demonstrated that a single Nvidia RTX 3090 GPU, released in 2020, can handle serving a modest large language model (LLM) like Llama 3.1 8B to over 100 concurrent users with acceptable throughput. This suggests that expensive enterprise GPUs may not be necessary for scaling LLMs to a few thousand users.

  6. This article explores the concept of quantization in large language models (LLMs) and its benefits, including reducing memory usage and improving performance. It also discusses various quantization methods and their effects on model quality.

    2024-07-14 Tags: , , , by klotz
  7. GPU-accelerated LLMs on Odrange Pi 5, which features a Mali-G610 GPU. The authors used Machine Learning Compilation (MLC) techniques to achieve speeds of 2.3 tok/sec for Llama3-8b, 2.5 tok/sec for Llama2-7b, and 5 tok/sec for RedPajama-3b. They also managed to run a Llama-2 13b model at 1.5 tok/sec on a 16GB version of the Orange Pi 5+.

  8. 2023-12-24 Tags: , , , , by klotz
  9. 2023-08-03 Tags: , , , by klotz

Top of the page

First / Previous / Next / Last / Page 1 of 0 SemanticScuttle - klotz.me: tagged with "gpu+llm"

About - Propulsed by SemanticScuttle