Nvidia has expanded its Jetson lineup with the Jetson AGX Thor Developer Kit, a compact platform that carries the new Jetson T5000 system-on-module. Marketed as a developer system, the dimensions and form factor place it firmly in the realm of a mini PC, although its design and purpose align more with edge AI deployment than home computing.
This blog post details a fine-tuning workflow for the gpt-oss model that recovers post-training accuracy while retaining the performance benefits of FP4. It involves supervised fine-tuning (SFT) on an upcasted BF16 version of the model, followed by quantization-aware training (QAT) using NVIDIA TensorRT Model Optimizer. The article also discusses the benefits of using NVFP4 for even better convergence and accuracy recovery.
Nvidia’s NeMo Retriever models and RAG pipeline make quick work of ingesting PDFs and generating reports based on them. Chalk one up for the plan-reflect-refine architecture.
Not an Ubuntu bug, they say.
This comment details a workaround for nvidia-driver-390 on Ubuntu systems with kernel 6.5.0. It links to related bug reports and provides instructions to add a PPA and install updated drivers.
```
sudo add-apt-repository ppa:dtl131/nvidiaexp
sudo apt update
sudo apt install nvidia-drivers-390
```
```
sudo add-apt-repository ppa:dtl131/nvidiaexp
sudo apt update
sudo apt install nvidia-drivers-390
```
A user, nicholasdavidroberts, expresses gratitude to Daniel for providing a PPA and patched 390 driver that resolved their NVIDIA driver compilation issues on Ubuntu 22.04 with kernel 6.5.0-14.
```
execute_with_retries apt-get install -y -qq gcc-12
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-11 11
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-12 12
update-alternatives --set gcc /usr/bin/gcc-12
```
```
execute_with_retries apt-get install -y -qq gcc-12
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-11 11
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-12 12
update-alternatives --set gcc /usr/bin/gcc-12
```
A user reports issues compiling the NVIDIA driver with kernel 6.5.0-14 on Ubuntu 22.04, specifically for a GeForce GT 750M. They provide a patch and instructions for creating a custom deb package to resolve the issue.
This article details how to accelerate deep learning and LLM inference using Apache Spark, focusing on distributed inference strategies. It covers basic deployment with `predict_batch_udf`, advanced deployment with inference servers like NVIDIA Triton and vLLM, and deployment on cloud platforms like Databricks and Dataproc. It also provides guidance on resource management and configuration for optimal performance.
Running GenAI models is easy. Scaling them to thousands of users, not so much. This guide details avenues for scaling AI workloads from proofs of concept to production-ready deployments, covering API integration, on-prem deployment considerations, hardware requirements, and tools like vLLM and Nvidia NIMs.
