IBM has introduced Granite 4.0 3B Vision, a specialized vision-language model (VLM) engineered for high-fidelity enterprise document data extraction. Unlike monolithic multimodal models, this release uses a modular LoRA adapter architecture, adding approximately 0.5B parameters to the Granite 4.0 Micro base model. This design allows for efficient dual-mode deployment, activating vision capabilities only when multimodal processing is required. The model excels at converting complex visual elements, such as charts and tables, into structured machine-readable formats like JSON, HTML, and CSV. By utilizing a high-resolution tiling mechanism and a DeepStack architecture for improved spatial alignment, Granite 4.0 3B Vision achieves impressive accuracy in tasks like Key-Value Pair extraction and chart reasoning, ranking highly on industry benchmarks.
This guide helps engineers build and ship LLM products by covering the full technical stack. It moves from core mechanics (tokenization, embeddings, attention) to training methodologies (pretraining, SFT, RLHF/DPO) and deployment optimizations (LoRA, quantization, vLLM). The focus is on managing critical production tradeoffs between accuracy, latency, memory, and cost
This research introduces Doc-to-LoRA (D2L), a method for efficiently processing long documents with Large Language Models (LLMs). D2L creates small, adaptable "LoRA" modules that distill key information from a document, allowing the LLM to answer questions without needing the entire document in memory. This significantly reduces latency and memory usage, enabling LLMs to handle contexts much longer than their original capacity and facilitating faster knowledge updates.
IBM announces Granite 3.3, featuring a new speech-to-text model (Granite Speech 3.3 8B), enhanced reasoning capabilities in Granite 3.3 8B Instruct, and RAG-focused LoRA adapters for Granite 3.2. The release also includes activated LoRAs (aLoRAs) for improved efficiency and all models are open source.
Transformer Lab is an open-source application for advanced LLM engineering, allowing users to interact, train, fine-tune, and evaluate large language models on their own computer. It supports various models, hardware, and inference engines and includes features like RAG, dataset building, and a REST API.
A look at this year’s crop of LoRA alternatives, including SVF, SVFT, MiLoRA, PiSSA, and LoRA-XS, all based on SVD (Singular Value Decomposition). The article compares these techniques to the original LoRA method for fine-tuning Large Language Models.
| Method | Description | Key Feature(s) | Reference |
|--------------|---------------------------------------------|---------------------------------------------|-|
| LoRA | Freezes the model and trains a small pair of low-rank “adapter” matrices. | Saves memory and compute cycles by reducing the number of trainable parameters. | arxiv.org/abs/2106.09685 |
| SVF | Uses SVD on the model’s weight matrices and fine-tunes the singular values directly. | More economical in parameters than LoRA; makes tuned models composable. | arxiv.org/abs/2501.06252v2 |
| SVFT | Adds more trainable weights on the diagonal and evaluates various alternatives. | Provides more trainable values than just the diagonal, useful for better fine-tuning. | arxiv.org/abs/2405.19597 |
| PiSSA | Tunes only the large principal values. | Designed to approximate full fine-tuning by adapting the principal singular components. | arxiv.org/abs/2404.02948 |
| MiLoRA | Tunes only the small principal values. | Retains base model’s knowledge while adapting to new tasks. | arxiv.org/abs/2406.09044 |
| LoRA-XS | Similar to PiSSA but with a slightly different mechanism. | Shows good results with significantly fewer parameters than LoRA. | arxiv.org/abs/2405.17604 |
| DoRA | Splits weights into magnitudes and directions then tunes those. | | arxiv.org/abs/2402.09353 |
| AdaLoRA | Complex mechanism for finding the best tuning rank for a given budget of trainable weights. | | arxiv.org/abs/2303.10512 |
| Method | Description | Key Feature(s) | Reference |
|--------------|---------------------------------------------|---------------------------------------------|-|
| LoRA | Freezes the model and trains a small pair of low-rank “adapter” matrices. | Saves memory and compute cycles by reducing the number of trainable parameters. | arxiv.org/abs/2106.09685 |
| SVF | Uses SVD on the model’s weight matrices and fine-tunes the singular values directly. | More economical in parameters than LoRA; makes tuned models composable. | arxiv.org/abs/2501.06252v2 |
| SVFT | Adds more trainable weights on the diagonal and evaluates various alternatives. | Provides more trainable values than just the diagonal, useful for better fine-tuning. | arxiv.org/abs/2405.19597 |
| PiSSA | Tunes only the large principal values. | Designed to approximate full fine-tuning by adapting the principal singular components. | arxiv.org/abs/2404.02948 |
| MiLoRA | Tunes only the small principal values. | Retains base model’s knowledge while adapting to new tasks. | arxiv.org/abs/2406.09044 |
| LoRA-XS | Similar to PiSSA but with a slightly different mechanism. | Shows good results with significantly fewer parameters than LoRA. | arxiv.org/abs/2405.17604 |
| DoRA | Splits weights into magnitudes and directions then tunes those. | | arxiv.org/abs/2402.09353 |
| AdaLoRA | Complex mechanism for finding the best tuning rank for a given budget of trainable weights. | | arxiv.org/abs/2303.10512 |
Sergey Pletenev et al. explore the integration of new knowledge into Large Language Models (LLMs) using Low-Rank Adaptation (LoRA). The study focuses on fine-tuning the Llama-3.1-8B-instruct model with varying amounts of new information while aiming to retain previously learned knowledge. The researchers found that mixing known and new facts in training data yields the best results but also noted potential drawbacks, such as a decline in performance on external benchmarks and a bias towards overrepresented answers when the data is skewed. Additionally, the model sometimes becomes overly confident and hesitant to answer. These findings emphasize the need for careful consideration of training data composition and tuning parameters to balance the incorporation of new knowledge with maintaining overall model capabilities.
This tutorial guides readers on how to fine-tune the Mistral 7B large language model using QLoRA with the Axolotl library, focusing on managing limited GPU resources for efficient training. It covers environment setup, dataset creation, configuration of QLoRA hyperparameters, the fine-tuning process, and testing the fine-tuned model.
The article explores techniques to improve Large Language Model (LLM) accuracy, focusing on Lamini Memory Tuning. It discusses fine-tuning methods like Low-Rank Adaptation (LoRA), the advantages and disadvantages of fine-tuning, and practical steps using Lamini to achieve higher precision in SQL query generation. The author demonstrates a step-by-step approach to creating a high-quality dataset, fine-tuning, and evaluating model accuracy.
This article provides a comprehensive guide on fine-tuning the Llama 3.1 language model using Unsloth for efficient parameter-efficient training. It covers concepts like supervised fine-tuning, LoRA, QLoRA, and practical steps for training on a high-quality dataset.
