Tags: instruction tuning* + high-rank updating* + continual pretraining* + memory* + pretraining* + mathematical reasoning*

0 bookmark(s) - Sort by: Date ↓ / Title /

  1. This paper proposes a new method called MoRA for parameter-efficient fine-tuning of large language models (LLMs). The proposed method, MoRA, employs a square matrix to achieve high-rank updating, maintaining the same number of trainable parameters. The paper suggests that low-rank updating, as implemented in LoRA, may limit the ability of LLMs to effectively learn and memorize new knowledge. MoRA outperforms LoRA on memory-intensive tasks and achieves comparable performance on other tasks.

Top of the page

First / Previous / Next / Last / Page 1 of 0 SemanticScuttle - klotz.me: tagged with "instruction tuning+high-rank updating+continual pretraining+memory+pretraining+mathematical reasoning"

About - Propulsed by SemanticScuttle