Gordon Bell, engineer, entrepreneur, and visionary, passed away on May 17, 2024. He was instrumental in the development of minicomputers at Digital Equipment Corporation (DEC), co-founded the Digital Computer Museum, and contributed to the growth of Microsoft Research.
This paper proposes a new method called MoRA for parameter-efficient fine-tuning of large language models (LLMs). The proposed method, MoRA, employs a square matrix to achieve high-rank updating, maintaining the same number of trainable parameters. The paper suggests that low-rank updating, as implemented in LoRA, may limit the ability of LLMs to effectively learn and memorize new knowledge. MoRA outperforms LoRA on memory-intensive tasks and achieves comparable performance on other tasks.
Moreover, LAMBDA is the true lambda that we know and love: a lambda can be an argument to another lambda or its result; you can define the Church numerals; lambdas can return lambdas, so you can do currying; you can define a fixed-point combinator using LAMBDA and hence write recursive functions; and so on. (Additionally, since lambdas can be named, they can directly call themselves recursively, which is much more convenient than using a fixed-point combinator.)
