The article explores the architectural changes that enable DeepSeek's models to perform well with fewer resources, focusing on Multi-Head Latent Attention (MLA). It discusses the evolution of attention mechanisms, from Bahdanau to Transformer's Multi-Head Attention (MHA), and introduces Grouped-Query Attention (GQA) as a solution to MHA's memory inefficiencies. The article highlights DeepSeek's competitive performance despite lower reported training costs.
The article provides a detailed exploration of DeepSeek’s innovative attention mechanism, highlighting its significance in achieving state-of-the-art performance in various benchmarks. It dispels common myths about the training costs associated with DeepSeek models and emphasizes its resource efficiency compared to other large language models.
Scroll Wikipedia
Perplexity AI's founder Aravind Srinivas outlines a vision where AI agents become the target audience for digital advertising, potentially replacing human attention.
Inspectus is a versatile visualization tool for large language models, offering multiple views to provide diverse insights into language model behaviors. It runs in Jupyter notebooks via a Python API and supports visualization of attention maps, token heatmaps, and dimension heatmaps. The library can be installed using pip and provides API documentation and tutorials for Huggingface models and custom attention maps.
A Python-based, open-source visualization tool called Inspectus helps researchers and developers analyze attention patterns in large language models within Jupyter notebooks. It provides an intuitive interface with multiple views, including attention matrices, heatmaps, and dimension heatmaps, to facilitate detailed analysis.
In this paper, the authors propose a new position encoding method, Contextual Position Encoding (CoPE), that allows positions to be conditioned on context by incrementing position only on certain tokens determined by the model. This allows more general position addressing such as attending to the $i$-th particular word, noun, or sentence. The paper demonstrates that CoPE can solve selective copy, counting, and Flip-Flop tasks where popular position embeddings fail, and improves perplexity on language modeling and coding tasks.
This article is part of a series titled ‘LLMs from Scratch’, a complete guide to understanding and building Large Language Models (LLMs). In this article, we discuss the self-attention mechanism and how it is used by transformers to create rich and context-aware transformer embeddings.
The Self-Attention mechanism is used to add context to learned embeddings, which are vectors representing each word in the input sequence. The process involves the following steps:
1. Learned Embeddings: These are the initial vector representations of words, learned during the training phase. The weights matrix, storing the learned embeddings, is stored in the first linear layer of the Transformer architecture.
2. Positional Encoding: This step adds positional information to the learned embeddings. Positional information helps the model understand the order of the words in the input sequence, as transformers process all words in parallel, and without this information, they would lose the order of the words.
3. Self-Attention: The core of the Self-Attention mechanism is to update the learned embeddings with context from the surrounding words in the input sequence. This mechanism determines which words provide context to other words, and this contextual information is used to produce the final contextualized embeddings.
The Self-Attention mechanism is used to add context to learned embeddings, which are vectors representing each word in the input sequence. The process involves the following steps:
1. Learned Embeddings: These are the initial vector representations of words, learned during the training phase. The weights matrix, storing the learned embeddings, is stored in the first linear layer of the Transformer architecture.
2. Positional Encoding: This step adds positional information to the learned embeddings. Positional information helps the model understand the order of the words in the input sequence, as transformers process all words in parallel, and without this information, they would lose the order of the words.
3. Self-Attention: The core of the Self-Attention mechanism is to update the learned embeddings with context from the surrounding words in the input sequence. This mechanism determines which words provide context to other words, and this contextual information is used to produce the final contextualized embeddings.
This paper introduces Cross-Layer Attention (CLA), an extension of Multi-Query Attention (MQA) and Grouped-Query Attention (GQA) for reducing the size of the key-value cache in transformer-based autoregressive large language models (LLMs). The authors demonstrate that CLA can reduce the cache size by another 2x while maintaining nearly the same accuracy as unmodified MQA, enabling inference with longer sequence lengths and larger batch sizes.
Andrej Karpathy's recommended paper reading list, covering various aspects of Language Models (LLMs), including attention mechanisms, unsupervised multi-task learning (GPT-2), instruction-following language models (InstructGPT), LLaMA, reinforcement learning from human feedback (RLAIF), and early experiments of GPT-4, offering insights into significant research developments in LLM and their role in AI landscape, benefiting both novice and experienced AI enthusiasts.
