The article delves into how large language models (LLMs) store facts, focusing on the role of multi-layer perceptrons (MLPs) in this process. It explains the mechanics of MLPs, including matrix multiplication, bias addition, and the Rectified Linear Unit (ReLU) function, using the example of encoding the fact that Michael Jordan plays basketball. The article also discusses the concept of superposition, which allows models to store a vast number of features by utilizing nearly perpendicular directions in high-dimensional spaces.
An explanation of the differences between encoder- and decoder-style large language model (LLM) architectures, including their roles in tasks such as classification, text generation, and translation.
A Github Gist containing a Python script for text classification using the TxTail API
This article provides a beginner-friendly introduction to Large Language Models (LLMs) and explains the key concepts in a clear and organized way.
Exploring the architecture of OpenAI’s Generative Pre-trained Transformers.
training GPT to query with few-shot prompting.
