Natalie Wolchover writes that the 2014 invention of the attention mechanism by Dzmitry Bahdanau, Yoshua Bengio, and Kyunghyun Cho unlocked scalable machine translation, which became the foundation for transformer architectures and the broader large language model revolution. Wolchover examines how attention functions as a mathematical operation allowing neural networks to link words and build context, and whether this mechanism represents the core of intelligence. She concludes that while language modeling is so demanding it may be "AI-complete," human thought likely relies on non-linguistic processes as well, and that current transformer architectures remain far less efficient than the human brain at accomplishing similar tasks.
- Bahdanau was a 23-year-old intern at the University of Montreal in 2014 who described attention as "just obviously the right thing to do."
- The Google paper "Attention Is All You Need" (2017) has been cited more than 250,000 times.
- Llion Jones, co-author of "Attention Is All You Need" and CTO of Sakana AI, now believes attention is not sufficient for truly masterful language modeling and that a fundamentally different paradigm is needed.
- A 2024 Nature study suggests human thought activates more brain regions than just language networks, indicating intelligence is not purely linguistic.
- Bahdanau was a 23-year-old intern at the University of Montreal in 2014 who described attention as "just obviously the right thing to do."
- The Google paper "Attention Is All You Need" (2017) has been cited more than 250,000 times.
- Llion Jones, co-author of "Attention Is All You Need" and CTO of Sakana AI, now believes attention is not sufficient for truly masterful language modeling and that a fundamentally different paradigm is needed.
- A 2024 Nature study suggests human thought activates more brain regions than just language networks, indicating intelligence is not purely linguistic.
