Personal website of Alex L. Zhang, a PhD student at MIT CSAIL focusing on the efficiency and utilization of language models. His research spans ML systems, language model benchmarks, and specialized model development.
Key areas of work include:
- Recursive Language Models (RLMs) and Project Popcorn
- GPU programming competitions via KernelBot and GPU MODE
- Benchmarking capabilities through VideoGameBench and KernelBench
- Development of models like Neo-1 and KernelLLM-8B
Key areas of work include:
- Recursive Language Models (RLMs) and Project Popcorn
- GPU programming competitions via KernelBot and GPU MODE
- Benchmarking capabilities through VideoGameBench and KernelBench
- Development of models like Neo-1 and KernelLLM-8B
This page provides an overview of the 'Missing Semester' course, focusing on the importance of the shell as a powerful tool for computer scientists. It covers motivation, class structure, the basics of navigating and using the shell, and exercises to reinforce learning. The course aims to equip students with practical skills beyond rote memorization of commands, enabling them to automate tasks and solve complex problems efficiently.
A paraphrasing of Gerald Jay Sussman's explanation for MIT's switch from Scheme to Python in its undergraduate computer science program, focusing on the changing nature of programming and the need to adapt to modern systems and libraries.
Researchers at MIT’s CSAIL are charting a more "modular" path ahead for software development, breaking systems into "concepts" and "synchronizations" to make code clearer, safer, and easier for LLMs to generate.
MIT researchers are proposing a new software development approach centered around "concepts" and "synchronizations" to address issues of complexity, safety, and LLM compatibility in modern software.
Concepts are self-contained units of functionality (like "sharing" or "liking") with their own state and actions, whereas synchronizations are explicit rules defining how these concepts interact, expressed in a simple, LLM-friendly language.
The benefits include ncreased modularity, transparency, easier understanding for both humans and AI, improved safety, and potential for automated software development. Real-world application: has been demonstrated by successfully restructuring features (liking, commenting, sharing) to be more modular and legible.
Future includes concept catalogs, a shift in software architecture, and improved collaboration through shared, well-tested concepts.
MIT researchers are proposing a new software development approach centered around "concepts" and "synchronizations" to address issues of complexity, safety, and LLM compatibility in modern software.
Concepts are self-contained units of functionality (like "sharing" or "liking") with their own state and actions, whereas synchronizations are explicit rules defining how these concepts interact, expressed in a simple, LLM-friendly language.
The benefits include ncreased modularity, transparency, easier understanding for both humans and AI, improved safety, and potential for automated software development. Real-world application: has been demonstrated by successfully restructuring features (liking, commenting, sharing) to be more modular and legible.
Future includes concept catalogs, a shift in software architecture, and improved collaboration through shared, well-tested concepts.
A new study by MIT CSAIL researchers maps the challenges of AI in software development, identifying bottlenecks and highlighting research directions to move the field forward, aiming to allow humans to focus on high-level design while automating routine tasks.
An interdisciplinary research project exploring the history and ideas behind the influential ELIZA chatbot, created in the 1960s. The project aims to contextualize ELIZA, analyze its code, and examine its cultural impact on human-computer interaction.
This paper introduces Cross-Layer Attention (CLA), an extension of Multi-Query Attention (MQA) and Grouped-Query Attention (GQA) for reducing the size of the key-value cache in transformer-based autoregressive large language models (LLMs). The authors demonstrate that CLA can reduce the cache size by another 2x while maintaining nearly the same accuracy as unmodified MQA, enabling inference with longer sequence lengths and larger batch sizes.
