Deploying Large Language Models in streaming applications is limited by growing KV cache memory during decoding and an inability to generalize beyond training sequence length, with naive window attention failing once text exceeds the cache size. The authors observe an attention sink phenomenon where models assign strong attention scores to initial tokens even when semantically irrelevant, and find that retaining the KV of those initial sinks together with a sliding window recovers performance. This motivates StreamingLLM, a zero-shot framework that enables LLMs trained on finite windows to generalize to infinite sequence lengths without fine-tuning, achieving stable language modeling up to 4 million tokens on Llama-2, MPT, Falcon and Pythia.
- Attention sinks arise from strong attention to initial tokens acting as a sink for excess attention mass.
- A placeholder token added as a dedicated attention sink during pre-training further improves streaming deployment.
- StreamingLLM achieves up to 22.2x speedup over sliding-window recomputation in streaming settings.
- Paper is ICLR 2024 and code/datasets are released at mit-han-lab/streaming-llm.
- Attention sinks arise from strong attention to initial tokens acting as a sink for excess attention mass.
- A placeholder token added as a dedicated attention sink during pre-training further improves streaming deployment.
- StreamingLLM achieves up to 22.2x speedup over sliding-window recomputation in streaming settings.
- Paper is ICLR 2024 and code/datasets are released at mit-han-lab/streaming-llm.
A research collaboration between a U.S. Air Force cadet and an MIT Lincoln Laboratory researcher explored whether nontechnical service members can develop software via vibe-coding—using prompts to guide generative chatbots in writing code. The study revealed that while large language models are excellent prototyping tools for communicating user needs, they present significant challenges regarding security, accuracy, and the need for rigorous human review when handling sensitive data.
* Capability of nonexperts to create functional application prototypes
* Challenges in scaling from complex tactical uses toward practical document processing tasks
* Security risks associated with unintended data transmission during model interaction
* Capability of nonexperts to create functional application prototypes
* Challenges in scaling from complex tactical uses toward practical document processing tasks
* Security risks associated with unintended data transmission during model interaction
* **Problem:** LLMs struggle to derive reliable meaning from raw sensor signals, often producing non-actionable or factually incorrect interpretations of time-series data.
* **Methodology:** The study implements a structured RAG-based prompt structure that combines water consumption measurements with descriptive statistics and qualitative user information (such as household water practices).
* **Key Finding:** Augmenting prompts with multidimensional contextual information leads to much higher evaluation scores for grounding, pattern recognition, and actionable recommendations.
* **Methodology:** The study implements a structured RAG-based prompt structure that combines water consumption measurements with descriptive statistics and qualitative user information (such as household water practices).
* **Key Finding:** Augmenting prompts with multidimensional contextual information leads to much higher evaluation scores for grounding, pattern recognition, and actionable recommendations.
Personal website of Alex L. Zhang, a PhD student at MIT CSAIL focusing on the efficiency and utilization of language models. His research spans ML systems, language model benchmarks, and specialized model development.
Key areas of work include:
- Recursive Language Models (RLMs) and Project Popcorn
- GPU programming competitions via KernelBot and GPU MODE
- Benchmarking capabilities through VideoGameBench and KernelBench
- Development of models like Neo-1 and KernelLLM-8B
Key areas of work include:
- Recursive Language Models (RLMs) and Project Popcorn
- GPU programming competitions via KernelBot and GPU MODE
- Benchmarking capabilities through VideoGameBench and KernelBench
- Development of models like Neo-1 and KernelLLM-8B
Python implementation of Recursive Language Models for processing unbounded context lengths. Process 100k+ tokens with any LLM by storing context as variables instead of prompts.
Researchers at MIT’s CSAIL are charting a more "modular" path ahead for software development, breaking systems into "concepts" and "synchronizations" to make code clearer, safer, and easier for LLMs to generate.
MIT researchers are proposing a new software development approach centered around "concepts" and "synchronizations" to address issues of complexity, safety, and LLM compatibility in modern software.
Concepts are self-contained units of functionality (like "sharing" or "liking") with their own state and actions, whereas synchronizations are explicit rules defining how these concepts interact, expressed in a simple, LLM-friendly language.
The benefits include ncreased modularity, transparency, easier understanding for both humans and AI, improved safety, and potential for automated software development. Real-world application: has been demonstrated by successfully restructuring features (liking, commenting, sharing) to be more modular and legible.
Future includes concept catalogs, a shift in software architecture, and improved collaboration through shared, well-tested concepts.
MIT researchers are proposing a new software development approach centered around "concepts" and "synchronizations" to address issues of complexity, safety, and LLM compatibility in modern software.
Concepts are self-contained units of functionality (like "sharing" or "liking") with their own state and actions, whereas synchronizations are explicit rules defining how these concepts interact, expressed in a simple, LLM-friendly language.
The benefits include ncreased modularity, transparency, easier understanding for both humans and AI, improved safety, and potential for automated software development. Real-world application: has been demonstrated by successfully restructuring features (liking, commenting, sharing) to be more modular and legible.
Future includes concept catalogs, a shift in software architecture, and improved collaboration through shared, well-tested concepts.
A new study by MIT CSAIL researchers maps the challenges of AI in software development, identifying bottlenecks and highlighting research directions to move the field forward, aiming to allow humans to focus on high-level design while automating routine tasks.
SigLLM is an extension of the Orion library, built to detect anomalies in time series data using LLMs. It provides two types of pipelines for anomaly detection: Prompter (directly prompting LLMs) and Detector (using LLMs to forecast time series).
A new program from MIT helps children understand AI by letting them build small-scale language models.
MIT researchers have developed a method using large language models to detect anomalies in complex systems without the need for training. The approach, called SigLLM, converts time-series data into text-based inputs for the language model to process. Two anomaly detection approaches, Prompter and Detector, were developed and showed promising results in initial tests.
