klotz: arxiv* + llm*

0 bookmark(s) - Sort by: Date ↓ / Title / - Bookmarks from other users for this tag

  1. The article introduces a new approach to language modeling called test-time scaling, which enhances performance by utilizing additional compute resources during testing. The authors present a method involving a curated dataset and a technique called budget forcing to control compute usage, allowing models to double-check answers and improve reasoning. The approach is demonstrated with the Qwen2.5-32B-Instruct language model, showing significant improvements on competition math questions.

  2. Jupyter notebook for using LlamaIndex with arXiv papers for retrieval-augmented generation (RAG).

    2024-12-26 Tags: , , , , by klotz
  3. The paper titled "Attention Is All You Need" introduces the Transformer, a novel architecture for sequence transduction models that relies entirely on self-attention mechanisms, dispensing with traditional recurrence and convolutions. Key aspects of the model include:

    • Architecture: The Transformer consists of an encoder-decoder structure, with both components utilizing stacked layers of multi-head self-attention mechanisms and feed-forward networks. It avoids recurrence and convolutions, allowing for greater parallelism and faster training.
    • Attention Mechanism: The model uses scaled dot-product attention for computing attention scores, which scales down the dot products to prevent softmax from saturating.
    • Multi-head attention is employed to allow the model to attend to information from different representation subspaces at different positions.
    • Training and Regularization: The authors use the Adam optimizer with a particular learning rate schedule that initially increases the rate and then decreases it based on the number of training steps. They also employ techniques like dropout and label smoothing to regularize the model during training.
    • Performance: The Transformer achieves state-of-the-art results on machine translation benchmarks (WMT 2014 English-to-German and English-to-French), outperforming previous models with significantly less training time and computational resources.
    • Generalization: The model demonstrates strong performance on tasks other than machine translation, such as English constituency parsing, indicating its versatility and ability to learn complex dependencies and structures.

    The paper emphasizes the efficiency and scalability of the Transformer, highlighting its potential for various sequence transduction tasks, and provides a foundation for subsequent advancements in natural language processing and beyond.

  4. A method that uses instruction tuning to adapt LLMs for knowledge-intensive tasks. RankRAG simultaneously trains the models for context ranking and answer generation, enhancing their retrieval-augmented generation (RAG) capabilities.

  5. The article discusses the limitations of Large Language Models (LLMs) in planning and self-verification tasks, and proposes an LLM-Modulo framework to leverage their strengths in a more effective manner. The framework combines LLMs with external model-based verifiers to generate, evaluate, and improve plans, ensuring their correctness and efficiency.

    "Simply put, we take the stance that LLMs are amazing giant external non-veridical memories that can serve as powerful cognitive orthotics for human or machine agents, if rightly used."

  6. In this paper, the authors propose a new position encoding method, Contextual Position Encoding (CoPE), that allows positions to be conditioned on context by incrementing position only on certain tokens determined by the model. This allows more general position addressing such as attending to the $i$-th particular word, noun, or sentence. The paper demonstrates that CoPE can solve selective copy, counting, and Flip-Flop tasks where popular position embeddings fail, and improves perplexity on language modeling and coding tasks.

    2024-06-02 Tags: , , , by klotz
  7. AutoCrawler is a two-stage framework that leverages the hierarchical structure of HTML for progressive understanding and aims to assist crawlers in handling diverse and changing web environments more efficiently. This work introduces a crawler generation task for vertical information web pages and proposes the paradigm of combining LLMs with crawlers, which supports the adaptability of traditional methods and enhances the performance of generative agents in open-world scenarios. Generative agents, empowered by large language models, suffer from poor performance and reusability in open-world scenarios.

    2024-04-28 Tags: , , , , by klotz
  8. First, using the demonstrations significantly outperforms the no demonstrations method even with small k (k = 4), and performance drop from using gold labels to using random labels is consistently small across varying k, in the range of 0.8–1.6%.7 Interestingly, model performance does not increase much as k increases when k ≥ 8, both with gold labels and with random labels.

    2024-02-14 Tags: , , , , by klotz
  9. Large language models exhibit surprising emergent generalization properties, yet also struggle on many simple reasoning tasks such as arithmetic and parity. This raises the question of if and when Transformer models can learn the true algorithm for solving a task. We study the scope of Transformers' abilities in the specific setting of length generalization on algorithmic tasks. Here, we propose a unifying framework to understand when and how Transformers can exhibit strong length generalization on a given task. Specifically, we leverage RASP (Weiss et al., 2021) -- a programming language designed for the computational model of a Transformer -- and introduce the RASP-Generalization Conjecture: Transformers tend to length generalize on a task if the task can be solved by a short RASP program which works for all input lengths. This simple conjecture remarkably captures most known instances of length generalization on algorithmic tasks. Moreover, we leverage our insights to drastically improve generalization performance on traditionally hard tasks (such as parity and addition). On the theoretical side, we give a simple example where the "min-degree-interpolator" model of learning from Abbe et al. (2023) does not correctly predict Transformers' out-of-distribution behavior, but our conjecture does. Overall, our work provides a novel perspective on the mechanisms of compositional generalization and the algorithmic capabilities of Transformers.

    2024-01-07 Tags: , , , by klotz
  10. 2023-12-22 Tags: , , , , , by klotz

Top of the page

First / Previous / Next / Last / Page 1 of 0 SemanticScuttle - klotz.me: Tags: arxiv + llm

About - Propulsed by SemanticScuttle