This article explains how to use the Sentence Transformers library to finetune and train embedding models for a variety of applications, such as retrieval augmented generation, semantic search, and semantic textual similarity. It covers the training components, dataset format, loss function, training arguments, evaluators, and trainer.
A surprising experiment to show that the devil is in the details
Researchers from NYU Tandon School of Engineering investigated whether modern natural language processing systems could solve the daily Connections puzzles from The New York Times. The results showed that while all the AI systems could solve some of the puzzles, they struggled overall.
This article provides a beginner-friendly introduction to Large Language Models (LLMs) and explains the key concepts in a clear and organized way.
ColBERT is a new way of scoring passage relevance using a BERT language model that substantially solves the problems with dense passage retrieval.
- Embeddings transform words and sentences into sequences of numbers for computers to understand language.
- This technology powers tools like Siri, Alexa, Google Translate, and generative AI systems like ChatGPT, Bard, and DALL-E.
- In the early days, embeddings were crafted by hand, which was time-consuming and couldn't adapt to language nuances easily.
- The 3D hand-crafted embedding app provides an interactive experience to understand this concept.
- The star visualization method offers an intuitive way to understand word embeddings.
- Machine learning models like Word2Vec and GloVe revolutionized the generation of word embeddings from large text datasets.
- Universal Sentence Encoder (USE) extends the concept of word embeddings to entire sentences.
- TensorFlow Projector is an advanced tool to interactively explore high-dimensional data like word and sentence embeddings.
- This technology powers tools like Siri, Alexa, Google Translate, and generative AI systems like ChatGPT, Bard, and DALL-E.
- In the early days, embeddings were crafted by hand, which was time-consuming and couldn't adapt to language nuances easily.
- The 3D hand-crafted embedding app provides an interactive experience to understand this concept.
- The star visualization method offers an intuitive way to understand word embeddings.
- Machine learning models like Word2Vec and GloVe revolutionized the generation of word embeddings from large text datasets.
- Universal Sentence Encoder (USE) extends the concept of word embeddings to entire sentences.
- TensorFlow Projector is an advanced tool to interactively explore high-dimensional data like word and sentence embeddings.
With deep learning, the ROI for having clean and high quality data is immense, and this is realized in every phase of training. For context, the era right before BERT in the text classification world was one where you wanted an abundance of data, even at the expense of quality. It was more important to have representation via examples than for the examples to be perfect. This is because many Al systems did not use pre-trained embeddings (or they weren't any good, anyway) that could be leveraged by a model to apply practical generalizability. In 2018, BERT was a breakthrough for down-stream text tasks,
