Benchmarking long-form factuality in large language models. Original code for our paper "Long-form factuality in large language models".
Key concept: Setting mental models can help users understand how to interact with products that adapt over time. This chapter covers:
Identifying existing mental models
Onboarding in stages
Planning for co-learning
Accounting for user expectations of human-like interaction
Key concept: To build effective mental models of AI-powered products, consider what you want people to know about your product before their first use, how to explain its features, and when it will need feedback from them to improve.
Dan Russell is a Senior Research Scientist focusing on search quality and user happiness at Google. He emphasizes the importance of education and teaching users how to effectively search. His top time-saving tip is waking up early at 4am to complete his work before others start their day. Dan uses a combination of digital and paper tools to manage his tasks, including a physical to-do list synced with his online calendar and reminders. Besides his phone and computer, he highly appreciates his durable Maglite flashlight. He enjoys reading extensively across various fields, taking detailed notes that help him retain information. Dan prefers quiet surroundings while working and rarely listens to music. But he likes listening to podcasts for mindless tasks. He identifies himself as an introvert but adjusts his behavior to fit extroverted roles required by his job. Sleep is crucial to him, though he occasionally takes weekend naps if needed. He tries to avoid technology that may lead to high data fees while traveling internationally. Advice played a significant role in shaping Dan's perspective; he learned to listen to guidance given by others after receiving helpful feedback earlier in his career.
Lastly, he maintains three different sets of notes – worklog, journal, and summary – to keep track of his activities, thoughts, and insights.
- Search engines sometimes provide incorrect results due to misinterpretations of the query by the user, highlighting the need for careful evaluation of search results.
- Modifying queries and pursuing multiple avenues of research concurrently improves the effectiveness of searching.
- Advanced search techniques such as site:, filetype:, and double-quoting phrases enhance accuracy.
- Critical thinking skills are essential when analyzing search results to avoid accepting the first answer seen without questioning its validity.
llm-tool provides a command-line utility for running large language models locally. It includes scripts for pulling models from the internet, starting them, and managing them using various commands such as 'run', 'ps', 'kill', 'rm', and 'pull'. Additionally, it offers a Python script named 'querylocal.py' for querying these models. The repository also come
- create a custom base image for a Cloud Workstation environment using a Dockerfile
. Uses:
Quantized models from
The "LLM" toolkit offers a versatile command-line utility and Python library that allows users to work efficiently with large language models. Users can execute prompts directly from their terminals, store the outcomes in SQLite databases, generate embeddings, and perform various other tasks. In this extensive tutorial, topics covered include setup, usage, OpenAI models, alternative models, embeddings, plugins, model aliases, Python APIs, prompt templates, logging, related tools, CLI references, contributing, and change logs.
Bookmarks on this page are managed by an admin user. 