LLM EvalKit is a streamlined framework that helps developers design, test, and refine prompt‑engineering pipelines for Large Language Models (LLMs). It encompasses prompt management, dataset handling, evaluation, and automated optimization, all wrapped in a Streamlit web UI.
Key capabilities:
| Stage | What it does | Typical workflow |
|-------|-------------|------------------|
| **Prompt Management** | Create, edit, version, and test prompts (name, text, model, system instructions). | Define a prompt, load/edit existing ones, run quick generation tests, and maintain version history. |
| **Dataset Creation** | Organize data for evaluation. Loads CSV, JSON, JSONL files into GCS buckets. | Create dataset folders, upload files, preview items. |
| **Evaluation** | Run model‑based or human‑in‑the‑loop metrics; compare outcomes across prompt versions. | Choose prompt + dataset, generate responses, score with metrics like “question‑answering‑quality”, save baseline results to a leaderboard. |
| **Optimization** | Leveraging Vertex AI’s prompt‑optimization job to automatically search for better prompts. | Configure job (model, dataset, prompt), launch, and monitor training in Vertex AI console. |
| **Results & Records** | Visualize optimization outcomes, compare versions, and maintain a record of performance over time. | View leaderboard, select best optimized prompt, paste new instructions, re‑evaluate, and track progress. |
**Getting Started**
1. Clone the repo, set up a virtual environment, install dependencies, and run `streamlit run index.py`.
2. Configure `src/.env` with `BUCKET_NAME` and `PROJECT_ID`.
3. Use the UI to create/edit prompts, datasets, and launch evaluations/optimizations as described in the tutorial steps.
**Token Use‑Case**
- **Prompt**: “Problem: {{query}}nImage: {{image}} @@@image/jpegnAnswer: {{target}}”
- **Example input JSON**: query, choices, image URL, target answer.
- **Model**: `gemini-2.0-flash-001`.
**License** – Apache 2.0.
Key capabilities:
| Stage | What it does | Typical workflow |
|-------|-------------|------------------|
| **Prompt Management** | Create, edit, version, and test prompts (name, text, model, system instructions). | Define a prompt, load/edit existing ones, run quick generation tests, and maintain version history. |
| **Dataset Creation** | Organize data for evaluation. Loads CSV, JSON, JSONL files into GCS buckets. | Create dataset folders, upload files, preview items. |
| **Evaluation** | Run model‑based or human‑in‑the‑loop metrics; compare outcomes across prompt versions. | Choose prompt + dataset, generate responses, score with metrics like “question‑answering‑quality”, save baseline results to a leaderboard. |
| **Optimization** | Leveraging Vertex AI’s prompt‑optimization job to automatically search for better prompts. | Configure job (model, dataset, prompt), launch, and monitor training in Vertex AI console. |
| **Results & Records** | Visualize optimization outcomes, compare versions, and maintain a record of performance over time. | View leaderboard, select best optimized prompt, paste new instructions, re‑evaluate, and track progress. |
**Getting Started**
1. Clone the repo, set up a virtual environment, install dependencies, and run `streamlit run index.py`.
2. Configure `src/.env` with `BUCKET_NAME` and `PROJECT_ID`.
3. Use the UI to create/edit prompts, datasets, and launch evaluations/optimizations as described in the tutorial steps.
**Token Use‑Case**
- **Prompt**: “Problem: {{query}}nImage: {{image}} @@@image/jpegnAnswer: {{target}}”
- **Example input JSON**: query, choices, image URL, target answer.
- **Model**: `gemini-2.0-flash-001`.
**License** – Apache 2.0.
SRE.ai, a Y Combinator-backed startup, has raised $7.2 million to develop AI agents that automate complex enterprise DevOps workflows, offering chat-like experiences across multiple platforms.
Spotify, a human's digital jukebox, has been a data-driven company since day one, using data for various purposes including payments and experimentation. Managing the vast amount of data required a more streamlined approach, leading to the development of their internal data platform.
**Event Delivery System:**
- **On-Premises Setup:** Initially, Spotify used on-premises solutions like Kafka and HDFS. Event data from clients was captured, timestamped, and routed to a central Hadoop cluster.
- **Google Cloud Transition:** In 2015, Spotify moved to Google Cloud Platform (GCP) for better scalability and reliability. Key components include File Tailer, Event Delivery Service, Reliable Persistent Queue, and ETL jobs using Dataflow and BigQuery.
**Event Delivery System:**
- **On-Premises Setup:** Initially, Spotify used on-premises solutions like Kafka and HDFS. Event data from clients was captured, timestamped, and routed to a central Hadoop cluster.
- **Google Cloud Transition:** In 2015, Spotify moved to Google Cloud Platform (GCP) for better scalability and reliability. Key components include File Tailer, Event Delivery Service, Reliable Persistent Queue, and ETL jobs using Dataflow and BigQuery.
This is a hands-on guide with Python example code that walks through the deployment of an ML-based search API using a simple 3-step approach. The article provides a deployment strategy applicable to most machine learning solutions, and the example code is available on GitHub.
In this article, we explore how to deploy and manage machine learning models using Google Kubernetes Engine (GKE), Google AI Platform, and TensorFlow Serving. We will cover the steps to create a machine learning model and deploy it on a Kubernetes cluster for inference.
Notebooks are not enough for ML at scale
Launched in 2007, Chess.com is a premium platform for online chess and one of the largest of its kind. A Cloud SQL for MySQL shop, it transitioned to Cloud SQL Enterprise Plus edition, improving the user experience, cutting costs, and significantly reducing response times, decreasing p99 latency response from 14ms to 4ms. Read on to learn more.
llm-tool provides a command-line utility for running large language models locally. It includes scripts for pulling models from the internet, starting them, and managing them using various commands such as 'run', 'ps', 'kill', 'rm', and 'pull'. Additionally, it offers a Python script named 'querylocal.py' for querying these models. The repository also come
