This article discusses using entropy and variance of entropy (VarEntropy) to detect hallucinations in LLM function calling, focusing on how structured outputs allow for identifying errors through statistical anomalies in token confidence.
The author discusses the development of a function calling large language model (LLM) that significantly improves latency for agentic applications. This LLM matches or even exceeds the performance of other frontier LLMs. It is integrated into an open-source intelligent gateway for agentic applications, allowing developers to focus on more differentiated aspects of their projects. The model and the gateway are available on Hugging Face and GitHub, respectively.
Arch is an intelligent gateway for agents, designed to securely handle prompts, integrate with APIs, and provide rich observability, built on Envoy Proxy.
The ArchGW project focuses on simplifying the development of **agentic applications** – applications powered by Large Language Models (LLMs) that can perform actions and interact with tools. Here's a breakdown of the use cases and examples highlighted:
**Core Use Cases:**
* **Routing:** Intelligent routing of prompts to the correct agents or tools.
* **Tools Use:** Simplifying the integration of prompts with tools/APIs for common tasks.
* **Guardrails:** Centralized configuration for safety and preventing harmful outcomes.
* **LLM Access:** Centralized access and management of LLMs with retries for reliability.
* **Observability:** Providing W3C-compatible tracing and metrics for monitoring LLM interactions.
**Specific Examples & Demos:**
* **Weather Forecast Agent:** A sample application demonstrating core function calling capabilities.
* **Network Operator Agent:** An agent that can interact with network devices (retrieve statistics, reboot).
* **Connecting to SaaS APIs:** Demonstrates integrating 3rd party SaaS APIs into agentic chat experiences.
* **LLM Router:** Using Arch as a gateway to route requests to different LLMs (GPT-4o, Mistral) based on configuration or headers. The example shows how to switch between LLMs using the `x-arch-llm-provider-hint` header.
* **Currency Exchange Agent:** A quickstart guide builds an agent that fetches currency exchange rates from an API (Frankfurter.app). This demonstrates setting up configuration files, starting the gateway, and interacting with the agent via curl.
**Overall, ArchGW aims to address common challenges in building agentic apps:**
* Managing complex routing logic.
* Integrating with various LLMs and tools.
* Ensuring safety and reliability.
* Providing observability into LLM interactions.
The ArchGW project focuses on simplifying the development of **agentic applications** – applications powered by Large Language Models (LLMs) that can perform actions and interact with tools. Here's a breakdown of the use cases and examples highlighted:
**Core Use Cases:**
* **Routing:** Intelligent routing of prompts to the correct agents or tools.
* **Tools Use:** Simplifying the integration of prompts with tools/APIs for common tasks.
* **Guardrails:** Centralized configuration for safety and preventing harmful outcomes.
* **LLM Access:** Centralized access and management of LLMs with retries for reliability.
* **Observability:** Providing W3C-compatible tracing and metrics for monitoring LLM interactions.
**Specific Examples & Demos:**
* **Weather Forecast Agent:** A sample application demonstrating core function calling capabilities.
* **Network Operator Agent:** An agent that can interact with network devices (retrieve statistics, reboot).
* **Connecting to SaaS APIs:** Demonstrates integrating 3rd party SaaS APIs into agentic chat experiences.
* **LLM Router:** Using Arch as a gateway to route requests to different LLMs (GPT-4o, Mistral) based on configuration or headers. The example shows how to switch between LLMs using the `x-arch-llm-provider-hint` header.
* **Currency Exchange Agent:** A quickstart guide builds an agent that fetches currency exchange rates from an API (Frankfurter.app). This demonstrates setting up configuration files, starting the gateway, and interacting with the agent via curl.
**Overall, ArchGW aims to address common challenges in building agentic apps:**
* Managing complex routing logic.
* Integrating with various LLMs and tools.
* Ensuring safety and reliability.
* Providing observability into LLM interactions.
