LLMOps focuses on orchestration, observability, and evaluation.
* **PydanticAI:** type-safe outputs for LLMs, supporting multiple models and complex workflows for more reliable software-like behavior.
* **Bifrost:** gateway for multiple models/providers, offering a single API with features like failover, load balancing, and observability.
* **Traceloop / OpenLLMetry:** Integrates LLM with OpenTelemetry
* **Promptfoo:** CI/CD pipelines for automated checks.
* **Invariant Guardrails:** runtime rules between applications and LLMs/tools, enforcing constraints without code changes.
* **Letta:** version-controlled memory for agents, tracking interactions like a Git repository for debugging and rollback.
* **OpenPipe:** continuous model improvement through logging, data export, evaluation, and fine-tuning within a single platform.
* **Argilla:** human feedback and data curation for tasks like annotation and error analysis, improving model performance.
* **KitOps:** Packages models, datasets, prompts, and configurations into versioned artifacts for clean deployments and reproducibility.
* **Composio:** authentication, permissions, and execution for agents interacting with hundreds of external applications.
* **PydanticAI:** type-safe outputs for LLMs, supporting multiple models and complex workflows for more reliable software-like behavior.
* **Bifrost:** gateway for multiple models/providers, offering a single API with features like failover, load balancing, and observability.
* **Traceloop / OpenLLMetry:** Integrates LLM with OpenTelemetry
* **Promptfoo:** CI/CD pipelines for automated checks.
* **Invariant Guardrails:** runtime rules between applications and LLMs/tools, enforcing constraints without code changes.
* **Letta:** version-controlled memory for agents, tracking interactions like a Git repository for debugging and rollback.
* **OpenPipe:** continuous model improvement through logging, data export, evaluation, and fine-tuning within a single platform.
* **Argilla:** human feedback and data curation for tasks like annotation and error analysis, improving model performance.
* **KitOps:** Packages models, datasets, prompts, and configurations into versioned artifacts for clean deployments and reproducibility.
* **Composio:** authentication, permissions, and execution for agents interacting with hundreds of external applications.
