LLMOps focuses on orchestration, observability, and evaluation.
* **PydanticAI:** type-safe outputs for LLMs, supporting multiple models and complex workflows for more reliable software-like behavior.
* **Bifrost:** gateway for multiple models/providers, offering a single API with features like failover, load balancing, and observability.
* **Traceloop / OpenLLMetry:** Integrates LLM with OpenTelemetry
* **Promptfoo:** CI/CD pipelines for automated checks.
* **Invariant Guardrails:** runtime rules between applications and LLMs/tools, enforcing constraints without code changes.
* **Letta:** version-controlled memory for agents, tracking interactions like a Git repository for debugging and rollback.
* **OpenPipe:** continuous model improvement through logging, data export, evaluation, and fine-tuning within a single platform.
* **Argilla:** human feedback and data curation for tasks like annotation and error analysis, improving model performance.
* **KitOps:** Packages models, datasets, prompts, and configurations into versioned artifacts for clean deployments and reproducibility.
* **Composio:** authentication, permissions, and execution for agents interacting with hundreds of external applications.
* **PydanticAI:** type-safe outputs for LLMs, supporting multiple models and complex workflows for more reliable software-like behavior.
* **Bifrost:** gateway for multiple models/providers, offering a single API with features like failover, load balancing, and observability.
* **Traceloop / OpenLLMetry:** Integrates LLM with OpenTelemetry
* **Promptfoo:** CI/CD pipelines for automated checks.
* **Invariant Guardrails:** runtime rules between applications and LLMs/tools, enforcing constraints without code changes.
* **Letta:** version-controlled memory for agents, tracking interactions like a Git repository for debugging and rollback.
* **OpenPipe:** continuous model improvement through logging, data export, evaluation, and fine-tuning within a single platform.
* **Argilla:** human feedback and data curation for tasks like annotation and error analysis, improving model performance.
* **KitOps:** Packages models, datasets, prompts, and configurations into versioned artifacts for clean deployments and reproducibility.
* **Composio:** authentication, permissions, and execution for agents interacting with hundreds of external applications.
Walkthrough on building a Q and A pipeline using various tools, and distributing it with ModelKits for collaboration.
Kit is a free, open-source MLOps tool that simplifies AI project management by packaging models, datasets, code, and configurations into a standardized, versioned, and tamper-proof ModelKit. It enables collaboration, model traceability, and reproducibility, making it easier to hand off AI projects between data scientists, developers, and DevOps teams.
Explores KitOps, an open source project that bridges the gap between DevOps and machine learning pipelines by allowing you to leverage existing DevOps pipelines for MLOps tasks.
ModelKits are standardized packages that contain all the necessary components of an ML project, including the model, datasets, code, and configuration files.
ModelKits are defined using a YAML file called a Kitfile, which can be integrated seamlessly with existing DevOps pipelines, much like a Dockerfile for containerization.
ModelKits are standardized packages that contain all the necessary components of an ML project, including the model, datasets, code, and configuration files.
ModelKits are defined using a YAML file called a Kitfile, which can be integrated seamlessly with existing DevOps pipelines, much like a Dockerfile for containerization.
