Kagent is an open-source agentic AI framework for Kubernetes that aims to provide autonomous problem solving and remediation for cloud-native infrastructure, moving beyond traditional automation to a more intelligent and self-healing system.
Running GenAI models is easy. Scaling them to thousands of users, not so much. This guide details avenues for scaling AI workloads from proofs of concept to production-ready deployments, covering API integration, on-prem deployment considerations, hardware requirements, and tools like vLLM and Nvidia NIMs.
K8S-native cluster-wide deployment for vLLM. Provides a reference implementation for building an inference stack on top of vLLM, enabling scaling, monitoring, request routing, and KV cache offloading with easy cloud deployment.
vLLM Production Stack provides a reference implementation on how to build an inference stack on top of vLLM, allowing for scalable, monitored, and performant LLM deployments using Kubernetes and Helm.
A user is seeking advice on deploying a new server with 4x H100 GPUs (320GB VRAM) for on-premise AI workloads. They are considering a Kubernetes-based deployment with RKE2, Nvidia GPU Operator, and tools like vLLM, llama.cpp, and Litellm. They are also exploring the option of GPU pass-through with a hypervisor. The post details their current infrastructure and asks for potential gotchas or best practices.
Solo.io donated Kagent, its open source framework for AI agents in Kubernetes, to the CNCF, and introduced MCP Gateway. They also unveiled automated zero-downtime migration and cost-analysis tools for Ambient Mesh.
An in-depth look at Choreo, an open-source Internal Developer Platform (IDP) built on Kubernetes and GitOps, utilizing 20+ CNCF tools to provide a secure, scalable, and developer-friendly experience. The article discusses the challenges of Kubernetes management, the illusion of 'platformless' solutions, and how Choreo aims to bridge the gap between developer freedom and enterprise requirements.
This Splunk Lantern article outlines the steps to monitor Gen AI applications with Splunk Observability Cloud, covering setup with OpenTelemetry, NVIDIA GPU metrics, Python instrumentation, and OpenLIT integration to monitor GenAI applications built with technologies like Python, LLMs (OpenAI's GPT-4o, Anthropic's Claude 3.5 Haiku, Meta’s Llama), NVIDIA GPUs, Langchain, and vector databases (Pinecone, Chroma) using Splunk Observability Cloud. It outlines a six-step process:
1. **Access Splunk Observability Cloud:** Sign up for a free trial if needed.
2. **Deploy Splunk Distribution of OpenTelemetry Collector:** Use a Helm chart to install the collector in Kubernetes.
3. **Capture NVIDIA GPU Metrics:** Utilize the NVIDIA GPU Operator and Prometheus receiver in the OpenTelemetry Collector.
4. **Instrument Python Applications:** Use the Splunk Distribution of OpenTelemetry Python agent for automatic instrumentation and enable Always On Profiling.
5. **Enhance with OpenLIT:** Install and initialize OpenLIT to capture detailed trace data, including LLM calls and interactions with vector databases (with options to disable PII capture).
6. **Start Using the Data:** Leverage the collected metrics and traces, including features like Tag Spotlight, to identify and resolve performance issues (example given: OpenAI rate limits).
The article emphasizes OpenTelemetry's role in GenAI observability and highlights how Splunk Observability Cloud facilitates monitoring these complex applications, providing insights into performance, cost, and potential bottlenecks. It also points to resources for help and further information on specific aspects of the process.
1. **Access Splunk Observability Cloud:** Sign up for a free trial if needed.
2. **Deploy Splunk Distribution of OpenTelemetry Collector:** Use a Helm chart to install the collector in Kubernetes.
3. **Capture NVIDIA GPU Metrics:** Utilize the NVIDIA GPU Operator and Prometheus receiver in the OpenTelemetry Collector.
4. **Instrument Python Applications:** Use the Splunk Distribution of OpenTelemetry Python agent for automatic instrumentation and enable Always On Profiling.
5. **Enhance with OpenLIT:** Install and initialize OpenLIT to capture detailed trace data, including LLM calls and interactions with vector databases (with options to disable PII capture).
6. **Start Using the Data:** Leverage the collected metrics and traces, including features like Tag Spotlight, to identify and resolve performance issues (example given: OpenAI rate limits).
The article emphasizes OpenTelemetry's role in GenAI observability and highlights how Splunk Observability Cloud facilitates monitoring these complex applications, providing insights into performance, cost, and potential bottlenecks. It also points to resources for help and further information on specific aspects of the process.
| Project Name | Description | Key Features | Use Cases | GitHub Stars |
|--------------|-------------|--------------|-----------|--------------|
| Cluster API (CAPI) | A project for declaratively provisioning and managing Kubernetes clusters across different environments. | Extensible, open source, API-driven | Multi-cluster, multi-environment orchestration | 3,700 |
| KubeVirt | Brings VM workloads into Kubernetes clusters. | Supports VMs in Kubernetes, used by major enterprises | Cloud-native VM management, exit strategy from proprietary vendors | 5,000 |
| vCluster | Creates "virtual clusters" within a single host cluster for ephemeral dev environments. | Fast setup, low overhead, isolated environments | Ephemeral dev environments, Kubernetes as a Service (KaaS) | 8,000 |
| Kairos | Builds customizable bootable images for edge computing environments. | Secure, immutable images, supports Trusted Boot | Edge computing, secure and immutable environments | 1,200 |
| LocalAI | Provides a local inference API for AI models, compatible with OpenAI API specifications. | Local inference, privacy-focused | Local AI model deployment, privacy-sensitive use cases | 30,000 |
|--------------|-------------|--------------|-----------|--------------|
| Cluster API (CAPI) | A project for declaratively provisioning and managing Kubernetes clusters across different environments. | Extensible, open source, API-driven | Multi-cluster, multi-environment orchestration | 3,700 |
| KubeVirt | Brings VM workloads into Kubernetes clusters. | Supports VMs in Kubernetes, used by major enterprises | Cloud-native VM management, exit strategy from proprietary vendors | 5,000 |
| vCluster | Creates "virtual clusters" within a single host cluster for ephemeral dev environments. | Fast setup, low overhead, isolated environments | Ephemeral dev environments, Kubernetes as a Service (KaaS) | 8,000 |
| Kairos | Builds customizable bootable images for edge computing environments. | Secure, immutable images, supports Trusted Boot | Edge computing, secure and immutable environments | 1,200 |
| LocalAI | Provides a local inference API for AI models, compatible with OpenAI API specifications. | Local inference, privacy-focused | Local AI model deployment, privacy-sensitive use cases | 30,000 |
EnterpriseDB's CloudNativePG, a Kubernetes operator for PostgreSQL, has been accepted into the CNCF sandbox, simplifying database management within Kubernetes environments by automating high availability and failover.
