The article discusses 5 self-hosted services (NetBox, Harvester, Terraform, Ansible, and Kubernetes) that the author finds useful in their home lab despite being potentially overkill for typical setups. It details the benefits and use cases for each service, highlighting how they enhance the author's tinkering and learning experience.
Sipeed NanoCluster is a palm-sized cluster board with seven slots for Raspberry Pi CM4/CM5, Sipeed LM3H, and/or Sipeed M4N system-on-modules. It features a RISC-V Gigabit switch, independent UART and power control for each module, and supports up to 60W USB-C PD or PoE.
The Azure MCP Server implements the MCP specification to create a seamless connection between AI agents and Azure services. It allows agents to interact with various Azure services like AI Search, App Configuration, Cosmos DB, and more.
The article discusses the increasing complexity of Kubernetes and suggests that Silicon Valley is exploring alternative technologies for container orchestration, citing a benchmark showing a stripped-down stack outperforming Kubernetes.
The article discusses the potential shift away from YAML in Kubernetes 2.0, citing a leaked dashboard photo and the high percentage of production outages linked to YAML misconfigurations. It suggests a new command-line interface is being used for deployments.
Kagent is an open-source agentic AI framework for Kubernetes that aims to provide autonomous problem solving and remediation for cloud-native infrastructure, moving beyond traditional automation to a more intelligent and self-healing system.
Running GenAI models is easy. Scaling them to thousands of users, not so much. This guide details avenues for scaling AI workloads from proofs of concept to production-ready deployments, covering API integration, on-prem deployment considerations, hardware requirements, and tools like vLLM and Nvidia NIMs.
K8S-native cluster-wide deployment for vLLM. Provides a reference implementation for building an inference stack on top of vLLM, enabling scaling, monitoring, request routing, and KV cache offloading with easy cloud deployment.
vLLM Production Stack provides a reference implementation on how to build an inference stack on top of vLLM, allowing for scalable, monitored, and performant LLM deployments using Kubernetes and Helm.
A user is seeking advice on deploying a new server with 4x H100 GPUs (320GB VRAM) for on-premise AI workloads. They are considering a Kubernetes-based deployment with RKE2, Nvidia GPU Operator, and tools like vLLM, llama.cpp, and Litellm. They are also exploring the option of GPU pass-through with a hypervisor. The post details their current infrastructure and asks for potential gotchas or best practices.
