An in-depth look at Choreo, an open-source Internal Developer Platform (IDP) built on Kubernetes and GitOps, utilizing 20+ CNCF tools to provide a secure, scalable, and developer-friendly experience. The article discusses the challenges of Kubernetes management, the illusion of 'platformless' solutions, and how Choreo aims to bridge the gap between developer freedom and enterprise requirements.
This Splunk Lantern article outlines the steps to monitor Gen AI applications with Splunk Observability Cloud, covering setup with OpenTelemetry, NVIDIA GPU metrics, Python instrumentation, and OpenLIT integration to monitor GenAI applications built with technologies like Python, LLMs (OpenAI's GPT-4o, Anthropic's Claude 3.5 Haiku, Meta’s Llama), NVIDIA GPUs, Langchain, and vector databases (Pinecone, Chroma) using Splunk Observability Cloud. It outlines a six-step process:
- Access Splunk Observability Cloud: Sign up for a free trial if needed.
- Deploy Splunk Distribution of OpenTelemetry Collector: Use a Helm chart to install the collector in Kubernetes.
- Capture NVIDIA GPU Metrics: Utilize the NVIDIA GPU Operator and Prometheus receiver in the OpenTelemetry Collector.
- Instrument Python Applications: Use the Splunk Distribution of OpenTelemetry Python agent for automatic instrumentation and enable Always On Profiling.
- Enhance with OpenLIT: Install and initialize OpenLIT to capture detailed trace data, including LLM calls and interactions with vector databases (with options to disable PII capture).
- Start Using the Data: Leverage the collected metrics and traces, including features like Tag Spotlight, to identify and resolve performance issues (example given: OpenAI rate limits).
The article emphasizes OpenTelemetry's role in GenAI observability and highlights how Splunk Observability Cloud facilitates monitoring these complex applications, providing insights into performance, cost, and potential bottlenecks. It also points to resources for help and further information on specific aspects of the process.
| Project Name | Description | Key Features | Use Cases | GitHub Stars |
|---|---|---|---|---|
| Cluster API (CAPI) | A project for declaratively provisioning and managing Kubernetes clusters across different environments. | Extensible, open source, API-driven | Multi-cluster, multi-environment orchestration | 3,700 |
| KubeVirt | Brings VM workloads into Kubernetes clusters. | Supports VMs in Kubernetes, used by major enterprises | Cloud-native VM management, exit strategy from proprietary vendors | 5,000 |
| vCluster | Creates "virtual clusters" within a single host cluster for ephemeral dev environments. | Fast setup, low overhead, isolated environments | Ephemeral dev environments, Kubernetes as a Service (KaaS) | 8,000 |
| Kairos | Builds customizable bootable images for edge computing environments. | Secure, immutable images, supports Trusted Boot | Edge computing, secure and immutable environments | 1,200 |
| LocalAI | Provides a local inference API for AI models, compatible with OpenAI API specifications. | Local inference, privacy-focused | Local AI model deployment, privacy-sensitive use cases | 30,000 |
EnterpriseDB's CloudNativePG, a Kubernetes operator for PostgreSQL, has been accepted into the CNCF sandbox, simplifying database management within Kubernetes environments by automating high availability and failover.
This skill path by Bryce Yu guides users through the basics of managing databases on Kubernetes using KubeBlocks. It covers installation, deployment, upgrades, backup, observability, and auto-tuning of database clusters.
OpenTelemetry, a Cloud Native Computing Foundation incubating project, helps software engineers collect and analyze data about system and application performance. Created from the merger of OpenTracing and OpenCensus in 2019, it addresses the challenges of observability in large-scale systems, especially with the rise of Kubernetes. The article discusses its rapid adoption, current challenges, and future innovations like profiling signals.
A comprehensive walkthrough for building a multicluster GitOps platform using popular open source tools in the Kubernetes space, focusing on choosing a cloud provider, selecting a Git provider, establishing a platform domain and DNS provider, defining Infrastructure as Code, selecting a GitOps engine, and defining management pillars.
This article provides a cheatsheet on the Infrastructure as Code (IaC) landscape, highlighting the benefits of scalable infrastructure provisioning in terms of availability, scalability, repeatability, and cost-effectiveness. It discusses strategies such as containerization, container orchestration, and tools like Terraform, Kubernetes, and Ansible. The article also introduces GitOps as a method for automating infrastructure updates through Git workflows and CI/CD.
A Microsoft engineer demonstrates how WebAssembly modules can run alongside containers in Kubernetes environments, offering benefits like reduced size and faster cold start times for certain workloads.
OpenAI is blaming one of the longest outages in its history on a 'new telemetry service' gone awry, which caused major disruptions to ChatGPT, Sora, and its developer-facing API.
Postmortem Incident Investigation Report
Incident Summary
On December 13, 2024, OpenAI experienced a major service outage affecting its AI-powered chatbot platform, ChatGPT, its video generator, Sora, and its developer-facing API. The incident began around 3 p.m. Pacific Time and lasted approximately three hours before all services were fully restored.
Root Cause
The outage was caused by the deployment of a new telemetry service designed to collect Kubernetes metrics. This telemetry service was intended to monitor Kubernetes operations, but an issue with its configuration inadvertently triggered resource-intensive Kubernetes API operations.
Detailed Analysis
- New Telemetry Service: The telemetry service was rolled out to collect Kubernetes metrics. However, its configuration led to unintended and resource-intensive Kubernetes API operations.
- Kubernetes API Overload: The resource-intensive operations overwhelmed the Kubernetes API servers, disrupting the Kubernetes control plane in most large Kubernetes clusters.
- DNS Resolution Impact: The affected Kubernetes control plane impacted DNS resolution, a critical component that converts IP addresses to domain names. This complication delayed visibility into the full scope of the problem and allowed the rollout to continue before the issues were fully understood.
- DNS Caching: The use of DNS caching further delayed visibility and slowed the implementation of a fix, as the system relied on cached information rather than the actual, disrupted state.
Mitigating Factors
- Detection Delay: OpenAI detected the issue "a few minutes" before customers noticed the impact, but was unable to quickly implement a fix due to the overwhelmed Kubernetes servers.
- Testing Shortcomings: The testing procedures did not catch the impact of the changes on the Kubernetes control plane, leading to a slow remediation process.
Preventive Measures
- Improved Monitoring: Implementing better monitoring for infrastructure changes to detect issues early.
- Phased Rollouts: Adopting phased rollouts with enhanced monitoring to ensure smoother deployment and quicker detection of issues.
- Kubernetes API Access: Ensuring that OpenAI engineers have mechanisms to access the Kubernetes API servers under any circumstances to improve the remediation speed.
