A curated guide to code sandboxing solutions, covering technologies like MicroVMs, application kernels, language runtimes, and containerization. It provides a feature matrix, in-depth platform profiles (e2b, Daytona, microsandbox, WebContainers, Replit, Cloudflare Workers, Fly.io, Kata Containers), and a decision framework for choosing the right sandboxing solution based on security, performance, workload type, and hosting preferences.
Running GenAI models is easy. Scaling them to thousands of users, not so much. This guide details avenues for scaling AI workloads from proofs of concept to production-ready deployments, covering API integration, on-prem deployment considerations, hardware requirements, and tools like vLLM and Nvidia NIMs.
K8S-native cluster-wide deployment for vLLM. Provides a reference implementation for building an inference stack on top of vLLM, enabling scaling, monitoring, request routing, and KV cache offloading with easy cloud deployment.
vLLM Production Stack provides a reference implementation on how to build an inference stack on top of vLLM, allowing for scalable, monitored, and performant LLM deployments using Kubernetes and Helm.
A user is seeking advice on deploying a new server with 4x H100 GPUs (320GB VRAM) for on-premise AI workloads. They are considering a Kubernetes-based deployment with RKE2, Nvidia GPU Operator, and tools like vLLM, llama.cpp, and Litellm. They are also exploring the option of GPU pass-through with a hypervisor. The post details their current infrastructure and asks for potential gotchas or best practices.
Flagger is a progressive delivery Kubernetes operator that automates the release process for applications running on Kubernetes. It reduces the risk of introducing a new software version in production by gradually shifting traffic to the new version while measuring metrics and running conformance tests.
This is a hands-on guide with Python example code that walks through the deployment of an ML-based search API using a simple 3-step approach. The article provides a deployment strategy applicable to most machine learning solutions, and the example code is available on GitHub.
In this article, we explore how to deploy and manage machine learning models using Google Kubernetes Engine (GKE), Google AI Platform, and TensorFlow Serving. We will cover the steps to create a machine learning model and deploy it on a Kubernetes cluster for inference.
apiVersion: v1
kind: Namespace
metadata:
name: "mailserver"
---
apiVersion: "v1"
kind: "PersistentVolumeClaim"
metadata:
name: "mailserver-pvc"
namespace: "mailserver"
labels:
app: mailserver
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 15Gi
---
apiVersion: "v1"
kind: "PersistentVolumeClaim"
metadata:
name: "mailserver-data-pvc"
namespace: "mailserver"
labels:
app: mailserver
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 15Gi
---
kind: Service
apiVersion: v1
metadata:
name: mailserver
namespace: mailserver
labels:
app: mailserver
annotations:
metallb.universe.tf/address-pool: bgp-range
spec:
selector:
app: mailserver
ports:
- name: smtp
port: 25
targetPort: smtp
- name: smtp-auth
port: 587
targetPort: smtp-auth
- name: imap-secure
port: 993
targetPort: imap-secure
type: LoadBalancer
externalTrafficPolicy: Local
loadBalancerIP: 192.168.4.2
---
kind: ConfigMap
apiVersion: v1
metadata:
name: mailserver.config
namespace: mailserver
labels:
app: mailserver
data:
postfix-receive-access.cf: |
blacklisted@recipient.example REJECT
postfix-send-access.cf: |
blacklisted-sender REJECT
postfix-accounts.cf: |
user1@example.com|{SHA512-CRYPT}$6$2YpW1nYtPBs2yLYS$z.5PGH1OEzsHHNhl3gJrc3D.YMZkvKw/vp.r5WIiwya6z7P/CQ9GDEJDr2G2V0cAfjDFeAQPUoopsuWPXLk3u1
postfix-virtual.cf: |
@example.org user1@example.com
SigningTable: |
*@example.org my._domainkey.example.org
*@example.com my._domainkey.example.com
KeyTable: |
my._domainkey.example.org example.org:my:/etc/opendkim/keys/example.org-mail.key
my._domainkey.example.com example.com:my:/etc/opendkim/keys/example.com-mail.key
TrustedHosts: |
127.0.0.1
localhost
spamassassin-rules.cf: |
header SPAMMER_1 From =~ / a-z » +d{2,4}.com/i
score SPAMMER_1 5.0
sa-cron: |
# This assumes you're having `environment: ONE_DIR=1` in the docker-compose.yml,
# with a consolidated config in `/var/mail-state`
#
# m h dom mon dow user command
#
# Everyday 2:00AM, learn spam from a specific user
# spam: junk directory
0 2 * * * root sa-learn --spam /var/mail/*/*/.Junk --dbpath /var/mail-state/lib-amavis/.spamassassin
# ham: archive directories
15 2 * * * root sa-learn --ham /var/mail/*/*/.Archive* --dbpath /var/mail-state/lib-amavis/.spamassassin
# ham: inbox subdirectories
30 2 * * * root sa-learn --ham /var/mail/*/*/cur* --dbpath /var/mail-state/lib-amavis/.spamassassin
---
kind: Secret
apiVersion: v1
metadata:
name: mailserver.opendkim.keys
namespace: mailserver
labels:
app: mailserver
type: Opaque
data:
example.org-mail.key: "key here"
example.com-mail.key: "key here"
---
kind: Deployment
apiVersion: apps/v1
metadata:
name: mailserver
namespace: mailserver
labels:
app: mailserver
spec:
selector:
matchLabels:
app: mailserver
strategy:
type: Recreate
template:
metadata:
labels:
app: mailserver
spec:
nodeSelector:
has/mail-server: "true"
subdomain: mailserver
containers:
- name: mailserver
image: radicand/docker-mailserver:gha
# image: radicand/docker-mailserver:b9f43ab
resources:
requests:
memory: "500Mi"
cpu: "750m"
limits:
memory: "1500Mi"
cpu: "900m"
imagePullPolicy: Always
securityContext: {}
ports:
- name: smtp
containerPort: 25
- name: smtp-auth
containerPort: 587
- name: imap-secure
containerPort: 993
env:
- name: DOMAINNAME
value: "example.com"
- name: ONE_DIR
value: "1"
- name: SMTP_ONLY
value: "0"
- name: ENABLE_SRS
value: "0"
- name: ENABLE_SPAMASSASSIN
value: "1"
- name: SA_SPAM_SUBJECT
value: " SPAM » "
- name: ENABLE_FAIL2BAN
value: "0"
- name: ENABLE_POSTGREY
value: "0"
- name: ENABLE_CLAMAV
value: "0"
- name: ENABLE_SASLAUTHD
value: "0"
- name: SRS_EXCLUDE_DOMAINS
value: example.com
- name: POSTMASTER_ADDRESS
value: "postmaster@example.com"
- name: POSTFIX_MESSAGE_SIZE_LIMIT
value: "25000000"
- name: SSL_TYPE
value: "manual"
- name: SSL_CERT_PATH
value: "/etc/ssl/mailserver/tls.crt"
- name: SSL_KEY_PATH
value: "/etc/ssl/mailserver/tls.key"
- name: OVERRIDE_HOSTNAME
value: "mail.example.com"
- name: REPORT_RECIPIENT
value: "postmaster@example.com"
volumeMounts:
- name: config
subPath: postfix-accounts.cf
mountPath: /tmp/docker-mailserver/postfix-accounts.cf
readOnly: true
- name: config
subPath: postfix-main.cf
mountPath: /tmp/docker-mailserver/postfix-main.cf
readOnly: true
- name: config
subPath: postfix-receive-access.cf
mountPath: /tmp/docker-mailserver/postfix-receive-access.cf
readOnly: true
- name: config
subPath: postfix-virtual.cf
mountPath: /tmp/docker-mailserver/postfix-virtual.cf
readOnly: true
- name: config
subPath: postfix-send-access.cf
mountPath: /tmp/docker-mailserver/postfix-send-access.cf
readOnly: true
- name: config
subPath: spamassassin-rules.cf
mountPath: /tmp/docker-mailserver/spamassassin-rules.cf
readOnly: true
- name: config
subPath: sa-cron
mountPath: /etc/cron.d/sa-learn
readOnly: true
# mode: 0644
- name: config
subPath: SigningTable
mountPath: /tmp/docker-mailserver/opendkim/SigningTable
readOnly: true
- name: config
subPath: KeyTable
mountPath: /tmp/docker-mailserver/opendkim/KeyTable
readOnly: true
- name: config
subPath: TrustedHosts
mountPath: /tmp/docker-mailserver/opendkim/TrustedHosts
readOnly: true
- name: opendkim-keys
mountPath: /tmp/docker-mailserver/opendkim/keys
readOnly: true
- name: state
mountPath: /var/mail-state
- name: data
mountPath: /var/mail
- name: tls
mountPath: /etc/ssl/mailserver
readOnly: true
volumes:
- name: config
configMap:
name: mailserver.config
# items:
# - key: sa-cron
# path: /etc/cron.d
# mode: 0644
- name: opendkim-keys
secret:
secretName: mailserver.opendkim.keys
- name: state
persistentVolumeClaim:
claimName: "mailserver-pvc"
- name: data
persistentVolumeClaim:
claimName: "mailserver-data-pvc"
- name: tls
secret:
secretName: mail-example-com-tls
kind: Namespace
metadata:
name: "mailserver"
---
apiVersion: "v1"
kind: "PersistentVolumeClaim"
metadata:
name: "mailserver-pvc"
namespace: "mailserver"
labels:
app: mailserver
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 15Gi
---
apiVersion: "v1"
kind: "PersistentVolumeClaim"
metadata:
name: "mailserver-data-pvc"
namespace: "mailserver"
labels:
app: mailserver
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 15Gi
---
kind: Service
apiVersion: v1
metadata:
name: mailserver
namespace: mailserver
labels:
app: mailserver
annotations:
metallb.universe.tf/address-pool: bgp-range
spec:
selector:
app: mailserver
ports:
- name: smtp
port: 25
targetPort: smtp
- name: smtp-auth
port: 587
targetPort: smtp-auth
- name: imap-secure
port: 993
targetPort: imap-secure
type: LoadBalancer
externalTrafficPolicy: Local
loadBalancerIP: 192.168.4.2
---
kind: ConfigMap
apiVersion: v1
metadata:
name: mailserver.config
namespace: mailserver
labels:
app: mailserver
data:
postfix-receive-access.cf: |
blacklisted@recipient.example REJECT
postfix-send-access.cf: |
blacklisted-sender REJECT
postfix-accounts.cf: |
user1@example.com|{SHA512-CRYPT}$6$2YpW1nYtPBs2yLYS$z.5PGH1OEzsHHNhl3gJrc3D.YMZkvKw/vp.r5WIiwya6z7P/CQ9GDEJDr2G2V0cAfjDFeAQPUoopsuWPXLk3u1
postfix-virtual.cf: |
@example.org user1@example.com
SigningTable: |
*@example.org my._domainkey.example.org
*@example.com my._domainkey.example.com
KeyTable: |
my._domainkey.example.org example.org:my:/etc/opendkim/keys/example.org-mail.key
my._domainkey.example.com example.com:my:/etc/opendkim/keys/example.com-mail.key
TrustedHosts: |
127.0.0.1
localhost
spamassassin-rules.cf: |
header SPAMMER_1 From =~ / a-z » +d{2,4}.com/i
score SPAMMER_1 5.0
sa-cron: |
# This assumes you're having `environment: ONE_DIR=1` in the docker-compose.yml,
# with a consolidated config in `/var/mail-state`
#
# m h dom mon dow user command
#
# Everyday 2:00AM, learn spam from a specific user
# spam: junk directory
0 2 * * * root sa-learn --spam /var/mail/*/*/.Junk --dbpath /var/mail-state/lib-amavis/.spamassassin
# ham: archive directories
15 2 * * * root sa-learn --ham /var/mail/*/*/.Archive* --dbpath /var/mail-state/lib-amavis/.spamassassin
# ham: inbox subdirectories
30 2 * * * root sa-learn --ham /var/mail/*/*/cur* --dbpath /var/mail-state/lib-amavis/.spamassassin
---
kind: Secret
apiVersion: v1
metadata:
name: mailserver.opendkim.keys
namespace: mailserver
labels:
app: mailserver
type: Opaque
data:
example.org-mail.key: "key here"
example.com-mail.key: "key here"
---
kind: Deployment
apiVersion: apps/v1
metadata:
name: mailserver
namespace: mailserver
labels:
app: mailserver
spec:
selector:
matchLabels:
app: mailserver
strategy:
type: Recreate
template:
metadata:
labels:
app: mailserver
spec:
nodeSelector:
has/mail-server: "true"
subdomain: mailserver
containers:
- name: mailserver
image: radicand/docker-mailserver:gha
# image: radicand/docker-mailserver:b9f43ab
resources:
requests:
memory: "500Mi"
cpu: "750m"
limits:
memory: "1500Mi"
cpu: "900m"
imagePullPolicy: Always
securityContext: {}
ports:
- name: smtp
containerPort: 25
- name: smtp-auth
containerPort: 587
- name: imap-secure
containerPort: 993
env:
- name: DOMAINNAME
value: "example.com"
- name: ONE_DIR
value: "1"
- name: SMTP_ONLY
value: "0"
- name: ENABLE_SRS
value: "0"
- name: ENABLE_SPAMASSASSIN
value: "1"
- name: SA_SPAM_SUBJECT
value: " SPAM » "
- name: ENABLE_FAIL2BAN
value: "0"
- name: ENABLE_POSTGREY
value: "0"
- name: ENABLE_CLAMAV
value: "0"
- name: ENABLE_SASLAUTHD
value: "0"
- name: SRS_EXCLUDE_DOMAINS
value: example.com
- name: POSTMASTER_ADDRESS
value: "postmaster@example.com"
- name: POSTFIX_MESSAGE_SIZE_LIMIT
value: "25000000"
- name: SSL_TYPE
value: "manual"
- name: SSL_CERT_PATH
value: "/etc/ssl/mailserver/tls.crt"
- name: SSL_KEY_PATH
value: "/etc/ssl/mailserver/tls.key"
- name: OVERRIDE_HOSTNAME
value: "mail.example.com"
- name: REPORT_RECIPIENT
value: "postmaster@example.com"
volumeMounts:
- name: config
subPath: postfix-accounts.cf
mountPath: /tmp/docker-mailserver/postfix-accounts.cf
readOnly: true
- name: config
subPath: postfix-main.cf
mountPath: /tmp/docker-mailserver/postfix-main.cf
readOnly: true
- name: config
subPath: postfix-receive-access.cf
mountPath: /tmp/docker-mailserver/postfix-receive-access.cf
readOnly: true
- name: config
subPath: postfix-virtual.cf
mountPath: /tmp/docker-mailserver/postfix-virtual.cf
readOnly: true
- name: config
subPath: postfix-send-access.cf
mountPath: /tmp/docker-mailserver/postfix-send-access.cf
readOnly: true
- name: config
subPath: spamassassin-rules.cf
mountPath: /tmp/docker-mailserver/spamassassin-rules.cf
readOnly: true
- name: config
subPath: sa-cron
mountPath: /etc/cron.d/sa-learn
readOnly: true
# mode: 0644
- name: config
subPath: SigningTable
mountPath: /tmp/docker-mailserver/opendkim/SigningTable
readOnly: true
- name: config
subPath: KeyTable
mountPath: /tmp/docker-mailserver/opendkim/KeyTable
readOnly: true
- name: config
subPath: TrustedHosts
mountPath: /tmp/docker-mailserver/opendkim/TrustedHosts
readOnly: true
- name: opendkim-keys
mountPath: /tmp/docker-mailserver/opendkim/keys
readOnly: true
- name: state
mountPath: /var/mail-state
- name: data
mountPath: /var/mail
- name: tls
mountPath: /etc/ssl/mailserver
readOnly: true
volumes:
- name: config
configMap:
name: mailserver.config
# items:
# - key: sa-cron
# path: /etc/cron.d
# mode: 0644
- name: opendkim-keys
secret:
secretName: mailserver.opendkim.keys
- name: state
persistentVolumeClaim:
claimName: "mailserver-pvc"
- name: data
persistentVolumeClaim:
claimName: "mailserver-data-pvc"
- name: tls
secret:
secretName: mail-example-com-tls
