Open sourced our k8s-native AI platform for distributed multi-model inference at scale

[u/EmbeddedMagicX]

Hey everyone,

I’m one of the co-founders of axem. We recently open sourced shaide, the infrastructure we’ve been building for running multiple LLMs across GPU servers.

It’s probably not very interesting if your setup is one model on one GPU. We started building it when our setups grew beyond that and we needed to run several models at the same time, scale them independently across GPU nodes, route traffic between replicas, and keep everything running inside infrastructure we controlled.

Instead of configuring all of those pieces separately for every cluster, we ended up packaging them into one Kubernetes-native platform.

Current setup:

• vLLM for inference
• llm-d for multi-instance orchestration
• multiple models running and scaling independently
• KV-cache-aware scheduling
• internal OCI registry for container images + model weights
• OpenAI-compatible API
• the entire platform is managed as infrastructure as code
• interactive installer that runs from Docker against an existing Kubernetes cluster
• can operate fully air-gapped with no cluster egress

We mainly use it with on-prem RKE2 clusters, but it also works with EKS/GKE/AKS.

It’s Apache 2.0 and still early, so we’re at the point where feedback from people actually running multi-GPU servers is especially useful.

GitHub:
https://github.com/axem-solutions/shaide

I'm curious how people here handle this once a setup grows beyond a single machine.

If you're running several models in parallel, what are you using for routing/orchestration?