Round-Robin with llama-server?

[u/HlddenDreck]

Hi,

I'm running a local server with three AMD MI50. Tensor parallelism is not an option since it's very slow with PCIe 3.0 and those cards are not on the same NUMA node.

In order to balance the load, I wanted to do something like round-robin. Every graphic card is running the same model with the same settings and llama-server has to manage requests so request 1 goes to card 1, request 2 to card 2 and so on.

It's possible to run one llama-server instance on each gpu, however I don't want to do load balancing on the client side with setting different providers with different ports.

Can this be done with llama-server only or maybe with some middleware?

[u/Baul]

llamacpp doesn't directly support this, you'll need some type of middleware.

But services like litellm are configurable in just about any arrangement you can think of for dispatching requests to different instances.

You would use many instances of llama-server, and one instance of litellm to route.

[u/HlddenDreck → u/Baul]

Thanks, that sound very interesting and might be the solution. I will try!

[u/wolf001zra → u/Baul]

Yeah this is what I do

[u/DeathByPain]

I use llama-swap as the front end for my llama server. It lets you specify available models and it hot swaps them depending on the request. It almost seems like it would have a kind of load balancing feature like you want but sadly it doesn't afaik.

I've never used vllm myself, but I think the vllm router would accomplish what you want. Good luck

[u/recro69]

I’d probably place a proxy in front of the three servers and let it manage round-robin distribution. It feels cleaner, than having every client need to know about each GPU endpoint

[u/ali0une]

You don't need round-robin specifically — you need a proxy in front of three llama-server instances that (a) spreads load and (b) keeps each conversation on the same GPU so the KV cache is actually reused. That last part is the one most suggestions in this thread miss: blind round-robin or least-conn will scatter a multi-turn chat across all three cards and every turn pays full prefill cost.

Your question is actually what sent me to shepllama — i'd tested it in my homelab a few months ago and filed it away. Your last comment about requests from the same task needing to stay on the same instance is exactly the gap it had, so today i came back and implemented sticky sessions specifically to answer this thread. My fork: shepllama — small Go binary, no Python runtime, OpenAI-API compatible:

# one server per GPU
llama-server -m model.gguf --device 0 --port 8080 &
llama-server -m model.gguf --device 1 --port 8081 &
llama-server -m model.gguf --device 2 --port 8082 &

# balancer in front
shepllama --port 8114 --backends "http://localhost:8080,http://localhost:8081,http://localhost:8082"

Point your client at http://localhost:8114. What you get:

• Model-aware routing: discovers which model lives on which backend at startup, unified GET /v1/models
• Least-busy + LRU distribution instead of blind round-robin
• Sticky sessions (the part that matters for your context-size concern): each client/session is pinned to one backend for a sliding TTL (default 30 min). Pin key resolution, most specific first:
  1. X-Affinity-Key request header — set it to your conversation/user id if your frontend can add headers; this is the robust option
  2. Authorization header — one API key = one GPU, zero client changes if you already use keys
  3. TCP connection — free keep-alive stickiness for clients that send neither

TTL is configurable (--affinity-ttl, 0 disables and gives you pure load balancing). Pins are validated against the model's backend list, so a pin can never target a GPU that doesn't host the requested model.

Caveats: stickiness guarantees same-GPU routing, not a guaranteed cache hit — under heavy concurrency llama-server can still evict slots. And if your client opens a fresh connection per request with no key/header, use X-Affinity-Key or you lose the pin.

Hope this helps.

[u/TheDailySpank]

GPU Stack Server in a docker container (CPU only). Add provider and its models. Point to that MI50. Repeat as necessary. Add route. Specify all three providers above and the model. Priority is relative. 100, 100, 100 = 33, 33, 33

If a provider stops working, you will need to manually update the route (or setup some sort of automation to check health). These routes are dumb.

[u/LetsGoBrandon4256]

How about the built-in -np?

Not sure about decoding speed when compared to having everything self-contained on one card but you might get some extra prefill speed.

[u/HlddenDreck → u/LetsGoBrandon4256]

Not an option, since I need the full context. -np basically divides your context e.g. context size is 128k, -np 2, then you need to limit request context to 64k, otherwise a request might get rejected because it doesn't fit the context.

[u/Ulterior-Motive_ → u/HlddenDreck]

You could increase the context limit. Even if you're already at the model's max context size, you can set it higher and as long as context size / np <= max supported context then you won't have any issues.

[u/HlddenDreck → u/Ulterior-Motive_]

Interesting, I didn't even think about that. However, with my actual settings the VRAM is almost full.
Pipeline parallelism could be another solution, however on my hardware it doesn't scale good. I have almost no performance gain.

[u/conifer_v11]

llama-server will not round robin across three processes by itself. stand up three llama-server --device 0/1/2 on 8080-8082, then one caddy reverse_proxy with lb_policy least_conn in front.

skip session affinity unless you reuse a slot. independent completions want least_conn, not rr, so a long decode does not pile onto a busy card.

[u/Pyrolistical]

Just run 3 instances of llama-server, slap haproxy in front of it with sticky sessions and concurrency limit if 3

[u/HlddenDreck → u/Pyrolistical]

But does this redirect requests from the same task always to the same llama-server instance? Performance wise it has to, otherwise the gpu has to compute the whole context again.

[u/Pyrolistical → u/HlddenDreck]

that is what sticky sessions are for

[u/harrro → u/Pyrolistical]

Yep. In fact, instead of round-robin, /u/HlddenDreck must use sticky-sessions as Llama has a prefill cache so you want a request (that has the previous request cached) to go to the same llama server it was actually talking to before.

Round robin means server-1 processes prefill (+ saves to cache), but 2nd request arrives at server-2 or server-3 -> no cache so prefill runs from scratch.

[u/alexkey]

Just run 2 llamas 1 on each card and the it nginx in front for RR between them.

[u/gh0stwriter1234]

No idea what you are talking about -sm tensor works fine on MI50.... I get nearly a 2x speed up on dense models. Maybe you are trying to run MOE with tensor split which is not as advantageous.

For dense models you greatly benefit as you'll be compute starved without tensor split

If you have these in a server you should have plenty of bandwidth between those cards even on different numa nodes... tensor split that bandwidth heavy. If you can't reply why the downvote?

I have no problems running -sm tensor with llama.cpp for models like Qwen 3.6 27b... 50-60t/s with MTP enabled with two cards on 16x PCIe 3.0 also note that Q8 runs best... its oddly enough as fast as Q4. if you try anyhing other than q8 or q4 you will loose performance because these cards do not perform those math types natively and or require too much math to do the convertions of the quants.