Tiller Router - (another) self hosted LLM router to fix your model picking and provider failure worries!

[u/dellarb]

So I've been digging around many self hosted LLM routers to sit between my various agents/coding tools and providers and none seemed to have what I was looking for.

1. The ability to point an agent/tool at a single router URL & model and then in the router UI change what provider & model that actually resolved to without changing client settings.

2. There are often good free/discount providers but their performance can be unreliable so you find yourself resubmitting requests a lot. I wanted a seamless fall-back chain where I could have the router detect a provider failure and seamlessly fall-back to other providers without the client seeing an error.

So my project Tiller Router was born. Full disclosure lots of AI generated code here but I've given it a good testing manually and written this post all in my words.

https://github.com/dellarb/tiller-router

This is a self-hosted router that sits between your agents/tools and providers and aggregates up all your providers providing a slick interface to steer your client keys to whichever model or catalogue of model you want without messing with client settings.

Key Feature 1 - One client one model - steer it in Tiller

Just point your favourite client/agent/coding tool at tiller as it's endpoint. Add your providers and keys and select which model you want that client to be served. Regardless of model client requests, your selected model in Tiller will be served transparently to the client. You can also choose to serve a curated catalogue of models to a client to simplify pickers if you still want client control.

Key Feature 2 - Virtual models with Fall-back

Create a virtual model with any name you like and either map it to a real model with one of your providers or better yet add a fall-back chain. The models will be tried in the order you set and if it fails or times out, the request will be sent to the next model and so on fully transparent to the client.

Bonus - the selected model for one-client-one-model can be a virtual model with fallback

Today is the very first beta 1 release of this and I'm pretty fresh to the publishing FOSS game despite many years of coding personally. Feedback is welcome.

I hope this is helpful and look forward to someone showing me something obvious I've missed in a product out there shortly!