Fixing my model router took 90 seconds. Paying for it would take a hundred years
Choosing the right AI model for each task can be automated, but not out of the box: my router was confidently wrong until I measured it and retrained it on my own tasks. It now gets 85% right, and it's switched off, because the hardware it needs would cost more than it saves.
Tokens cost money. Still, the reflex to run everything on the top model is hard to resist: when a client pays, the client expects the best. Except that's often wrong, and not only because of the bill. A typo doesn't need the strongest model; it needs the right one.
So in Kelter, the agent runtime I'm building (open source on GitHub), every coding task first gets one of four answers: no coding agent at all, the cheap model, the mid-priced one or the expensive one. The expensive one always waits for my click. A task counts as done only when a real test passes, and if it fails, it goes up a level.
I hooked that decision up to a router from a new family of models that pick from a list you give them, with a probability next to each option, instead of writing an answer. Its first answer was about a typo in a README. It sent the fix to the mid-priced model, 0.9 confident.
Lol, wtf.
I couldn't decide between depression and impostor syndrome. It should have worked, so I'd probably screwed something up, but I had no idea what.
People call me the PoC guy: before I put more into something, I check whether it's worth it. So I checked. Forty-eight tasks where I knew the right answer, twelve for each path. The router got 17 right: 35 per cent, against 25 for guessing blind among the four. And it was almost never unsure. Wrong answers came back at 0.96, even 1.0. The obvious setup is a confidence threshold: trust anything above 0.8. It would have waved all of those through.
I hadn't screwed anything up. The router just didn't know my tasks. I retrained it on 120 other examples; the big model underneath stayed as it was and only a small layer on top learned, in a minute and a half on an ordinary processor. Same 48 tasks: 41 right, 85 per cent. Small fixes and ordinary tasks it got every time. It sent four tasks that needed no coding agent to one anyway, and gave three tasks that needed the expensive model to the mid-priced one. The second mistake is cheap: the test fails and the task goes up a level.
One caveat: an agent wrote and labelled those tasks, following my rules. So the 85 per cent shows the router handles wording it hadn't seen. It doesn't show that people would label the tasks the same way.
Then it turned out it didn't fit on my home server. It wanted about 21 GB of memory out of 62, all the time. The rest of the machine started to choke. I switched it off. Once I'd cooled down, the maths was simple: whatever I'd save on tokens by routing, I'd have to sink into infrastructure that would pay for itself in about a hundred years.
So the PoC answered the question I actually had. The router works once you measure it and retrain it. For me, it doesn't pay off.
For now simple rules decide, and I don't know yet how often they're wrong themselves. That's the next thing to measure, on the same 48 tasks.
If you're putting AI decisions into a product and want them measured before they ship, I take on projects like this: Projects.
Drafted with AI agents, edited and fact-checked by me.