When the answer is not an LLM (and how to tell)
About one in four strategy engagements ends with us recommending something more boring. Three questions we ask first.
A feature that costs two cents a call is fine at a thousand users and painful at a million. Cost has to be designed in, not discovered on the invoice.
Route by difficulty
Send easy requests to a small, cheap model and only escalate hard ones. In most systems we run, 70–80% of traffic never needs the largest model.
Cache aggressively
Many questions repeat. Caching answers and prompt prefixes is the cheapest optimisation there is.
Measure per feature
Tag every call with the feature that made it. You cannot fix the expensive feature if all you see is one monthly total.