How to evaluate whether a cheaper model is good enough for a task

I run a lot of bulk classification through the smallest model I can get away with, because at volume the cost difference is real and the latency difference matters. The mistake I nearly made was deciding "get away with" by eyeballing a few outputs. They looked fine. On the axis that actually mattered for the task, they weren't, and the only way I found out was building a gold set and holding both models against it.

The task is a lead-scoring agent for a side project. It looks at a small business, works out what web presence they actually have, and decides one thing: is this a rebuild lead worth pursuing, or should I drop it. Everything hangs on that keep-or-drop call. I had it running on the cheap model and it was producing sensible-looking output all day.

So I built a gold set. Sixty-five businesses I'd labelled by hand, with the presence category stripped out and the platform hints neutralised so the model couldn't cheat off them. Then I scored the same sixty-five twice, once on the cheap model and once on the larger one, through the identical pipeline. On the easy majority, the businesses with no owned site at all, the two models agreed with each other and with my labels around 90 percent of the time. If I'd only looked there, I'd have concluded they were interchangeable and shipped the cheap one.

The disagreement was concentrated in exactly the place the task lives. The larger model made zero mistakes on the keep-or-drop boundary. The cheap one made seven, every one of them the expensive direction: calling a thin, templated, one-page site "good" and silently dropping a real rebuild lead. Seven dropped leads out of sixty-five, all invisible if you're glancing at outputs rather than grading them, because a wrongly-dropped lead looks like a lead that wasn't there, not an error.

Here's the part I'm most glad I caught. My first A/B run gave the larger model a hardened prompt (one extra sentence spelling out that a dated or templated or one-page site is a rebuild lead, not a good site) but ran the cheap model on the old plain prompt. That confounds the model with the prompt. Any difference could be either. So I re-ran the cheap model on the identical hardened prompt, which gave me a clean three-way read: cheap-plain, cheap-hardened, larger-hardened. The prompt fix alone took the cheap model from seven mistakes to three, recovering four leads including the single best one on the whole list, a business with over a thousand reviews sitting behind a one-page site. Most of the gap was prompt adherence, not raw capability. But not all of it. Three cases remained where the cheap model still called a thin template "good" and the larger one correctly called it a rebuild lead. That residual three is a genuine capability difference, and at volume it compounded: on a fresh batch of thirty-two unseen businesses, the cheap model qualified eleven leads and the larger one qualified twenty-two.

The A/B threw off two things I wasn't expecting. Cost turned out to be a rounding error either way, one to two pence per lead, and the larger model was not slower despite being larger, because it made fewer, more targeted tool calls. And the gold set surfaced its own staleness: four businesses both models independently flagged as having real modern sites, which my hand labels had marked as leads. Both independent models finding a working site is strong evidence my labels were the thing that was wrong, so those four got retired from the list. A good eval cleans your ground truth as a side effect.

The general lesson I took from this holds well beyond one scoring agent. Model choice is not a vibe, and it is not settled by looking at output that looks right. Find the one axis the task actually turns on. Build a small held-out set that measures that axis and nothing else. Control your variables so you're testing the model and not accidentally the prompt. And be honest about which failures are expensive, because the cheap model failing on the thing you don't care about is fine, and the cheap model failing on the thing you do care about, quietly, is how you ship a system that looks like it works.