A large product catalogue needed a category before a launch. The obvious proposal was the fashionable one: send every product photo through a vision model and let it decide.
That meant a recurring inference bill on every pass, a runtime nobody wanted to wait for, and failure modes nobody could reason about.
I looked at the data first. The product title already carried almost all of the signal, so the job did not need to see anything. It needed to read.
A lightweight text model classifying by title handled the whole catalogue for a small fraction of what the vision route would have cost, fast enough to rerun whenever the catalogue changed. The job is resumable: if it stops halfway, it picks up where it left off instead of paying for the same rows twice. The vision model stays in the toolbox for the products whose titles say nothing.
That is what orchestration means in practice: not using AI less, but sending each row to the cheapest model that can actually read it — and proving that before paying for anything heavier.
The Default That Costs You
A lot of AI architecture starts from the wrong question. Not “What is the simplest mechanism that solves this?” but “Where can the model go?”
Models are extraordinary at the parts of a problem that need judgement. Pointed at the parts that do not, they add cost and uncertainty without adding accuracy. Unmeasured model calls are technical debt with a recurring invoice attached.
Here is what happens when structured data goes through a large model by default:
- You lose determinism. A rule that always gives the same answer becomes a probability. Now the output needs its own validation step.
- The latency tax. A composite database index lookup takes milliseconds. An external model roundtrip takes hundreds of milliseconds. Multiply that across a large catalog, and your ingestion pipeline collapses under its own weight.
- Unit economics run backwards. Cost grows linearly with volume, so the more the business scales, the more the pipeline eats into the margin — the opposite of what software is supposed to do.
When Not to Use the Heavy Model
Before a job reaches for a bigger model, it has to fail on a smaller rung first. You never pull a heavier tool when a lighter one solves the equation:
- Rung 0: Structured fields. If the category, size or material is already a field, read the field. Nothing gets inferred that was already stated.
- Rung 1: Deterministic matching. Normalised tokens, lookup tables and composite SQL indexes. Rules are calculated, not guessed.
- Rung 2: A lightweight text model. Cheap, fast, batchable — the rung where most of the catalogue landed.
- Rung 3: A multimodal model. The last resort, for items where the text genuinely says nothing and only the image carries the signal.
Measure each rung on a sample before it goes near the full catalog: accuracy against a hand-checked set, cost per thousand items, time per run. The numbers pick the rung, not the hype.
The orchestrator’s job is routing: every item goes to the lowest rung that handles it, and only the leftovers climb. Most commercial problems resolve on the first three rungs — which is exactly why the fourth is affordable when you do need it.
The Builder’s Real Job
Wiring a model into a pipeline takes an afternoon. The real work is deciding where it belongs: looking at a messy, high-volume system, measuring what each step actually needs, and shipping something that runs unattended without surprising anyone on the invoice.
If you are a founder, judge your AI architecture on two numbers: accuracy on a hand-checked sample, and cost per thousand items. What looks sophisticated is not on the list.
→ The rule: Before a job climbs to a heavier model, show that the rung below it fails on a real sample. If you cannot show that, the heavier model is not solving a problem yet — it is just more expensive.