Start from the register, not from the scrape

For a while I had a pipeline that worked in what felt like the obvious direction. Find candidate organisations out in the world, gather what you can about each one, then check each against the official companies register to confirm it is what it appears to be. Every stage of that made local sense when I wrote it, and the whole thing generated a steady stream of work I had come to think of as the cost of doing business. Same-name collisions needing a human eye. A review queue that never quite emptied. Model spend burned on judging records that turned out not to be eligible at all.

That work existed because of the ordering, and almost none of it survived reversing it. Matching a scraped entity back to an authoritative source is a fuzzy join, and fuzzy joins manufacture ambiguity. Two businesses in different towns share a trading name, and now something has to decide which one you meant, using evidence that was never designed to answer that question. Multiply by a few thousand records and you have built yourself a permanent triage desk. The register, meanwhile, was sitting there the whole time: free, authoritative, and structured around exactly the identity question I was paying a language model to guess at.

Reading from the register first inverts all of it. Harvest from the authoritative source, enrich afterwards, judge last. The attribute that was hardest to establish becomes certain and costs nothing, because it is the primary key of the thing you started from rather than a conclusion you inferred. Everything downstream gets cheaper as a consequence: expensive judgement only ever runs on records that have already cleared the deterministic filters, so the model is never asked to reason about something that a free lookup could have excluded. I had been paying for inference to compensate for a data-sourcing decision.

Running the flipped version at volume turned up a second thing worth keeping. The pipeline records its progress as an append-only event ledger and derives current state by reading that log, rather than maintaining a running status somewhere. Partway through a large run, a third-party search API started returning payment errors. Nothing had been written to handle that, and nothing needed to be. Resuming was re-running the same commands, because the ledger already knew which records had genuinely been processed. Recovery code I never wrote was the recovery code that worked.

Then the part I nearly missed. That same ledger had quietly written roughly seven hundred and twenty-five error events using the identical shape as successful ones. Downstream, every one of them read as a completed lookup that happened to return nothing. "The request failed" and "we looked and there was nothing there" are opposite facts with opposite consequences, and for a stretch of that run they were indistinguishable in my own data. Derive-on-read is a genuinely good property, and it means a malformed record stays wrong every single time anyone reads it, silently, until somebody asks precisely the right question. The failure mode of a system that reconstructs truth from a log is a log that lies consistently. This is the same shape as verification that reads an output while the input underneath it is quietly broken, one layer further back.

The run also settled an argument I had been having with myself in prose. I had previously asserted, on reasoning rather than evidence, that swapping one search backend for another would degrade results for a British dataset. Instead of relitigating that, I ran forty identical companies through both, with byte-identical scoring on the other side, and read the difference. Whatever the answer had been, the method is the reusable part: a prior directional call, held in place by nobody having tested it, is cheap to convert into a measurement when the only variable you change is the one under dispute. That is the same discipline as choosing a cheaper model on a controlled comparison instead of a hunch, applied to infrastructure rather than inference.

One friction refused to go away, and I want to record it honestly because it has now bitten three separate times in the same build. Eligibility kept being implemented as a predicate evaluated once, at the moment a record entered the system, and then trusted forever by every stage after it. A record that became ineligible later, or that entered by a path which skipped the check, sailed through everything downstream without anything noticing. Building the right filter and applying it in the right place turns out to be two different pieces of work, and I have only reliably done the first.

Harvest from the source of truth, enrich, then judge. If a pipeline is spending money to establish something a register already knows, the ordering is the bug and no amount of better judgement downstream will fix it.