There is an asymmetry inside almost every software company, and almost nobody looks straight at it.

A function that returns the wrong value breaks a test. A manifest that drifts from what’s deployed raises an alert. A dependency with a known CVE stops the pipeline. Everything you execute is watched by something.

And then there is the sentence on your privacy page promising which region your customer’s data lives in. Somebody wrote that sentence fourteen months ago, holding the whole context in their head. The infrastructure has changed three times since. The sentence hasn’t. And there is nothing, anywhere, that turns red when it stops being true.

We ran that exercise against our own site: take every published claim and find, in our code and in our architecture decisions, the file that backs it. It didn’t come through clean. We fixed what needed fixing in the same pass, and what we learned is worth more than the inventory of our mistakes.

What we took away, in five lines:

Every public claim needs a citable source in the repository, not somebody who remembers.
The audit has three outcomes, not two: confirmed, contradicted, and no source.
The dangerous one is the third, because it doesn’t look like a failure.
Your internal documents are not authority. What executes is.
What a script can check, a script should check — and a 200 is not a verification.

Claims are written once and read forever

The problem isn’t that people lie in their marketing. It’s that claims have a strange life cycle: they’re written exactly once, at the moment of maximum knowledge, and from then on they are only read.

Code, by contrast, gets touched. Every time somebody opens it they have a chance to notice that something no longer adds up. A product page can go two years without anyone technical reading it line by line, while underneath it the infrastructure, the plans, the limits and the providers all move.

A small example from our own pile, with no gravity to it at all. In our agent’s install documentation, the example command pointed at the wrong Service: the generic name and port that ship with the reference chart, instead of the real ingest service. Anyone who copied that command ended up with an agent that started cleanly and reported nothing. No visible error, no red log line.

Nobody caught it for months, for a very simple reason: documentation doesn’t execute. Nothing tests it. It’s prose, and prose doesn’t fail — it just ages.

Three outcomes, not two

The trap in this exercise is framing it as a true-or-false exam. Do that and you end up with a list of corrections and the reassuring feeling that everything else is fine. It isn’t.

Every claim lands in one of three buckets:

Outcome	What it looks like	What it actually is	What you do
Confirmed	Correct	Correct, and checkable again tomorrow	Record the source next to the claim
Contradicted	A mistake	A mistake with an obvious fix	Fix the text, never the source
No source	Correct	Unknown	Find who decides. If nobody does, pull it

That third bucket is the interesting one, and it’s exactly what you lose if you only hunt for contradictions. A claim with no source isn’t wrong: it’s unverified. Nobody can confirm it and nobody can refute it, you included. It may have been true by coincidence for two years, and stop being true next Tuesday with nothing to record the change.

A claim with no source is not a correct claim. It’s a claim nobody can refute — you included.

When you find one, the useful question isn’t “is this true?”. It’s “who decides whether it’s true, and where does that decision live?”. If the answer is a file, you have a source. If the answer is a person, you have a different kind of problem. And if there is no answer, the claim shouldn’t be published.

Your internal documents are not authority

This is the mistake we hit most often, and the easiest one to make in good faith.

When you audit, you need something to check against. The temptation is to use internal documentation: the repository’s context file, the spec, the wiki. It’s convenient, it’s written in prose, and it answers fast.

It is also exactly the same kind of artifact you are auditing. An internal document was written once and read many times too. It ages too. Treat it as authority and you don’t find the error — you launder it, because now you have two texts agreeing with each other and neither of them verified.

This happened to us literally. A claim on the site didn’t match the product’s own context file, and the comfortable conclusion was to fix the site. Going down to the code, the thing that was out of date was the context file. The site was telling the truth.

The rule we came away with is uncomfortable but short: authority is whatever executes. The migration, not the spec. The config file, not the diagram. The line in the handler, not the comment above it. Everything else is a well-meaning opinion about the system.

The silent failure: data with misplaced confidence

The second example we like to tell broke nothing, which is precisely why it’s the better one.

We had run an A/B test on the home page: two versions, a cookie-based split, signups attributed to each arm. An ordinary setup. In August we retired the experiment and shipped a new home page.

We retired the experiment. We did not retire the split.

For a good month, the middleware kept assigning a cohort to every new visitor and storing it for ninety days, while the page always served the same home. The result wasn’t an error — it was two analytics systems describing the same person differently. One reported the hero that actually rendered. The other reported the arm of a coin toss that no longer corresponded to anything anyone had seen. And signups were being tagged with an imaginary cohort.

No alert fired, because from the outside everything worked. The dashboards filled with numbers. The numbers had the right shape. They simply weren’t measuring what they said they measured.

That’s the signature of the third bucket: not a failure, but unearned confidence. And it only surfaces when somebody sits down and asks where each number comes from.

What a script can check, a script should check

A good part of this work is careful reading, and that part doesn’t automate. But there is a subset that does, and it’s worth moving out of the realm of human judgement as early as you can:

Parity across languages. If you publish in two languages, the key counts should match and nothing should be orphaned on one side. It’s a three-line check that catches entire sections that exist in only one language.
Commands that resolve. If your documentation names a service, a port or a path, verify they exist in the manifests you publish.
Queries that run. If a report prints SQL for other people to paste, execute it against the database before you publish it. A SELECT with a misspelled column reads beautifully.

And one lesson that cost us a while, because it’s counterintuitive. We wanted to link a filtered list of releases on GitHub, and the URL with the filter parameter returned 200 OK. It looked right. Counting the items in the response, it turned out to be returning the entire unfiltered list: the parameter was being ignored silently.

A 200 is not a verification. It confirms that the server answered you. If you want to know whether something filters, sorts or excludes, you have to compare the result against one you know to be different. That holds for the URLs you link, and it holds in exactly the same way for the claim you wrote about them.

What’s left

We found more than this article covers, and fixed it in the same pass. But an inventory of our stumbles ages just as fast as the page that held them, and it isn’t the part that helps you.

What actually changed isn’t the list of corrections: it’s that every public claim now carries, in the code itself, a note pointing at the source that holds it up. When that source changes, the change shows up as a diff in a code review, in front of a human, instead of sitting still on a page nobody re-reads.

It’s the same idea the product we sell rests on, turned on our own house: detecting is worth little if nothing remembers. A claim with its source beside it has a memory. A loose claim only has the memory of whoever wrote it.

If you want to see how it landed, our trust page came out of this audit worse than any other and changed the most. It’s published, with its sources, at /trust.