Home/Methods

Process · Product claims

The honesty gate: no feature claim without proof in the code

Before any submission we compare every claim on a plan page, a listing or a public page with the code that is supposed to back it. A claim stays only if a merchant can reach a working implementation; otherwise we build the feature or delete the claim. A script catches the mechanical cases, and the rest is read by hand.

Updated 2026-10-11

How we found out

On 13 July 2026 one of us clicked the button to start a 7-day trial in FAQ Genie, and nothing happened. Following that thread through our Shopify apps showed that the upgrade button was dead in 14 of 17 apps, that three apps would have created test subscriptions in production, and that we were selling features that did not exist.

Four apps had no real product. One showed ad metrics produced by a random number generator, one sold videos and had no encoder, one audited events from a pseudo-random generator, and one sold scheduled price refreshes with no scheduler at all. Four more had a paid tier that added nothing. TermForge, in review at the time, published a data-protection page saying it could not read orders while its configuration requested order access. Several repositories contained a copy file marked do not claim for exactly the features the plan page then sold.

None of it was deliberate. The copy was written before the product, and nobody compared the two at the end. The honesty gate is that comparison.

The rule

No bullet without proof in the code, and no page without proof in the code. A plan bullet, a listing claim or a sentence on a privacy, terms, FAQ or data-protection page may exist only if a reachable implementation backs it. Reachable means a merchant can get to it through the interface, or an automatic worker, webhook or scheduled job really runs it. Code that exists and is never called is not a feature.

When a claim is not true there are two options: build it or delete it. In the first remediation the agents were told to build when the work was small, and they produced real schedulers, CSV exports, a web pixel and a video renderer. Where a feature needed an external approval rather than code, we removed the paid tiers altogether, stripped the promises from every public page, and wrote a demo-data warning inside the exported file itself, because a file outlives a banner.

The mechanical half: a script

A script runs lint checks over each app's code and exits non-zero on any blocker. The first version had 12. They look for an upgrade path that swallows the success signal or never renders its error; test billing as the default; a paid priority queue that runs in the wrong direction (in our queue library 0 is both the default and the highest priority, and three apps gave paying customers 1); automatic or scheduled sold without a registered repeating job; a model call with no meter in front of it; a plan flag that nothing reads; the chassis app's name leaking into a fork's public pages; a data-protection page that contradicts the declared scopes; an entitlement checked in the interface but not in the action or the worker; and a record fetched by id without scoping it to the shop.

Every rejection since then has become a check, confirmed to fire on the code as it was before the fix: form inputs with a name but no value or change handler, which a reviewer cannot type into; a login route with no component, which rendered a bare null; the billing redirect that shows a 401 inside the iframe; a web pixel that called register without importing it and so never ran. That pixel had collected zero events for every shop that ever installed it, including four review stores.

A gate that reports blockers on correct code gets ignored, which is how real bugs hid in the first place. The first version produced four false blockers; each fix is now a comment in the script explaining why the check is narrow.

The manual half: a claim table

No script can tell whether a sentence is true. For every claim we write one row: the claim, a verdict and the evidence as a file and line. The verdict is works (implemented, reachable, and it does what the words say), partial (real but narrower than the words) or missing. We judge the words, not the intent. Some words lie more often than others: unlimited, every, whole catalogue, automatic, scheduled, priority, alerts, multi-store, export.

Search counts are not evidence. A symbol can exist as a type, a stub or dead code. One stub returned the literal string draft-order-deposits-coming-soon while the plan page sold the feature. We open the file and follow the call chain to an interface entry point or a worker. A number in the copy must be the number in the code: one app sold bulk changes across every audited page and capped them at 500, and sold full history while keeping 365 audits. A database column that nothing reads is not a feature, and a guard that hangs off a field nothing ever writes will never run.

The hardest class is a function that is called, wired all the way to the interface, and returns success for work it never did. One of our apps shipped an integration module whose own header described every function as a stub; the app showed a green synced badge and metered a billable unit for the feature it was named after. The script passed it with zero blockers. For any claim that moves money or touches an external system, we now read the body of the function that fulfils it, and we search every claimed path for stub, TODO, scaffold and coming soon.

Prove the effect, not the code

Code can be correct while the effect is absent. TermForge's balance page read rows that its order webhook was supposed to write. At the source, 72 webhooks had arrived and every one had been ignored, so the core feature had never fired once. We now ask of every dashboard what writes the rows it reads, and count webhooks in against rows out.

Tests can enshrine a bug. A charge failed on every attempt because an idempotency key was 40 characters against a platform limit of 32, and the unit test asserted the 40-character shape. Mocks can be kinder than the platform. In one of our Google Workspace add-ons, a mock answered a call that the declared scope refuses, and the suite stayed green while Outline did not work for anyone, on any document: 13 installs and zero opens in 10 days. Our mocks now keep a list of allowed methods per object and throw the platform's own error for anything else, and we check them by mutation: put the bug back and the suite must turn red.

So for anything that touches money or a platform, the last step is the real loop on a live store: place the order, watch the row appear, press the money button, see it marked paid.

Where we were wrong

We thought one careful audit was enough. A second pass the next day, with 11 apps already in review, found six more real bugs. Among them: an app whose entire product could never be created, because its extension and scopes had not reached the platform; a headline paid feature that anyone could keep using after cancelling a trial; one merchant able to read another's records; model spend with no upper bound, because failed calls were free and the merchant could trigger the failure; a page describing a safety check that never ran. The first pass had caught none of them.

The worst bug of that pass was ours, written while fixing the others. A billing webhook called a function with a variable that did not exist in its scope. The build was green because five apps did not type-check their routes, the code deployed, and no shop could have started paying. A subagent porting the same change to another app caught it by reading the shipped bundle. Since then, a fix to a money path gets a second reviewer, and routes must be type-checked.

The gate itself was also wrong after that. It accepted a null check as a tier gate, and it flagged working code that kept its cancel logic in another file. We fixed both, because a muted gate is how the real bugs hid the first time.

Limits

The gate shows that known classes are absent and that each claim has a reachable implementation. It does not show that the implementation is correct, that the platform accepts it, or that anyone wants the feature. It is a lint, not an oracle. The claim table and the live loop stay manual, and so far each new rejection has added a class we did not have before.

Checklist

Run the script on the app and fix every blocker. Write the claim table for every plan bullet, listing line and public page, with file and line. Build or delete every partial and missing row. Find the constant behind every number you print.

Read the body of every function behind a claim about money or an external system. Count ingest webhooks against stored rows. Click the upgrade in a real browser through to Shopify's approval page. Have someone else check your fix to any money path. Run the gate again after any change to a plan page, a public page, billing or a model call.

Questions and answers

How do I make sure an app's pricing page only promises features that exist?

Making sure an app's pricing page only promises real features means comparing every claim with the code that should back it before submission. Each claim gets a row with a verdict of works, partial or missing, plus evidence as a file and line. A claim stays only if a merchant can reach a working implementation. Partial and missing claims are either built or deleted.

What counts as a real feature when auditing app claims against code?

A real feature is one a merchant can reach through the interface, or one that an automatic worker, webhook or scheduled job really runs. Code that exists and is never called is not a feature. A database column that nothing reads is not a feature, and a guard that depends on a field nothing writes will never run. Search counts are not evidence; follow the call chain.

Can an automated script check whether an app's marketing claims are true?

An automated script can catch the mechanical cases but cannot tell whether a sentence is true. Lint checks can flag test billing as the default, scheduled features with no registered repeating job, plan flags nothing reads, and privacy pages that contradict declared scopes. Whether a claim matches the code still needs a hand-written claim table, and reading the body of every function behind a claim about money.

Why can passing tests still hide a broken app feature?

Passing tests can hide a broken feature because tests can enshrine a bug and mocks can be kinder than the platform. A unit test may assert the same wrong value the code produces, and a mock may answer a call that the real platform refuses. Mocks should throw the platform's own error for disallowed methods, and each fix should be proved by putting the bug back and watching the suite fail.

How do I verify that an app's billing and webhooks actually work?

Verifying an app's billing and webhooks means proving the effect, not just the code. Ask of every dashboard what writes the rows it reads, and count incoming webhooks against stored rows. For anything that touches money, run the real loop on a live store: place the order, watch the row appear, press the money button and see it marked paid. A fix to a money path should get a second reviewer.

All methods