Home/Methods

Research · Method

Triple deep research, fused into one roadmap

At the start of every project, and again before every major version, we run the same research question through three AI deep-research engines and fuse the reports into one master roadmap. The value comes from the disagreements between engines and from verifying every load-bearing claim at source; the reports themselves are drafts, not findings.

Updated 2026-10-11

What this is for

A first specification describes the obvious version of a product. The ritual exists to map the full feature space before we commit to an architecture: what the best competitors ship, what neighbouring industries do that nobody on our platform has copied, what a much better version would contain, and which of those ideas merchants actually keep using.

We run it at kickoff, after the first version is live, and before each large version, because a live app changes what is worth building next. On one of our Shopify apps the fused research produced the rich-text editor and the whole third phase of the roadmap, none of which was in the original spec. We also used the same ritual on our own build-and-ship process, not just on products.

Step 1: write the prompt as a case file

The same prompt goes to all three engines. It asks for five things: everything best-in-class competitors ship, with citations to their listings; what adjacent tools do that no app on the platform has copied; what a far better version would include; every idea ranked by value against build effort; and what is overhyped compared with what actually retains users.

When the question is strategic rather than a feature map, the prompt carries our own data, and that is where it goes wrong first. One draft of a strategy prompt contained five wrong figures, four of them material. One was a count quoted per week that had actually been measured over 28 days. Another was presented as retention and was in fact the install completion rate. So every load-bearing number is re-read from its primary source on the day the prompt is sent, with its time window and unit written next to it.

We then attack the draft with six independent critics: a fact-checker with access to sources, a reader looking for leading questions, a consultant who lists what is missing, a reader with no context, a devil's advocate on the narrative, and one that answers the prompt and then judges its own answer. We keep what at least two of them found, plus anything proved wrong by one, and rewrite. The last critic is consistently the most useful: it is how we learned that six open questions produce six essays at a depth where nothing can be wrong.

Step 2: three engines, run in parallel

ChatGPT deep research and Gemini deep research are run by a person and the reports brought back. Claude runs its own multi-source pass at the same time, reading competitor listings, documentation and reviews directly.

Each engine reliably finds things the others miss. In our runs ChatGPT gave the most breadth on competitor features, Gemini the strongest angles on the Google ecosystem and search, and Claude the implementation feasibility and the honest positioning. On our build-process research, three gaps were missed by both external engines and came only from the third leg. The point of three engines is not three times the content. It is that disagreement between them surfaces inventions that any single report would present with full confidence.

Step 3: fuse, don't stack

Stacking three reports gives you a long document with three versions of every idea. Fusing means: remove duplicates, cluster what is left into themes, then run an adversarial critique over the whole set. What did all three miss? What is hype? What would break our platform permissions or data-protection level? The output is one master roadmap in the app's repository, split into phases that can each ship on their own.

We tag each item by how many sources converged on it. On one fusion we combined five passes and marked items by agreement across five, four or three of them. Convergence is a signal about priority, not proof of truth: five reports can repeat the same mistake if they read the same weak source.

Step 4: verify at source before anything depends on it

Deep-research reports are good at structure and weak at specifics. We measured this. Across six Gemini reports on one market, three delivered no app identifiers at all even though the prompt required them, which made half the deliverable impossible to check; and 2 of 37 identifiers were fabricated. In a larger round, 171 claims from nine reports were taken to primary sources and 46% did not hold: 40% refuted, 6% unverifiable. Raw figures copied from official records were mostly right. The interpretation layer built on top of them was where most errors lived.

The errors had a direction. Numbers were inflated consistently upwards, a currency mix-up turned an ordinary margin into an extraordinary one, and every claim of white space, meaning that nobody offers something, failed whenever we tested it. So every name, number and mechanism a roadmap item depends on is checked against the marketplace, the official documentation or the public record before the item is scheduled.

The traps we hit, and how we catch them now

The fact list propagates errors. To stop agents inventing things we give them a fence of confirmed facts. The fence also carries our mistakes, with the same authority. Building a question bank, this happened four times; one inverted fact produced a question with no correct answer, and the explanation repeated the inversion, so the generator's own checker could not catch it. Three independent verifiers found it only because they read the official text in full rather than our list. Facts now live in a versioned file, each with its source and the date it was read, and verifiers are told to challenge the list, not just the output.

A shared verifier cancels a different angle. In one sweep we ran five independent angles, each followed by an adversarial verifier. The verifier was a single prompt written for the main question. The fifth angle asked something different, so its findings had a different shape, and the verifier rejected all of them: 3 raw findings, 0 confirmed. One of the rejected findings was real and more urgent than the original question. Now each angle gets its own refusal criterion, and we report raw findings next to confirmed ones, because a zero made by the harness looks exactly like a zero made by reality.

Proposals name mechanisms that do not exist. Of ten app ideas proposed by AI research for a developer platform, four depended on a module, a schema field, an event or an endpoint that the platform does not have or does not allow apps to use. Each of those fails silently after deployment. No roadmap item that depends on a platform mechanism is accepted until we can paste the documentation sentence saying the mechanism performs the action.

Circular validation. One report returned a figure from our own prompt as if it were an independent finding and built four sections on it. We no longer put numbers into a prompt that we want the research to verify. We give the question, not the answer.

Where we were wrong

We have not always run all three legs. Once the Claude leg was throttled by the search service and came back with a false empty result, so its sections were written from our own knowledge of the codebase instead. Another time the ChatGPT quota ran out and all six reports came from one engine. Verification at source was the only defence left, and it was enough only because that marketplace could be scanned. On a channel we cannot scan, a single-engine set should not drive a decision.

We also over-spent on process. A generic research workflow with adversarial voting consumed about a million tokens once and returned zero sources, because every fetch had been rate-limited. We now probe the rate limit with one search and one fetch before launching anything expensive, and for narrow technical questions a focused pass of parallel searches is cheaper and better.

Limits

The ritual widens the option space and orders it. It does not establish demand; that is a separate test against marketplace data. It does not prove that a feature will retain anyone, and convergence between engines can be shared error. A fused roadmap is a plan to be re-run when a phase lands, not a forecast.

Checklist

1. Gather facts first, from primary sources, with dates, windows and units. Then write the prompt.

2. Run six critics over the draft; keep what two found and every proven error.

3. Send the same prompt to all three engines. Do not skip a leg because one engine can search.

4. Dedupe, cluster, critique, and write one phased master roadmap.

5. Verify every name, number and mechanism an item depends on before it is scheduled.

6. Give each verification angle its own criterion; report raw and confirmed counts together.

7. Re-run the ritual when a phase ships.

Questions and answers

Why use several AI deep-research tools instead of one?

Using several AI deep-research tools pays off because each engine reliably finds things the others miss, and disagreement between them exposes claims a single report would state with full confidence. In practice one engine gave the most breadth on competitor features, another the strongest angles on search, and a third the implementation feasibility. The goal is not more content but a way to surface inventions before they shape a roadmap.

How accurate are AI deep-research reports?

AI deep-research reports are good at structure and weak at specifics. When 171 claims from nine reports were taken to primary sources, 46% did not hold, and raw figures copied from official records were mostly right while the interpretation built on them carried most errors. The errors leaned one way: numbers inflated upwards and claims that nobody offers something failed whenever tested. Verify every load-bearing claim at source.

How do you combine several deep-research reports into one product roadmap?

Combining several deep-research reports into one roadmap means fusing them, not stacking them. Remove duplicates, cluster what remains into themes, then run an adversarial critique over the whole set: what all reports missed, what is hype, and what would break platform permissions or data protection. The result is one master roadmap split into phases that can each ship on their own, with items tagged by how many sources converged.

How should a prompt for AI deep research be prepared?

A prompt for AI deep research should be written like a case file, with every load-bearing number re-read from its primary source on the day the prompt is sent, along with its time window and unit. The draft is then attacked by several independent critics, keeping what at least two found plus any error proven by one. Numbers the research is meant to verify should be left out, to avoid circular validation.

Why can a single verifier wrongly reject valid research findings?

A single verifier can wrongly reject valid research findings when it was written for the main question and is applied to an angle that asks something different. Findings from that angle have a different shape, so the verifier refuses all of them, and one rejected finding may be real and urgent. Each angle needs its own refusal criterion, and raw findings should be reported next to confirmed ones.

All methods