Home/Research

Post-mortem

A negative search result is not evidence of absence

12 of 20 app listings sat rejected for over two weeks because the reviewers' questions lived in a ticket system, and the mailbox search that would have found the notifications was never validated.

Published 2026-08-27

What we found

A routine check of our marketplace listings returned a number nobody expected: 12 of 20 were sitting rejected. Not queued, not in review — rejected, and rejected for over two weeks. Nobody on our side had reacted, because nobody on our side knew.

The interesting part is not the count. It is that a rejection there is not necessarily a verdict on the software. Each submission to that marketplace opens a support ticket, and every question a reviewer asks lives in that ticket. One of the available rejection reasons is simply the review expiring because nobody answered.

So a listing can fail for silence. The code builds, the app runs, the reviewer asks something ordinary, and the clock runs out in a room we were not standing in.

The verdict lives in the ticket, not the inbox

Treat this as an architectural fact about the marketplace rather than a matter of etiquette. The ticket is the channel of record. The email is a notification about the ticket. Read the notification and you learn that something happened; open the ticket and you learn what.

The distinction matters because the two channels fail in different ways. Mail can be filtered, filed under a category, marked read by a sweep, or simply never matched by the query you ran. The ticket cannot: it holds the state, and the state is what the review acts on.

Our mental model had email as the primary channel. That is the one assumption everything else went wrong on top of.

Where we were wrong

We searched the mailbox for the marketplace's main domain. Nothing came back. The error was twofold. First, the notifications are sent from a different domain belonging to the same company, so the query could not match by construction. Second, the mail client filed those messages under a secondary category, so they never surfaced in the primary inbox either — two independent causes of the same silence.

Then we did the damaging thing. On the strength of an empty result we told the owner there had been no review correspondence at all, and we built a theory of the rejections on top of that. The mailbox in fact held 83 messages from the notification domain, 77 of them carrying ticket links, covering 57 distinct ticket ids. All of them unread.

The lesson is narrower and more useful than "check your email". A negative search result is not evidence of absence until the query itself has been validated. An empty result is a statement about your query first and about the world second, and from the result alone you cannot tell the two apart.

Validation is cheap: obtain a known-good message by another route, then confirm that your query actually returns that message. If it does not, the query was wrong, and every conclusion drawn from its emptiness is void.

What the silence cost

The unanswered tickets expired in a batch. 13 listings were rejected for timeout. 2 more were rejected for a genuine security defect — real, ours, and it had to be fixed. All of them were repaired and resubmitted the same day.

The temptation at that point is to resubmit everything and move on. It does not work: resubmitting without reading the ticket only restarts the same clock, with the same unanswered question waiting at the far end. One listing we resubmitted blind moved from "timeout" to a real verdict that had been sitting in the ticket all along. We had bought a delay and learned nothing.

A timeout rejection carries no information about the software. It only tells you where you were not looking.

Access to your own ticket is not guaranteed

One further obstacle deserves stating plainly, because it breaks the obvious fix. When we went to read the backlog, 7 of 12 tickets returned "no access" to the very account that had opened them.

Requesting access is itself a ticket. The recovery path for a ticket you cannot read therefore runs through the same system that is timing you out, which makes an access problem discovered late an access problem discovered too late.

The process consequence is simple: prove you can open the ticket at submission time, while somebody is still watching and the clock does not yet matter.

How to run the same check

At submission time, capture the ticket identifier, open it once to prove you can, and put it on a poll. A submission is not complete when the form is sent; it is complete when the ticket it created is identified, readable and being watched.

Then validate every negative search before you report it. Confirm the sender domain from a known-good message instead of assuming it matches the marketplace's public domain. Include the spam folder and the secondary categories in the query. If it still returns nothing, say "the query returned nothing" — and only say "there was no correspondence" once the query has demonstrated that it finds a message you know exists.

What we run now is small: an alarm twice a day against the two channels that actually carry a verdict. It classifies each message and raises a notification only when something needs an answer. The value is not in the classifier; it is that the check is periodic and does not depend on anyone remembering.

Limitations

This is one company's experience of a handful of marketplaces over a short period. The specific domains, categories and interfaces will differ, as will the rejection reasons a reviewer has available. Nothing here describes how any particular marketplace behaves today.

What generalises is the shape: a review conversation held somewhere other than where you are looking, and a null search result accepted as a fact. Both are cheap to test for and expensive to leave untested.

The counts are from the moment we looked. We have not measured how often the pattern recurs, nor whether an alarm twice a day is frequent enough for the deadlines each marketplace sets.

Questions and answers

Why was my app rejected for timeout in a marketplace review?

An app is rejected for timeout in a marketplace review when the review expires because nobody answered a reviewer's question in the support ticket the submission opened. A timeout rejection carries no information about the software; it only shows where nobody was looking. In one case, 12 of 20 listings sat rejected for over two weeks because the questions lived in tickets while the team watched email.

How do I validate an empty email search?

Validating an empty email search means proving that the query can find a message you know exists. Obtain a known-good message by another route, confirm the real sender domain from it, include spam and secondary categories in the query, and check that the query returns that message. Until then, report only that the query returned nothing, not that there was no correspondence.

Should I resubmit an app straight after a timeout rejection?

Resubmitting an app straight after a timeout rejection, without reading the ticket, only restarts the same clock with the same unanswered question at the far end. One listing resubmitted blind moved from timeout to a real verdict that had been sitting in the ticket all along. The useful step is to open the ticket, answer the reviewer and fix any genuine defect before resubmitting.

What should I do when submitting an app to a marketplace that uses support tickets?

When submitting an app to a marketplace that uses support tickets, capture the ticket identifier, open the ticket once to prove access, and put it on a regular poll. Access is not guaranteed: in one backlog, 7 of 12 tickets returned no access to the account that had opened them, and requesting access is itself a ticket. A periodic alarm on the channels that carry verdicts removes reliance on memory.

Why can a mailbox search miss marketplace review emails?

A mailbox search can miss marketplace review emails because notifications may come from a different domain owned by the same company, and the mail client may file them under a secondary category. Searching for the marketplace's main domain then returns nothing by construction. In one mailbox, that empty search hid 83 unread notification messages covering 57 distinct tickets.

All research