Home/Research

Measurement · refusal risk

Geography predicts refusals. Shipping cost only leaks the answer.

A refusal model trained on roughly 628,000 finished orders scores AUC 0.686 on an honest temporal holdout, and its riskiest fifth of orders holds 39% of all refusals. The first version scored 0.55 because it was reading the answer off a field that only exists after the parcel comes back. Most of the address rules we were confident about did nothing once we put a control group next to them.

Published 2026-08-24

What the ranking buys you

The model is trained on roughly 628,000 finished orders. About 16% of them were refused. 92% were cash on delivery, which is the mechanism that makes a refusal nearly free for the customer and expensive for us: the parcel travels twice and nobody pays for it.

We evaluate on a temporal holdout. Train on the past, test on the most recent weeks, never on a random split. A random split lets the model learn from orders that happened after the ones it is scoring, and every delivery dataset has enough seasonality and enough repeat customers for that to matter. On the honest split the model scores AUC 0.686.

What that is worth in operations: the riskiest 20% of orders contains 39% of all refusals, at 29% precision, which is 1.9x the base rate. Tighten the cut to the riskiest 10% and it holds 24% of refusals at 37% precision. That is a call list ordered by expected waste, not a judgement about any one parcel.

The strongest features are postcode and city. Geography is the single strongest predictor in this dataset. Order value and the customer's own prior refusal rate, keyed on phone number, are also among the strongest. What the model leans on describes where the parcel is going and who ordered it, not what is in the box.

The first model was reading the answer

The first run scored AUC 0.55 and shipping cost dominated the feature importances. That combination should have been the first warning. A field that dominates a model and still leaves it barely above chance is usually not a feature. It is a symptom.

Shipping cost leaks the label. A returned parcel costs more, because the recorded figure includes the return leg. The number is therefore partly a description of the outcome we are trying to predict, and, more importantly, it does not exist at the moment the prediction is needed. When the order is placed, nobody knows the parcel will travel back.

Removing shipping cost and the other post-fulfilment fields raised the honest AUC from 0.55 to 0.686. That direction surprised us, because leakage is supposed to inflate a score rather than depress it. The explanation we find most plausible, and we did not need to settle it, is that the return leg is billed late, so in the recent weeks used for testing the extra cost had not landed yet. The model leaned on a column that was informative in the training period and empty of that information exactly where it was scored.

The rule we now apply without exception: in a delivery or refusal model, never use a field that is only known after the outcome. The practical version is to go column by column and name the process that writes it. Anything written by fulfilment, by courier settlement, or by the returns desk is out, however strong it looks.

Two obvious features that did nothing

We then added two features we were confident about: phone-number validity, and a parse of address completeness. Both are cheap to compute, both are standard advice, and both describe exactly the kind of sloppiness we assumed precedes a refusal.

AUC moved from 0.686 to 0.687. Neither feature appeared in the top fifteen by gain.

The reason is not that the idea is wrong. It is that the signal was already in the model. Geography and the customer's own history carry it, and a structurally invalid phone number is rare enough that there is almost no variance left for the feature to explain. A feature can be true and still be redundant.

So the check to run before adding an obvious feature is whether it is already subsumed. Train with it and without it on the same holdout, and read both the score and where it lands in the importances. An idea that survives that test is worth the code. One that does not is a maintenance cost with nothing in return.

The address study, and what a control group does to it

The second measurement is separate from the model. On 24 August 2026 we took 7,426 orders that the courier itself had flagged as having a bad address, and compared every candidate signal against a control group of orders it had not flagged. The control group is the whole point. Without it, any rule that fires often enough will produce convincing examples.

Our own address validator called 95% of those orders valid. Its objections turned out to predict almost nothing. Addresses with no house number ended in a loss 8.9% of the time. Addresses whose postcode could not be derived came in at 7.0%, which is below the 8.4% average. A street field the validator classed as garbage gave 9.7%. Only one objection carried real signal: locality not found, at 15.0%, about twice the average.

The best signal we produced ourselves is cruder than any of those rules. An address containing no digit at all appears on 14.6% of courier-rejected orders versus 3.2% in the control group, a factor of 4.5. No street number, no block, no flat, nothing numeric anywhere in the field.

The strongest signal was not ours at all. It was the courier integration's own address verdict, which we had been treating as noise. Its INCOMPLETE verdict appears on 13.2% of rejected orders versus 0.9% of the control, a factor of 14.7. UNKNOWN appears on 7.2% versus 0.5%, a factor of 15.4. Together those two verdicts catch 20.4% of all courier rejections while touching only 1.4% of orders, which is the ratio that makes a manual queue affordable. The same integration's WRONG verdict is weak by comparison, at 1.8 times.

Four assumptions the control group killed

A village address is not riskier than a city address: 8.1% against 8.3%. We had built the opposite into our intuition, and into the way we talked about rural deliveries.

A block-of-flats address is safer, not riskier, at 7.1%. A missing postcode is below average at 5.6%, which fits the finding that our validator's postcode objections are worthless. And the absence of a street-type prefix made no difference at all.

Every one of these beliefs came from real cases. We had all seen a parcel lost to a village address with no number on the house. What we had not done was ask how often the same trait appears on the parcels that arrive without incident. That question is the control group, and it is the cheapest instrument in the whole exercise.

How to run the same check yourself

Split by time, not at random. Fit on everything up to a cut-off date and score the weeks after it. If the score drops when you move from a random split to a temporal one, the earlier number was never real.

Audit every column for its write time. For each field, name the process that fills it and the point in the order's life at which it stops being null. Fields written by fulfilment, courier settlement or returns are unusable, even when they look like properties of the order. Shipping cost is the trap, because on old rows it is populated and looks static.

Measure every rule against a control group, and report the ratio rather than the raw rate. A rule that fires on 8.9% of bad orders means nothing until you know what it does on good ones. Put the base rate in the same table, so an objection that lands below it is visible immediately.

Then check coverage before you celebrate a ratio. A rule with a factor of 15.4 that touches a handful of orders is a curiosity. The pair of courier verdicts above is useful because 20.4% of rejections sit inside 1.4% of orders. The ratio tells you whether to trust a flag. Coverage tells you whether to build a process around it.

Limitations

This is one market, one courier ecosystem, and an order mix that is 92% cash on delivery. Refusal is close to free for the customer here, which is not true where payment is taken up front. None of these figures should be carried to a card-paid market without repeating the measurement.

The model is a ranking tool, not a verdict on an individual order. At 29% precision, most of the orders in the riskiest fifth still arrive and are paid for. Anything built on top of it has to be cheap and non-punitive: a confirmation call, an address check, a different packing decision. Cancelling on the score would destroy more than it saves.

The two measurements are also not the same measurement. The model predicts customer refusal. The address study describes orders the courier flagged as undeliverable, from a single day's extraction. They overlap, but they are different failures, and a signal that separates one does not automatically separate the other. Feature importance by gain, finally, is a statement about what the model used, not about cause.

Questions and answers

What predicts whether a cash-on-delivery parcel will be refused?

Geography predicts cash-on-delivery refusals best: postcode and city were the strongest features in the model described. Order value and the customer's own prior refusal rate, keyed on phone number, were also among the strongest. The features that matter describe where the parcel is going and who ordered it, not what is in the box. The model is a ranking tool, not a verdict on one order.

Why should shipping cost not be used to predict refused parcels?

Shipping cost should not be used to predict refused parcels because it leaks the label. A returned parcel costs more, since the recorded figure includes the return leg, and that cost does not exist when the order is placed. In the article, removing shipping cost and other post-fulfilment fields raised the honest AUC from 0.55 to 0.686.

Should a delivery prediction model use a random or a time-based split?

A delivery prediction model should use a time-based split: train on everything up to a cut-off date and score the weeks after it. A random split lets the model learn from orders placed after the ones it is scoring, and delivery data has enough seasonality and repeat customers for that to matter. If the score drops on a temporal split, the earlier number was never real.

Why does a control group matter when testing address risk rules?

A control group matters when testing address risk rules because any rule that fires often enough produces convincing examples. Comparing each signal against orders the courier did not flag shows that many assumed risks, such as village addresses or missing postcodes, carried little or no signal. The courier integration's own INCOMPLETE and UNKNOWN verdicts turned out to be the strongest address signals.

What should a business do with a high refusal-risk score?

A business should use a high refusal-risk score only for cheap, non-punitive actions such as a confirmation call, an address check or a different packing decision. The refusal model is a ranking tool, not a verdict on one order, and most orders in the riskiest group still arrive and are paid for. Cancelling orders on the score would destroy more than it saves.

All research