A winning variant can still be the wrong ship

Product teams ask us to bless a ship because the primary metric crossed a line. Sometimes the honest readout is that the line was crossed and the feature still should not go to everyone.

The feature the control never saw

If variant B only wins because control was a broken checkout, you have learned that broken checkout is costly. You have not learned that variant B is the product you want to maintain. We have written that sentence in Johor Bahru for a squad that had already booked a release train.

Assignment bugs sit in the same family. A “win” among users who were easier to assign is not a win among the users you will meet after the flag opens.

Effects that will not survive the calendar

A two-week window that covers a payday, a 11.11 campaign, or the week before Hari Raya can move payment features for reasons that will not be there in ordinary March. The readout should say whether the window was typical. If nobody can say, the recommendation should be iterate, with a second window booked.

Holdouts exist for features whose effect arrives late: reminders, pause offers, credit-limit copy. Shipping at the end of the short test because the table is green is a choice. It should be named as a choice, not as an inevitability of the threshold.

Cost the metric does not see

Support load, app-size, and a design that only works on high-end phones in the Klang Valley can all stay invisible to a primary conversion rate. Guardrails are supposed to catch some of this. When they were never defined, the readout cannot invent them, but it can refuse to speak as if they were checked.

A mild version of the same problem: the winning variant uses urgency copy the brand will not want on the home screen in six months. That is not a statistics problem. It still belongs in the briefing, because the product manager will be asked about it the day after ship.

Request a readout if this is the conversation your squad is already having.