Readouts / Holdout and delayed-effect reading
When two weeks is not the story
Holdout and delayed-effect reading
An interpretation of holdout groups and longer windows when a short experiment period is likely to mislead.
A two-week experiment on a savings reminder or a subscription pause can look finished while the real effect is still arriving. Holdouts exist for that delay. They only help if someone kept them.
We have read holdouts that were accidentally included in a later “all users” copy change. The readout in those cases is short and blunt. A contaminated holdout is not a small caveat; it is the end of the claim.
This engagement suits product teams who already ran the short test and now have a later snapshot they are afraid to open.
Included
- A judgement on whether the holdout still means what the design said
- Reading of delayed effects the original window could not show
- Language for leadership when the short test and the long window disagree
Not included
- Creating a holdout after the fact
- Survey design
- Lifetime-value models sold as a product
How the work proceeds
-
Integrity
We check whether the holdout stayed a holdout: flag leaks, support overrides, and marketing lists.
-
Windows
We read the original window and the later snapshots as separate claims, then as a pair.
-
Brief
The memo states which window should govern the ship decision and why.
Preparation
Holdout definition, dates it was intact, later snapshots of the same users, and any contamination you already know about.
Limits
If the holdout was quietly given the feature, we can say the reading is unsafe. We cannot reconstruct a holdout that was never kept.
Next step
Describe the holdout in the contact form before attaching later snapshots.