Readouts / Pre-launch experiment design review
Before traffic is assigned
Pre-launch experiment design review
A written review of hypothesis, metrics, sample size, assignment, and stop rules before the feature experiment goes live.
Most painful readouts start as painful designs. A checkout experiment that measures page views, a four-variant test with traffic for two, a stop rule that is really “when the VP looks.” The design review is the cheaper conversation.
We read the brief as if we will have to write the later memo. If a metric cannot be explained in one sentence to a designer who did not write it, it will not survive the readout either.
Squads in Johor often share traffic with a KL campaign calendar. We ask about that before we bless a two-week window that falls on Hari Raya or a 11.11 sale.
Included
- A pass on whether the primary metric can actually answer the hypothesis
- Comments on guardrails that would change a ship decision
- A note on sample size given your usual weekly traffic, not a generic calculator printout
- Stop-rule language the squad can paste into the experiment record
Not included
- Writing the feature itself
- Legal or privacy review of tracking
- A readout after the experiment unless booked separately
How the work proceeds
-
Read the brief
We mark where the hypothesis, the variant, and the metric disagree with one another.
-
Traffic honesty
We compare the planned sample with the traffic you actually have in a normal week, including festive slowdowns in Malaysia.
-
Return
You get the annotated note and a short call. Changes to the live experiment remain yours to make.
Preparation
Send the hypothesis, screens or copy for each variant, metric list, planned sample, and the rule you will use to stop.
Limits
We review the design. We do not recruit users or set the flag.
Next step
Attach the draft experiment brief in the contact form before you assign the first user.