Peeking, festive weeks, and the sentence the squad needs to hear

Looking at an experiment before the planned stop is a habit, not a moral failing. The failing is to treat the early look as if it never happened when the final sentence is written.

Say the look out loud

If someone checked the table on day five because a director asked, the readout should include that fact. Early looks change how surprising the final number is. They also change behaviour: squads stop experiments that look sad and extend ones that look happy.

We do not need a confession booth. We need one honest line in the memo so the launch review is not later ambushed by a Slack screenshot from week one.

Malaysian weeks are not interchangeable

A feature experiment that runs through Chinese New Year, Hari Raya Aidilfitri, or Deepavali in a payments or commerce product is not the same experiment as one that runs in a quiet October. Traffic mix shifts. Family-group behaviour shifts. Campaign banners appear that were not in the design.

The sentence the squad needs is sometimes: the number is real for that window, and that window is not the year. Iterate means a second window, not a quieter tone of voice in the same memo.

Stopping early

Harm that shows up in a guardrail can justify an early stop. Boredom cannot. Neither can a director’s flight schedule. If you stop early for harm, the readout should describe the harm with counts, then recommend whether the flag stays down.

If you stop early because the primary metric looked good, write that you stopped early because the primary metric looked good. Do not write that the experiment “concluded as designed.” The next reader, six months later, deserves the plainer sentence.

Request a readout if this is the conversation your squad is already having.