Much has been said on this blog and elsewhere about the evils of peeking. Many experimenters, however, violate this assumption, peeking at the stats while the test is running and abandoning tests early when results are convincing-introducing unquantified error into their decision-making. The classic breed of A/B testing methods-known as fixed-horizon tests-assumes that a specific sample size has been committed to in advance the statistics rely on this assumption for validity. Sacrificing relationship capital with stakeholders eager to make a decision.Delaying the monetization of a valuable change.Prolonging the exposure of a poor experience.Forgoing the launch of other experiments.Time is a cost of running a test, but, in practice, it also represents the cost of continuing a test: In my experience, this is a dilemma almost every experimenter faces at one point or another-regardless of company size or traffic count. If your arm is waving vigorously in the air right now, this post is for you. Raise your hand if you’ve ever struggled with a decision between disciplined testing procedures and expedient decision-making.įor example, think of a time when you’ve had to decide between sticking to your A/B test design-namely, the prescribed sample size-and making a decision using what appears to be obvious, or at least very telling, test data.
0 kommentar(er)
