Live Experiment Results Report
Live Experiment Results Report
SAMPLE subject: landing page test for a meal-planning app
SAMPLELive Experiment Results Report
Use ← → keys to navigate
Research behind the method / SKPTST
Current methodological context. These references are not evidence for this historical run.
SKPTST: planned methodology. The engine and paid traffic acquisition are not active.
Plan allocation and sample size before testing
Kohavi, R.; Longbotham, R.; Sommerfield, D.; Henne, R. M. (2009). Controlled experiments on the web: survey and practical guide. Data Mining and Knowledge Discovery, 18, 140–181. Online: 30 July 2008; issue: February 2009.
- Research finding
- Controlled experiments use randomization, statistical power and variance considerations.
Define stopping rules before inspecting outcomes
Kohavi, R.; Deng, A.; Vermeer, L. (2022). A/B Testing Intuition Busters: Common Misunderstandings in Online Controlled Experiments. Proceedings of ACM SIGKDD 2022, 3168–3177. Conference: 14–18 August 2022.
- Research finding
- Opportunistic stopping and post-hoc analysis can undermine A/B-test interpretation.
Separate a traffic pilot from proof of advertising impact
Lewis, R. A.; Rao, J. M. (2015). The Unfavorable Economics of Measuring the Returns to Advertising. The Quarterly Journal of Economics, 130(4), 1941–1973. Issue: November 2015. · Blake, T.; Nosko, C.; Tadelis, S. (2015). Consumer Heterogeneity and Paid Search Effectiveness: A Large-Scale Field Experiment. Econometrica, 83(1), 155–174. Issue: January 2015.
- Research finding
- Advertising returns can be difficult to estimate; the eBay experiment found effects differed across user groups.
The SKP suite
Observe real behavior
Test behavior, not opinions.
This report