A/B testing experiment design and analysis

Design statistically valid A/B tests for product features. Pre-test setup: 1. Define hypothesis clearly (adding reviews will increase conversion by 15%). 2. Choose primary metric (conversion rate, not multiple metrics to avoid false positives). 3. Calculate sample size: use online calculators, typically need 1000+ conversions per variant for significance. 4. Set test duration: run for full business cycles (include weekends), minimum 1-2 weeks. 5. Define success/failure criteria upfront. Implementation: 50/50 random split, ensure consistent user experience across sessions. Analysis: statistical significance (p<0.05), confidence intervals, practical significance (is 2% lift worth engineer time?). Avoid peeking at results mid-test. Tools: Optimizely, Google Optimize, VWO, internal feature flags. Document learnings for future tests.