2023 · 6 weeks·Data Scientist
Reading a pricing experiment that looked flat
A headline-flat A/B test hid a strong segment effect. The real story was in the cuts.
+9%
Hidden lift
SMB
Segment
Ship
Decision
Context
A pricing test showed no overall movement and was about to be killed. I was asked for a second read before the team walked away.
Approach
- 01Re-checked power and guardrail metrics before touching any segment cuts.
- 02Decomposed the flat aggregate into pre-registered segments to avoid p-hacking.
- 03Surfaced a clean, robust lift in one segment and a small drag in another.
- 04Told the story as a trade-off the team could decide on, not a single verdict.
Impact
The experiment shipped for the winning segment, capturing a 9% lift that the aggregate had completely masked.
A/B testingSegmentationStorytelling