← All case studies
2023 · 6 weeks·Data Scientist

Reading a pricing experiment that looked flat

A headline-flat A/B test hid a strong segment effect. The real story was in the cuts.

+9%
Hidden lift
SMB
Segment
Ship
Decision

Context

A pricing test showed no overall movement and was about to be killed. I was asked for a second read before the team walked away.

Approach

  1. 01Re-checked power and guardrail metrics before touching any segment cuts.
  2. 02Decomposed the flat aggregate into pre-registered segments to avoid p-hacking.
  3. 03Surfaced a clean, robust lift in one segment and a small drag in another.
  4. 04Told the story as a trade-off the team could decide on, not a single verdict.

Impact

The experiment shipped for the winning segment, capturing a 9% lift that the aggregate had completely masked.

A/B testingSegmentationStorytelling