To test or not to test
This post was originally published on my previous site, kohlio.com, on June 15, 2020. This may or may not have been edited slightly by AI.
Some things are worth testing; some things are not.
It’s easy for “Test Everything” to mix and taste like Kool Aid. There’s this well-intentioned willingness to just drink it up. And wikipedia’s entry on “dogma” is good reading: “Dogma in the broad sense is any belief held unquestioningly and with undefended certainty…” And more.
Reasons to test:
- To learn, above any predetermined wish to ‘validate’ something you just want to be true
- To gather real data and be open to any outcome
Reasons not to test:
- Somebody high up said ‘just do this’, but, like, validate it too
- Resources need prioritization
- A primary metric seems tied to business objectives, but isn’t on point
- Flawed methodology
- Insufficient timelines, traffic segments, conversions
Nobody wants to admit it when business or zealous ideology interferes with your perfect testing programs. But they can. And the decision to base all decisions on data becomes a tagline, rather than a practice.
Sometimes it’s worth admitting, “This particular thing just isn’t working and the best optimization here is to kill it.” Other times, perfection is beautiful; the learning is crisp and clear; the results deliver improvement, but also raise more questions that excite people and inspire further exploration and learning together.