Josh Elman has some wise words for product managers that extend into the realm of analytics. This pithy quote says a lot about the struggle product managers find themselves in when it comes to making data-informed decisions and sticking to a time-bound roadmap:
Great product managers understand the very fine balance between getting it right and getting it out of the door
With that in mind…
As a product manager you’re responsible for driving the direction of product development. You translate the company’s (and your own) vision into the product. At the same time the product needs to deliver exceptional user experience and meet hard business goals like revenue and profit. Elman reckons the best way to do this is by measuring results - results of metrics and experiments that you, as the product manager, set up to generate data and drive direction.
Here’s what he suggests any product manager looking to become led by data does:
- Have a theory of the impact you want to have [on the product and business]
- Identify metrics to demonstrate that impact (be sure they’re clarity metrics)
- Generate data of what works [to drive growth and retention] and what doesn’t
- Keep an eye out for unexpected results.
But measuring data isn’t enough. You need to be systematic.
A data science methodology for product
Elman talks about adopting a data science methodology for product analytics and data. The idea is to have a clear framework in which to rigorously yet efficiently create data to help you find answers to your questions. The structure is something like this:
The above make up the “data science methodology” for product data analysis. Here’s an example of how it might apply to a SaaS product:
- Problem: many people sign up for a free trial but only <5% convert to paid accounts
- Hypothesis: people aren’t using the product enough during the free trial to see the value
- Experiment: extend the trial period by 14 days
- Analysis: did >5% convert to paid after an extended trial period?
- Theory: If yes to above, then a longer trial period helps customers get to know the product. If no to above, then retest the hypothesis with another experiment and/or develop and test a new hypothesis.
Having a clear, hierarchical framework in which to frame your product questions enables you to design and analyse analytics experiments that provide actionable data. By quickly designing and deploying experiments you can walk the fine balance between getting it right and getting it out the door. Having a user-friendly product analytics tool helps, of course.
Efficient, structured, and insightful product data
Generating data is easy but it’s a lot harder to consistently create data that helps you move product growth forward in a positive direction. By applying Elman’s ideas of measuring results using a data science framework it becomes easier to consistently deliver.