Why measure something? Why don't we just develop based on intuition?
Metrics help us in various ways. They help us
- to identify what problems to solve
- to prioritize the problems based on the reach and impact
- to validate if a problem is actually a problem
- to understand why it is a problem
- to validate if a certain fix or feature actually solved the problem
How do we measure?
Two key ways of measuring metrics are through user interviews and data analytics.
The below table explains when to prefer which approach.
What to measure?
Metrics should help us make better business decisions. We should not measure something just because we can, or just to show a good dashboard. It is important to ensure that every metric is actionable. "We should avoid vanity metrics and focus on actional metrics."
North Star Metric
North Star metric is a metric that is most predictive of the company's long-term success. The entire company/team should be able to understand what they need to build through the North Star metric. It should focus on the primary or the core value of the product.
Key Characteristics of a North Star metric:
- It should capture the core value proposition delivered to the user
- It should be predictive of company success
- It should be simple (Everyone in your company/team should be able to internalize and repeat it.)
The core value for the users is different for different industries. Here are a few examples.
The core job of the product owner is to make sure that the users are experiencing the core value of the product.
Metric Types:
Metrics should address the full customer life cycle.
Reach: Total User base that can be engaged.
Total number of people who have used the product in a recent time period or the maximum number of users who could reasonably be activated.
Examples:
- Total registered accounts
- Paid monthly users
- Market share (eg.: % of fortune 1000 as customers)
Activation: Rate at which new users convert to activated users.
The activation experience for a user is absolutely critical. If this experience is good, the chances of converting to an active user drastically increases. It is a foundational step that primes a new user to become an active user.
Active Users: Number of users that experience the product's core value at a natural usage frequency*.
It is the count of % of users that have taken a key action and received value from your product within a recent time period.
Examples:
- Monthly active rides (Uber)
- Daily Active People (Instagram)
- % of members that watch 2+ hours per week (Youtube)
Engagement: Depth, breadth, and frequency of usage. Not all users are created equal.
It accounts for depth (more intense usage), breadth (more varied features), and frequency (more number of days) of completing key actions.
Examples:
- Number of videos watched
- Messages sent
- Chat sessions with > 20 messages
- Number of orders
- Number of days active (in previous 7 days)
- Cart conversion rate
Retention: Rate at which users are still using the product days/weeks later.
Retention is another critical metric. It is the rate at which users come back to use a product within a certain time period.
Examples:
- Day-14 retention of watching videos
- Week-4 retention of riders
Revenue: Total money made by the business
Examples:
- ARR (Annual Recurring Revenue)/ MRR(Monthly Recurring Revenue)
- Ads revenue
- Affiliate revenue
- Transaction volume
Business-specific: Customer satisfaction, Costs, Efficiency of the product, etc.
Examples:
- Referral rate
- Data center burn
- Customer Survey results
*Natural Usage Frequency: The frequency at which we expect users to use our product. It depends on the type of product we refer to. For an email app like Gmail, it could be hourly, for a social networking platform - daily, for a cab booking app - weekly or monthly, for a tax filing app - yearly.
Proxy Metrics:
What percentage of the target audience displays a certain behavior which can be indicative of their long-term retention. This can become an indicating metric of the product's success or failure in the future.
Counter Metrics:
It is important to think about if our metrics would have any unintended impacts. We need to set some metrics to monitor these. Counter metrics are those checks that are set in place to avoid or minimize unintended impacts of our metrics.
Ask yourself:
- If I am fixated on the north star metric only, how can I unintentionally end up hurting the user experience and the business?
- What checks do I need to add to avoid such occurrences?
Examples:
Common Pitfalls when coming up with metrics:
- Avoid metrics that are not tied to real customer or business value
- Avoid too many metrics, which can cause analysis paralysis
- Avoid wrong metrics.
- Wrong metrics can lead to misaligned incentives which can lead to wrong outcomes. Eg.: For example, if we incentivize engineers by the number of bugs fixed, it can lead to engineers being more lenient during development which can leave room for more bugs in the application, which increases the testing cost and effort and can even have a negative impact on the customer experience.
- Wrong metrics can lead to missed opportunities which can lead to suboptimal outcomes.
In summary, metrics and analytics are essential to make better product decisions. Data brings clarity and helps us understand our customers better. They are not a supplement for user interviews, instead, they complement them. Metrics help us understand if the users are experiencing the value we think they should be experiencing. Analytics can be used to understand which users we need to talk to and help us make better product decisions. Metrics can help us build a confident story to get a buy-in from the top management and other stakeholders to make better product decisions.