Product Metrics

Product Metrics

Having the right set of product metrics is key to tracking and improving product performance and driving growth. However, there is a myriad of metrics product teams could be tracking and they are continuously evolving throughout the life cycle of the product. So, you could easily get overwhelmed or confused about which is the right set of metrics to pursue.

In this article we want to help you understand the basics of product metrics and make sure you don’t get confused by the different buzz words on this topic. So, we will see a general introduction to metrics, then explain the difference between One Metric That Matters (OMTM) and North Star Metric (NSM) and we will finish with an introduction to Metrics Framework.

This is an introduction to the huge world of product metrics, if you want more please comment and we will be adding more content. Like, how to define your North Star Metric, how to create your metrics framework or how to organize your product organization around a North Star Metric.

Introduction to Metrics

We measure to reduce uncertainty about a decision we must make. If metrics are not helping us make better and faster decisions then we should measure something else. For instance:

  • What should be the focus of the team for next quarter?
  • What should we test first?
  • How can we improve activation?
  • How could we reduce churn?
  • Shall we pivot?

Metrics also serve to alert us about parts of our product or business model that require immediate attention.

Leading and Lagging Indicators

There exist basically two types of metrics: lagging indicators and leading indicators. Also known as output metrics and input metrics.

  • Leading indicators predict what will happen to our business and should be used to track progress.
  • Lagging indicators tell us what happened and should be used to demonstrate success.

Output metrics or lagging indicators cannot be directly impacted. They change as a consequence of acting upon input metrics or leading indicators. That’s the reason we need to understand our business model in order to define an adequate metrics framework. Because, in essence, any business model can be expressed as an equation or relationship between several metrics.

Categories of Metrics

I was saying at the beginning that there is a myriad of metrics product teams could be tracking. Well, I don’t want to scare you, but that’s true.

There are five things product teams should be really good at measuring:

  • Customer Experience – these are an overall expression of how customers perceive your brand or product. Metrics such as NPS, Customer Satisfaction Score fit here. They answer to the question: How much do customers like our product?
  • Customer (Job) Performance – if we did our product discovery well, we identified several outcomes our customers value over anything else when using our product. We need to know customers’ expectations when using our product and how well we are addressing those expectations. When we buy a product or use a service we expect something to be done better, faster, with more production and/or cheaper. What are those things customers want to do better, faster or with more production? That’s the key question we must answer during product discovery and we must later on track performance. They answer to the question: How well is our product solving customers’ problems?
  • Business (Model) Performance – Revenue, Cost of Sales, Cost of Customer Acquisition, Operational Costs, and so on. They answer to the question: Is our business sustainable, healthy and scalable? Are we just surviving? Are we growing fast?
  • Process Performance – metrics to improve process performance like cycle time, lead time, WIP, defect rate, flow efficiency, and so on. They answer to the question: How good are we working?
  • Product Performance – metrics that the product team can impact typically around the AARRR funnel. Acquisition, Activation, Retention, Revenue and Referral. – They answer to the question: How well is our product ecosystem working?

In this article we are going to be talking about Product Metrics in a wide sense, because it is often difficult and a waste of time trying to classify and separate the different types of metrics. The important thing is that we know at all times what is our focus and what metrics we need for that. For instance, if we see an increase in churn we must analyze all related metrics and see which one might be causing the problem. It could be price, checkout experience, page load time or slowness in solving customer problems or sluggish product improvement.

Additionally, metrics should not only help us solve short-term issues, like increase in Churn, but also provide a long-term direction, team alignment and focus. So, we also need metrics to drive growth.

The first one short-term improvement metric is what is commonly known as The One Metric That Matters (OMTM) and the second one long-term growth metric is what is usually referred to as North Star Metric (NSM).

OMTM vs NSM Product Metrics

North Star Metric (NSM) and The One Metric That Matters (OMTM) are often confused or used interchangeably.

I first heard the concept of OMTM when I read the book Lean Analytics, and frankly I don’t see a difference in their definition and explanation with North Star Metric (NSM).

However, there seems to be a tendency in the growth community and Lean Startup experts to acknowledge that NSM is more of a long-term strategic metric, and OMTM is the one metric that you should be caring about anything else right now because there is a fire in your pants. But, if you search in Google you will see different interpretations and arguments on the topic.

A North Star Metric is the key measure of success for the product. It is the single metric that best captures the core value that your product delivers to customers. They are key lagging indicators that measure success for the product. Revenue or Average Order Price are not good NSM.

A NSM defines the relationship between the customer problems that the product team is trying to solve and the revenue that the business aims to generate by doing so.

Some people get confused between OMTM and North Star Metric, but there is a subtle yet important difference. OMTM is the “one metric that matters right now” and it is more of a tactical metric for fixing a short-term problem, whilst a North Star metric is a strategic guide.

Both, NSM and OMTM change over time. NSM typically changes as your product evolves and matures in response to strategic changes, and OMTM changes based on what are the levers you can tweak in the short term to impact your NSM.

Monthly Active Users (MAU) could be your NSM if you were LinkedIn or Facebook, but not if you were eBay, AirBnB or Amazon. If you were Deliveroo the NSM could be Monthly Orders Delivered On Time.

The OMTM could be any input metric that might help you improve your NSM. And, it would change depending on the team and the performance of the product. It could be Card Abandonment Rate, Activation Rate or Churn.

Let’s see now those in more detail and we will finalize the article with an introduction to a product metrics framework.

North Star Metric

North Star Metric - Lean Product Management - AKTIA Solutions

The North Star Metric (NSM) is a single measure that a company uses to provide focus and alignment to their growth efforts. It is the metric that most accurately captures the core value you create for your customers.

In his book Hacking Growth, Sean Ellis says:

To determine what that is you must ask yourself: Which of the variables in your growth equation best represents the delivery of that must-have experience you identified for your product?

It defines the relationship between the customer problems that the product team is trying to solve and the revenue that the business aims to generate by doing so.

The NSM Metric is a lagging indicator which results from the achievement of the results expected by our clients when doing a job.

Every company should experiment and find their own North Star. There is no silver bullet, you need to know very well your business and experiment to find the right NSM to drive growth.

Examples of North Star Metrics

  • AirBnB – nights booked
  • Facebook – daily active users (in the early Facebook the NSM was Number of Users Adding 7 or More Friends in the First 10 Days)
  • Uber – complete rides
  • Tinder – number of dates
  • Mint– percentage of users with increased balance in 3 months
  • Spotify – time spent listening to music
  • Deliveroo – monthly orders delivered on time
  • eBay – number of items sold

Attributes of a good North Star Metric

A good NSM should pass the following checklist:

  1. It is related to the value we bring to our customers.
  2. It is a leading indicator of success. The purpose of a NSM is to drive growth, so improving the NSM should drive growth.
  3. It aligns the whole product organization. So, for instance, if you have acquisition, activation and retention teams all of them should be able to influence the NSM and do not jeopardize each other’s efforts. If one of your pirate AARRR metrics goes up, your NSM should also rise.
  4. It’s actionable. We can make something to influence it. That’s the reason we need a metrics framework where we can see which input metrics influence the NSM, which is usually an output metric
  5. It’s understandable. It’s expressed in terms that everybody in the company can understand and can relate their work to.

The One Metric That Matters

The OMTM is that metric a team is focusing on during a period of time in order to benefit the overall company-wide NSM.

It is a tactical short-term metric that is usually related to a problem in your metrics framework or to an improvement you want to make in some of the input metrics that influence the NSM.

In this article we explain how to use the OMTM to improve product performance.

Essentially, before achieving Product-Market Fit you will be iterating your OMTM, because there is no point in driving growth with a NSM until you have a good product that is compelling to your target market. As you get closer to Product-Market Fit and start pushing for acquisition and activation then you will have to set the north with and NSM and every team will be working on different OMTMs.

Product Metrics Framework

North Star Metric - Lean Product Management - AKTIA Solutions

Now we have to put everything together. The way many product teams do that is by defining their metrics framework – A constellation of metrics related to the NSM. There are many different ways in which you can do this.

In the book, Lean Analytics, they provide different analytics frameworks for different types of businesses.

Amplitude uses a method to help define the input metrics that influence the NSM, by focusing on four dimensions: breath, depth, frequency and efficiency:

  • Breadth: number of customers doing something
  • Depth: how much of that they are doing
  • Frequency: number of times per period of time they do that
  • Efficiency: how well they can do that

The image above could be an example for Dropbox of Amplitude’s method for defining a metrics framework, where you can also see some of the initiatives (gray) you could do to improve metrics.

You also also choose to put the NSM on top, create a second level with the AARRR metrics (or similar) and a third level with the input metrics that would influence that second level.

Metrics Framework - Online Retail - Product Metrics - Lean Product Management - AKTIA Solutions

In this case, typically OMTM would be on the second or third level.

It doesn’t really matter which method you use, as long as you can synthesize your product equation in a metrics framework that you can use to drive growth.

Product Metrics and OKRs

When I talk about product metrics people often ask if that’s not the same as OKRs.

It is not. OKRs is a methodology for continuous improvement that drives alignment, focus and decision making. You can work with OKRs alone, with a metrics framework alone or combine both.

One part of the OKRs methodology is to define goals and key results, and if you have your metrics framework in place, it will be much easier. However, OKRs span the whole organization, not just the product, so you could have OKRs that are not directly related to the product. You could have OKRs about process, about customer experience or about employee experience.

Let’s put an example for clarification.

Imagine a soccer team. Which aims to win the League, the Champions League and the Cup. Those would be the three objectives, the O of the OKRs.

What would be the KRs? How will we know that we have achieved the objectives according to our success criteria?

  • 100M in advertising revenue
  • We have scored in all matches
  • We have conceded on average 0.5 goals per game
  • Possession of the ball> = 60%
  • Average stadium attendance> = 95%

Thanks to OKRs we can know which are the most important metrics at all times. In this case, if we see that we are not winning matches we can analyze the keys and see what metrics must be improved. For example, conceded goals, fouls committed or injuries.

If we see that people do not attend the stadiums we can analyze the causes and make a breakdown of metrics: nice game, ticket prices, …

In some cases a KR will match your North Star, some other times it won’t, or it will match your OMTM.

For small or growing companies, the metrics framework will typically match OKRs. For example, our objective for this quarter is to increase orders delivered on time, the KR will be the target for the metric. Then, as we have seen, we will write down that North Star metric that is a lagging indicator and its input metrics or leading indicators:

  • # of active users
  • # of items per order