Author : Aniket Pathak
Date : 07 July 2023
This is my second article in the three-part series on Product Metrics. If you haven't gone through Part 1, I highly recommend having a quick go-through using the below link. It is essential to understand what exactly is Product Metric before we can look at how to implement it.
Part 1 : http://www.mindthoughtz.com/post/Are-you-using-the-right-metric-to-measure-product-success-part-1
So assuming you know what is Product Metric and how to define it, we can now look at different strategies on how to implement it. If you try searching for product metrics you will get a lot of materials on internet that explain what they are and how to define them, but there is not much about how to implement one. So this is my attempt in explaining some steps I personally feel are vital to implement your product metric, to make it count, and make your product more effective.
So let us get on with it.
How do we start
Consider you are a product manager at a bank (excellent if you are already one) and designed a product to transfer money in fewer steps than other banks. As usual, the objective is to get as many customers as possible to use the product. Most of you have bank apps and in some cases more than one. You like some bank apps and hate others. So even if you are not a product manager but know the product metric (which is essentially to make it easier to transfer money) can you define what will be the metric/s for this product?
Take a pause and Give it a go.......
There can be different metrics, but let's define a few :
How long a user takes to finish the transfer: helps to understand if the app is taking few steps to transfer money or how long it takes for a user to complete
How many users use it more often: If only a few users use it again but the money is getting moved to other bank then it might indicate they may not like the app
What's next
So you define the metric and now wait for the data to arrive. Imagine only you being a product manager is the only one or at the most, the sponsors know the metric, but the rest of the team have no knowledge about it. Other departments think it is another money transfer app or new version of old system . Will that help? some might say, it's okay, each department will do their day to day task and we will just look at data at the end of the day. But what guarantees that data will be collected and more importantly the data you actually required?
Hope you got the idea, where we are going with this, which brings us to the topic of this article: how to implement the product metric.
I'm not sure if you have experienced this, but I have observed on numerous occasions that teams are often unaware of the metrics defined by the product team. Unfortunately, this lack of awareness has led to some unfortunate outcomes. So hope this article gives you some idea of how to approach the implementation.
Basic strategy
Product domain can get technical based on what products you offer and the nature of its ecosystem. hence there are many different strategies to implement the product metrics. in some exceptional cases there are even undissolved strategies that are used for more sensitive products to taste the market and to gather preliminary data. But that will need a completely separate article. Here we will be focusing on more of the most common strategy.
I tried to collate the implementation stages into below basic steps.
Define timeframe
Communicate
Implement
Review
We will go through each step in a little more of detail, to know what exactly we need to do to ensure our metrics are rightly measured to achieve desired success.
Define time frame : It takes its time
This is one of, if not, the most essential step. What is a sufficient timeframe you will give your product to measure your metric? There are various factors involved in this :how tough is the market, time to reach out to sufficient users, how many competitors there are, the time a users takes to get use to the product, etc. These factors will define how much time you allow the product to mature. if you try to measure too early then you will not have sufficient data and if you wait too long, then users might move on to other products.
You have seen many big organizations and start-ups which are not profitable but still into business. If you carefully read the owner's or promotor's justification for this, the most important factor you will notice that they are still waiting for a right time to start measuring their metrics. Some of them still waiting to achieve a certain market share or number of customers. Hence the time factor is extremely critical in defining the review period. if the market is really hard to enter you need to allow sufficient time for the product to show the results. Any pre-mature review will destroy the product's potential.
Defining these timelines is not easy, especially from a commercial point of view. You need to have a robust sustainable infrastructure to support the product till it matures. There could be lot of scenarios that may lead to impatient judgement, but well define timeframe will guide you to the right path. You can define various intermediate milestones but you should always focus on the end goal.
Take the case or Uber or Airbnb, if you read their case study there were a lot of phases where its creator needed to defend the product capabilities due to a lack of profits and commercial success. They may or may not know how much time they need to keep the product running, but with intermediate milestones they keep the check on the path they re heading. You have to wait until the product is matured to identify if it's at par with the defined metric.
Communicate : Does everyone get the metric?
This is most underestimated but the most important step even before you launch a product. You have to understand that defining the product metric is just a start, you have to make sure everyone concerned is aware of the metric.
We set off by using a bank application but to change our perspective lets take another example. Now you are a product manager defining a new handset for a mobile company. You have designed a very fast handset used for apps which need specific high speed. During design process you may have to compromise on camera or other factors as you have to add component which improves speed.
Lets see if you don't inform your team about the product goals -
As you know there are various teams and departments involved in launching, implementing, servicing, and supporting the product. It goes through several processes daily. All these need to be aligned with your product metric, only then it will achieve what is defined in the metric and to do so you first need to inform all stakeholders about what is the metric defined for that product. Only when you communicate everyone will understand what is the core objective.
Communication is not only about informing the teams, you have to organize communication channels, training, Q&A session, feedback mechanism, etc. The whole purpose is to make sure the whole team is aligned with the product objective and metric. At each stage the team needs to know what they are doing and most importantly why they are doing it. The data that you highly desired to review your metric will be created by these teams, hence it is important that they understand the objectivity behind it.
Implement : Defined individual metric
You have defined timelines, you communicated to the team, now what. The question is, isn't communication enough? Once we have informed everyone what we want to achieve, then we should be good to go. but are we?
Lets take example of our high speed mobile handset :
: Our goal is :
To be one of the fastest handset
Now let us say product manager informed everyone about this and now its time to go into the market and start rolling out the product. You may be in sales, services, marketing, support, or other related department. Is just knowing the goal enough for you or do you need precise direction on how to align with that goal and what are your own metrics. In other words what data points do you need to collect at your own junction to check if we have achieved this metric or not?
If we dissect this further and try to define it at each department level, some of the questions may pop-up are
Now as a product manager, you would have worked this out, and if not already, you may have to work this out in the design phase itself. Once we communicate the goals and metric at a high level we have to break it down at individual process levels and defined more precise metrics for each department. The whole purpose of defining a product metric is to able to measure it and we can only measure it if we collect the data. Only then we will be able to measure the data which we will further discuss in the review section.
Break down the metric into objectives
You need to think of all the departments and processes involved in the success of the product. You have various departments like sales, operations, logistics, support, and many more, which are essentials to keep the product rolling. You need to make sure each department performs its task around the product metric and collect specific data. Hence you must integrate your metric into these processes.
Review : Time to find out
You have done all the hard work and now it's time to find out how the product has turned out to be. The whole objective here is to check if the Product has achieved all the objectives defined in the metric. This is the stage where you observe data and publish the results.
Continuing with our example of a mobile handset, we can analyze our metrics listed below:
From all the data you have collected, you can create graphs and charts to confirm the results of yours. Let's analyze our journey so far -
Product objective: Mobile handset made for speed.
Some of the metrics and data associated with it:
From Sales :
What is the primary reason user bought phones: This will help us understand if users choose this handset as they were looking for high speed.
Did the user enquire about this handset: This will help to understand if the marketing campaign were successful in promoting high-speed handset.
From Services :
What is the most used app on the handset : This will help us understand what applications are used most on the handset and are those require high speed.
Are users using the handset for games: We are targeting the gaming community, so we need to check if the handset is mostly used for gaming.
From Support :
How user queries about high speed: this will help us understand that users are looking for high speed and if the current speed matches their requirements.
How many users raised issues related to gaming: This will help us analyze how many users are using the handset for gaming and if any specific category of games are having issues that need different levels of speed.
With all this data and information you can conclude the product performance and check if it is heading in the right direction and the right set of users are using the handset for the desired purpose.
If the product is not utilized for the desired purpose (gaming or high speed, in this case) then it is very likely that users do not recognize the product's true potential and it is under-used affecting the product performance and ultimately the monetary benefits.
Based on this data you can further enhance the product and plan the next improvements in its respective area: If the marketing need to improve? OR if the sales team needs more training? Do we need improvement in speed? Is the handset up to the standard for the new apps and games getting developed in future?
Conclusion
Hope you did get an idea of how the product metric needs to be communicated and implemented in a very strategic way. Defining the product metric is just a start, implementing it in the whole ecosystem is the real challenge. You have to make sure the whole team is working towards the metric. Defining the metric in isolation will never lead to success its a teamwork and not individual success.
Relevant links
Comments