Common Mistakes Teams Make when Using Product Metrics
Many organizations and teams measure and maintain many metrics but still struggle to measure and understand the value their products deliver, which is necessary to validate whether they are advancing toward their product’s strategic goals. This typically leads them to measure and focus on things like velocity, releases, and number of clicks, or rely solely on profit to gauge performance. These approaches lead to consistent poor decision making.
Here are common mistakes to avoid that organizations and Scrum Teams often make when measuring the success and value of their products and what to try instead.
Trying to measure too much. There is an abundance of information available and many teams often fall into the habit of collecting and maintaining as much information as they can, which becomes counterproductive. In the end, they don’t know what information is useful because everything might be helpful.
What to try instead. It’s important to deduce what information is worth paying attention to at the moment. To see if a piece of information might be useful to measure, some questions to ask and answer are:
What questions do I need to answer now?
Does this information give me insight into business impact?
Does this information tell me about progress toward goals?
How does this information help us know if we are delivering value to our customers?
Can I make an informed decision based on this information?Focusing on vanity metrics. Vanity metrics are metrics that look good on the surface but do not translate into anything meaningful for the team to take action. Vanity metrics may look impressive but they do not inform on progress toward goals. For example, 50,000 total registered accounts in a month might seem superb, but is a weak value indicator if there are only 100 active users for a product.
What to try instead. The metrics you use will never be perfect and should evolve. To best understand customer needs and if your product is meeting those needs, a team should monitor and interact with customers as closely or directly as they can. There are a variety of ways to do this, such as:
Measuring product usage to see how people use the product and their functionalities
Speaking with customers and users directly
Observing people interacting with the product - through prototyping or directly in the fieldConfirmation bias. People tend to process information by looking for, or interpreting, information that is consistent with their existing beliefs.* Although confirmation bias is typically unintentional, it leads individuals to disregard information that contradicts with what could be true. Consider two colleagues working on a marketing campaign. The first colleague wanted the campaign to “work,” so they found user data and information to support that it did. The second colleague thought the campaign was a waste of time, so they found user data and information to prove that it failed.
What to try instead. Biases are hard to break especially when people are unaware that they have them. Presenting objective data will unlikely help someone with confirmation bias make objective decisions and could lead to agitated reactions. To try to counter confirmation bias, have conversations about outcome-driven goals, define them clearly and then choose ways to measure them. Run experiments toward those goals. Create a holistic view of metrics that don’t hone in on a few specific metrics, especially vanity metrics.
All humans have a degree of confirmation bias. That’s why seeking and receiving feedback from stakeholders and others in the Scrum Team is a good way to challenge your own biases. Be transparent about your ideas, the goals you are working on and the ways you want to measure progress toward them.Using activities and outputs as value measurements. Activities (things people do, like writing code or attending meetings) and outputs (things people produce, like work completed, velocity or product releases) are not good measures of value. They are easy to measure, and often get misused as delivery goals, which can create an environment of micromanagement that pushes the team to work faster without actual customer feedback.
What to try instead. Activity and output metrics inform capacity, not value. In addition to knowing the impact their product delivers to the business, Scrum Teams should measure the value of the product Increment. Until the Scrum Team actually delivers a product Increment and measures the result on customer outcomes, they can’t be sure that they really understand what customers would like to achieve from the product (outcomes) or what their “satisfaction gap” is - the gap between the outcomes that customers currently experience and the outcomes that they would like to experience.
Outcomes are harder to measure, but ultimately they determine how valuable a product is to a customer and hence the organization and give better insight into their behavior and what they might find valuable.
Focusing on Key Performance Indicators (KPIs) on set targets instead of goals. The purpose of KPIs is for an organization to know when a set of targets have been met in a certain amount of time. When organizations and teams focus on KPIs, they risk prioritizing short-term achievements and overlooking goals. They also miss areas for improvement because KPIs are condensed metrics focused on performance and activities. They are not measuring whether experiments are helping the team progress toward greater goals and strategies. Focusing solely on KPIs as goals to be achieved leads to output-driven delivery without understanding if those outputs deliver a better outcome for customers. They can also lead to undesirable behavior and loss of morale when teams are rewarded or punished by meeting KPI targets.
What to try instead. Teams should create short (Sprint/Tactical), medium (Product-Intermediate), and long-term (Strategic) goals that will tell them if their product is contributing to the overall strategy of the organization. The goals should be specific and communicate the “why” while the teams create experiments that will then be measured to validate them. These types of goals keep track of customer outcomes and create better buy-in and alignment throughout a team and organization. The results of these goals are not positive or negative, but are purely information about the experiments that the Scrum Team can then discuss with each other and stakeholders as they seek to improve. They can do this at least during the Sprint Review and Product Backlog refinement.
Using Outdated Metrics. Outdated metrics are ones that do not inform the current goals a team has. Scrum Teams cannot rely on them to make informed decisions. Relying on old metrics can cause an organization to miss market shifts, trends, and opportunities causing them to invest poorly in their product(s) and future.
What to try instead. Keep your product’s evolution in mind. As your product moves back and forth in its lifecycle, the goals and metrics you have for that product will change. Be sure to visit and adjust them accordingly and discuss with the team what makes a metric useful for them. Some questions to discuss:
Where is the product in the product life cycle?
Do our metrics reflect the team's current goals and priorities?
Does this metric tell me information I want to answer now about the product?
Does this metric inform us about the future potential value of the product?
It is easy for Scrum Teams to fall into the trap of measuring everything or nothing, so they lack the ability to determine reality and make better-informed decisions when working to deliver valuable outcomes to customers. Using a consistent and outcome-driven approach to measuring value is essential to improve the product for customers and also creates shared understanding of goals within the organization. It also allows the team to have more aligned and engaging discussions that are evidence-based about the current state of the product and future opportunities with stakeholders.
*Definition from Encyclopedia Britanica
Resources:
Unlocking Business Agility with Evidence-Based Management: Satisfy Customers and Improve Organizational Effectiveness (The Professional Scrum Series) by Patricia Kong, Todd Miller, Kurt Bittner, and Ryan Ripley; Addison-Wesley Professional, October 31, 2023. ISBN-13: 978-0138244576