February 26, 2021

How Can you Measure the Impact of your Scrum Masters?

Have you been wondering how you can measure the impact of your Scrum Masters? And how these Scrum Masters or Agile coaches can really help with the bottom-line result? Working with agile or Scrum generally means having several Scrum Masters in place, but can you measure their effectiveness and impact. If so - how? More importantly, will more Scrum Masters significantly improve the overall result? Or can we apply a ‘less is more’ paradigm, where it makes sense to reduce their numbers? As an agile leader, how can you make the right decision?

Can you measure the impact of your Scrum Masters?

Before we answer these questions, let’s look at the core responsibility of these coaches. According to the Scrum guide, Scrum Masters are primarily accountable for unlocking the power of Scrum. Their work has three focus areas:

  1. Serve the Scrum Team – helping to improve self-organization, tapping into high-value increments, ensuring scrums take place.
  2. Serve the Product Owner – supporting effective product goals, proper backlog management, empirical product planning.
  3. Serve the organization – supporting scrum adoption, advising on agile implementation, driving an overall empirical approach for complex work, removing barriers.

These are all important and valid goals, but how can we measure a Scrum Master’s effectiveness in achieving them? While that’s a simple question, it’s a difficult one to answer - for two reasons. Understanding those two reasons will help us to figure it out.

1. It’s a supportive role

Firstly, measuring effectiveness is hard because the Scrum Master’s role is a supportive one. Their success can’t be measured by looking at work they’ve done (the so-called output). Success is indirectly measured in the growth of others. It’s seen in the maturity of teams, product owners and the organization. So, how can it be measured? If their jobs are not complex (e.g. a task like tomato-picking), growth can be measured by increased output. While that growth may not all be down to the Scrum Master, if a team grows faster than one without a Scrum Master, that’s pretty good proof. When teams thank a Scrum Master for assisting in their growth, that’s also proof. But unfortunately the role in question is a complex one! Let’s have a closer look.

2. Team roles can be complex

The second reason why it’s hard to measure a Scrum Master’s effectiveness relates to teams and product owners’ actual complex jobs. The tangible value from a Scrum team isn’t as simple as ‘number of tomatoes picked’. We live in an unpredictable, complex environment with many unknowns. An increase (or decrease) in turnover isn’t directly related to output from the last sprints, let alone to the impact of a Scrum Master. That increase (or decrease) is also impacted by competitors, customer satisfaction, seasons, and numerous other factors. It’s the job of the entire Scrum team to review all the internal and external factors and use collective intelligence to improve results. It’s a complex challenge, which involves a lot of collaboration, honesty and trust. The guidance of a good coach makes all of this much easier!

To create a metaphor: A Scrum Master’s role can be compared to that of a personal trainer (PT). If clients see no physical improvement, the PT is ineffective. If clients become fitter the PT is succeeding. When many people start attributing increased fitness to the PT, that PT is having a high impact.

So how can you measure the impact of these coaches? After 15+ years in the agile field, I conclude that you – unfortunately – can’t objectively measure it. But – luckily – you can have a pretty good indication! I consider two ingredients to be most crucial.

  1. Did the teams grow?
  2. Are the teams thankful for a Scrum Master’s contribution?

So, it is possible to indicate a Scrum Master’s impact by looking at the above ingredients. But this immediately makes it important for leaders to facilitate and create clarity on what ‘grown’ means, what ‘better’ looks like. So the team’s development should be very tangible. How can agile leaders do that? By applying the next three tangible tips.

Tool 1 – Key Value Indicators (KVIs)

First of all, it should be clear what it means to deliver ‘more value’. It’s important to outline when teams have increased impact and effectiveness, preferably via a tangible metric. In this regard, Key Value Indicators (or KVIs) are good practice. Rather than measuring internal efficiency (e.g. velocity, output, reliability, system uptime or process delays) KVIs measure actual value delivered to customers. Metrics include daily active users, revenue from satisfied customers, or other value metrics from the EBM-framework.

Tool 2 – Growth of Scrum Teams

Teams grow faster when they have clear expectations from their stakeholders. There are several ways to do this. Describe how better look like, by (next to the above value) measure stakeholder satisfaction, team happiness, sprint goal reliability (how often sprint goals are reached), customer satisfaction, mean-time-to-handle-feedback, mean-time-to-resolve, etc. While these are less important than actual value delivered, these metrics offer lots of quick feedback. Another good practice is to create a growth path or growth phases. Learn how to do this effectively in the ‘ownership-model’ section of my book (Agile Leadership Toolkit).

Tool 3 - Growth of Coaches or Scrum Masters

A third important tool is clarity on what’s expected of Scrum Masters. Agile coaching stances are very powerful in this respect. These stances emphasize the different skills, expertise and experiences Scrum Masters need to have an impact. At the same time, it’s unfair to expect a Scrum Master who’s just started to have all of this from day one. Different skills are needed for different teams. Simply put; new teams need more teaching, while experienced teams need more coaching. This, of course, is far more complex in reality, but organizations must support their Scrum Masters and create clarity and context to enable them to grow over the years. Agile leaders should be transparent and open to collaborating on these growth paths. Through candid conversations. Leaders shouldn’t email the Scrum Masters with the expectations, but collaborate with these coaches to create these paths.

While the role of Scrum Master may previously have been considered superfluous, it has matured over the years to become a valid profession and an area of expertise. A growing number of organizations see Scrum Masters (or similar job titles) as crucial to creating high performing organizations and delivering value to customers and users. By investing in the growth of these people, they unlock the growth of the entire organization.

Some Pitfalls

One potential pitfall is that these tools could be used to judge or condemn teams, Scrum Masters, or Product Owners. It’s important to understand that the emphasis is on transparency, to unlock the power of continuous improvement. Healthy teams are like flowers. Their instinct is to grow and bloom and in the right environment, that happens. Failure to thrive is more often down to the surrounding context than lack of will. When teams don’t grow, successful agile leaders will lead open, candid conversations that inspect and adapt the context. This is a key success factor with self-managing teams. [See also my previous blog post]

So, do you need more or less Scrum Masters?

When teams appreciate the support and mentoring of the Scrum Masters and teams indicate that they can grow faster with more help, it’s likely that you need more Scrum Masters. If teams grow as fast as they can and Scrum Masters have plenty of idle time, it’s likely that you could have less. But the latter hardly ever happens in an organization. It’s more the question of the mindset of the leader. If the leader believes that high performing teams are only 20% more effective, it’s hard to invest in team-development. But high performing teams are often 50% to even 300% more effective, this gives plenty of reasons to invest properly in skilled and experienced team coaches. Netflix calculates high performing teams to be 10x more effective than default teams!

The shifting role of the agile leader

This immediately creates a shift in the leader’s responsibility in an agile context. They become the person who builds and improves a suitable context for self-managing teams to thrive. Their focus is less (or not at all) on product strategy details (that’s delegated to product owners), or technical details (that moves to the scrum teams). Agile leaders elevate the role of coaches and Scrum Masters, seeing them as crucial organizational and people developers!

Want to know more?

Want to know how to measure the Scrum Master’s impact? Contact me, Mijno van der Ploeg or other leadership coaches and we will be happy to help you!

---------------

About the authors

Peter Koning is author of 'Agile Leadership Toolkit' (part of the Scrum.org series), public speaker and senior leadership consultant. He has over 15 years of experience with Agile and Agile Leadership. He has founded re-lead.co | Refreshing Leadership. He actively coaches, mentors and trains leaders of large international organizations.

Mijno van der Ploeg is a senior coach at the Dutch Railway. He strongly believes in the power of the collective brain. He empowers people, teams and departments to achieve a high level of cooperation:

  • so they can manage even the most complex situations.Mijno van der Ploeg
  • To achieve the highest value possible.
  • To get the customer what they really need.
  • So they can move mountains.
  • So they can achieve true excellence.

At home he is a father of two boys that, just like dad, love to climb in trees.