Quality metric in SCRUM

Last post 04:02 pm April 9, 2017
by Jonathan Mailhot
10 replies
Author
Messages
12:35 am January 20, 2014

one of the metrics of software quality is defect rate/density

I would like to know from other practitioners how the quality metric is being tracked in scrum practice.

Assume there is a new product development is going on and features are being released quarterly to the customers.

in scrum there is only one product backlog, so I assume there is not separate PB for bugs.

the bugs/defects that are raised from customers will be added to product backlogs and based on priority they gets addressed in the upcoming sprints.

thanks

01:28 am January 20, 2014

That's right, there is just one Product Backlog. There is no separate defects backlog, because once work has been accepted as meeting the Definition of Done it cannot be subsequently considered to be undone.

Any defects should be triaged as new requirements by the Product Owner, and the DoD should be revised to satisfy this change in scope.

Arguably, a more useful metric is to track waste incurred. Some defects (or newly discovered requirements if you prefer) are very serious and must be addressed immediately in-sprint. As such they pre-empt anything on the Product Backlog or which has been forecast for delivery in the current Sprint Backlog. Unplanned work is a form of waste, and it is important to capture metrics for this, because it can divert team effort from achieving Sprint Goals.

01:38 am January 20, 2014

> the DoD should be revised to satisfy this change in scope. 

I should point out that this is just one of the inspect & adapt opportunities. If releases occur quarterly as you suggest then this should be challenged, as more frequent releases would lower batch sizes, and potentially improve the team's response to defect discovery.

03:00 am January 20, 2014

Hi Ian,

thanks for the explanation, still I'm not clear on the metrics of software quality in agile eco system

What you are saying to achieve highest quality during pre-production. in pre-production we have a goal or commitment of zero bugs.

The quality of the software that is measured only when the code is in production. some of the metrics would be bugs surfaced per year or something like that and customer satisfaction.

If someone wants to know how good the quality of product which was delivered, what is the metric to look for ?

all the inspect and adapt is happening during pre-production.
Is tracking metrics, inspect and adapt during post production a characteristic of agile?

03:47 am January 20, 2014

> If someone wants to know how good the
> quality of product which was delivered, what
> is the metric to look for ? 

There are many to consider: ROI, customer churn rate, market share etc. The important thing is that metrics should be actionable - vanity metrics are to be avoided.

> all the inspect and adapt is happening during pre-production. 
> Is tracking metrics, inspect and adapt during
> post production a characteristic of agile? 

Yes it is, at least in agility at scale. The Scrum Guide doesn't have much to say about this but other documented approaches do, such as Lean Startup and Agility Path.

05:22 am January 20, 2014

I believe there is no direct measure for quality. If the PO chose the wrong PBIs, but the dev team implemented them in a high quality, you will still have a low ROI etc.
Defect rate is slightly better, but the number alone doesn't say much.
You can complement this with metrics like test coverage, static code analysis etc.
You can also measure, how often an increment is really shippable, meaning there are no bugs left which prevent it from being shipped.
But in the end what counts is what the customers perceive as quality.

10:16 am January 20, 2014

I agree with Ludwig. The metric that counts for measuring quality is customer perception about the product. For example, you can create less-user friendly product with zero defect. And that doesn't mean that the product is high-quality.

11:20 am January 20, 2014

> The metric that counts for measuring quality
> is customer perception about the product. 

We have to be careful here, because customer perception might not be an actionable metric. It can be a vanity measure depending upon how questions are framed, and even then the value of the data can be questionable. For example Steve Jobs was adamant that it isn't the customers' job to know what they want, and Eric Ries has taken a similar position.

09:29 am January 21, 2014

I think you are right, Ian, however Steve Jobs' quote is missing context here. As far as I know, he said the customer doesn't know what he wants before he sees it (and knows that's not what he wanted). He quoted Henry Ford: "If I had asked people, they would have told me they needed a faster horse". This effect is the reason for Apple not doing market research, and it's also the reason for building shippable increments in Scrum.
So, I understand "customer perception" as perceiving the product (increment), and that is important for success, however it is difficult to measure.

04:08 pm January 25, 2014

> I understand "customer perception" as
> perceiving the product (increment), and that is
> important for success, however it is difficult to measure.

It's certainly very difficult to measure, or even to define satisfactorily. However, we may have a scrummish word for what is needed. In Welsh rugby there's a concept known as "hwyl", which is what happens when the team and supporting crowd get together, find their flow and common cause, and pull in the same direction.

All impossible to measure of course. Yet the most successful Scrum Teams I've seen all exhibit hwyl, even if they aren't familiar with the word, and their customers are in on the fun.

11:07 pm April 8, 2017

Great discussion. Very helpful insight.  Thanks to all the participants.