Story-Point Estimation: Value-Based instead of Effort-Based
Hi everyone, I searched the forums to find this dichotomy and could not, so I am posting a thread here. I appreciate your insights.
Recently we had a consultant say that we were doing story-point estimation 'all wrong' and that 'the numbers we are producing from our teams don't mean anything' because they are effort based, not value based. We asked him to explain, at which point he went into discussing that each team needs to create a Story Gauge of what a 1, 2, 3, 5, 8, etc is. OK, fine so far, we agree that is healthy, but then said that the story points should be based on the value to the company, not the mount of time something takes to complete. OK, agree that we should not equate a 3 to 3 days or 3 hours of course, but what did he mean by value pointing instead of effort pointing? Right now, let's say we have two user stories to integrate our financial software with the Vendor A's API and Vendor B's API. The level of technical effort and knowns/unknowns are the same, but here's the difference - Vendor A's people are easy to work with, fast, low bureaucracy, and a developer can be done with it in a few days maybe (story points: 3) and move on to working on something else. Vendor B's process to get the same value integrated, is arduous, and requires much more documentation, as they have multiple levels of management that need to approve it, etc so it may take one developers time for a whole sprint (story points: 8)
He is saying that is wrong, and that the estimation of a user story should be based on what we get out of it, the value, not the time it takes to complete. His analogy was to think of story points as "miles" and velocity is "miles per hour". A "3" story value is still 3 miles, but sometimes those 3 miles might be uphill, in the snow, during a hurricane, etc, so you might be able to do some 3's way faster than other 3's, but they are still all 3's he said. My team would have a higher number for Vendor B instead of Vendor A because of the time and opportunity loss, even though the outcome/value is the same. The main problem this proposes is that a Velocity of 22 and a Velocity of 40 from the same team might actually be the same productivity, since the first sprint is loaded with more 'admin heavy' items of the same value. This makes predictability difficult, and he said 'your goal is adaptability, not predictability, and it's ok to have fluctuations in velocity like that from sprint to sprint'.
So, are you all doing effort-based or value-based pointing or a mix like it seems he is proposing? Everything I have been taught through my career is all about effort-based (various companies seeking agility - some doing SAFe, some doing their own flavor, etc), but that doesn't mean it's right. Maybe this is something new or I've been wrong all along?
Going to get this off my chest first.
Recently we had a consultant say that we were doing story-point estimation 'all wrong' and that 'the numbers we are producing from our teams don't mean anything
There is no "wrong" way and the numbers only need to have meaning to the Scrum Team not to anyone else. If I had a consultant tell me that, I'd probably be finding a way to get a different consultant.
Now that is off my chest, I will say that the rest of what you convey from him is not exactly wrong. Story points have no right way to do it. And yes, a story pointed as a 3 from one team could be pointed as a 13 by another. And that is perfectly ok.
In my opinion, Story Points are a mechanism to help teams arrive on consensus of what a story entails and estimation of that story in relation to others. How does it relate to other stories? That is up to the team to determine and as long as it is consistent across all stories and all time, then it has meaning to them. I do encourage them to not think in terms of hours because the time it takes can vary widely for a variety of reasons such as who does the work.
His example of thinking in "miles" is not a bad one because it shows how a 3 can mean the same thing but be impacted in many ways and not all of them can be anticipated. But I'm not sure I would equate that to value. Interesting thought but still not convinced it is correct.
what did he mean by value pointing instead of effort pointing?
He appears to have confused value pointing with effort pointing. A Product Backlog item should have a description, a value, and order and an estimate. Value and estimation are different things. Planning poker cards can, however, be used for both.
The purpose of estimation is to help a Development Team get its arms around how much work it can take on. What that work is worth -- the value -- is a quite separate concern. The value of an item does not make it easier or harder for a team to implement, or cause it to take more or less development time. Value is maximized using techniques such as Product Backlog management (e.g. ordering) for which the Product Owner is accountable.
Thank you all so far for your comments.
And yes, a story pointed as a 3 from one team could be pointed as a 13 by another. And that is perfectly ok.
I was not comparing teams, I was talking about within a team two items that are both "3" in value, may take very different amounts of time to work and he is saying that they are both still a 3, even if one takes a day and one takes 2 weeks.
Additionally though, he wants us to create a Story Gauge where a 3 is a 3 is a 3 across the org. He says it is valuable for stakeholders outside of the teams to know what teams can handle when forecasting, so there aren't different amounts for each team. For example, here's a simplified Story Gauge he proposed...
- a 1 might be a UI change
- a 2 might be a UI and a DB change
- a 3 might be a UI, DB and config update
- a 5 might be a UI, DB, config update and a template update
- an 8 might be a UI, DB, config update, template update, and an API integration
- a 13 might be a UI, DB, config update, template update, API integration and a new VM setup
- Then he said anything over 13 should be broken down if it cannot be done within one sprint.
So, with all that in mind, my question to you all is - In the instance I gave where we're integrating with Vendor A (easy to work with, takes 1 day to complete) and we're also integrating with Vendor B (hard to work with, takes over a week to complete) he is saying both of those are a "3" since they just have a UI, DB and config update (see pseudo-gauge above) regardless of the time it takes for us to coordinate, since the realized value of both are the same. Is this accurate or would your teams point the harder Vendor to work with a higher number?
"..your goal is adaptability, not predictability.."
It's true that the goal is adaptability. In my opinion, the best way to adapt is to understand the full context of the problem/task you are working on. That way you can decide if you need to pivot/adapt. The discussion around the story pointing brings clarity, transparency and shared understanding. The story pointing helps understand what all is involved in getting this piece of work to your definition of done.
The value (or "what we get out of it") is what the title of the story states (as x, i would like to do y, so that z) and the order/priority of your backlog. The value should be clear to the development team before the story pointing occurs, if not, the Product Owner has work to do prior to the dev team estimating.
Relative estimation (story points, t-shirt sizes, animals, etc) is a method to assist Scrum Teams (PO, Dev Team) in planning sprint work and gauging what they may be capable of completing within a specific time frame.
While a separate value-based rating can also provide Scrum Teams with visibility into what the business considers most critical, in no way should Scrum Teams confuse these ratings with relative estimates.
To paraphrase Dan, I would strongly question the consultant's assertion that a Scrum Team can effectively plan using value-based ratings.
He says it is valuable for stakeholders outside of the teams to know what teams can handle when forecasting
The benefit of using story points to estimate the effort of a story is so that the Product Owner (and others) can forecast work.
Trying to use the same story point range across teams/products is not advisable, as there are many factors involved in teams choosing a number.
I would also agree with Timothy's suggestion to "strongly question the consultant's assertion that a Scrum Team can effectively plan using value-based ratings".
In your example, if the effort of working with Vendor A is less than working with Vendor B, they should have different estimates, regardless of the final product.
I consider five main benefits to estimating:
- Helps the Development Team forecast
- Helps the Product Owner forecast
- Helps the Product Owner prioritize
- Helps the Scrum Team identify misunderstandings about scope
- Helps the Scrum Team identify when to make items smaller
I would say the Development Team should choose the system that best achieves this.
I was talking about within a team two items that are both "3" in value, may take very different amounts of time to work and he is saying that they are both still a 3, even if one takes a day and one takes 2 weeks.
And believe it or not, I agree with him on this. In this thread (https://www.scrum.org/forum/scrum-forum/8360/mapping-story-points-hours) @Timothy Baffa gives a great analogy of story pointing a task of moving a rock. Go to that thread and search for "rock".
Additionally though, he wants us to create a Story Gauge where a 3 is a 3 is a 3 across the org. He says it is valuable for stakeholders outside of the teams to know what teams can handle when forecasting, so there aren't different amounts for each team.
Wow, he has a very different view of story points. Yes, I have heard other "coaches" advocate this but I have never seen it work. Story Points are just estimates. And estimates will vary based on who is making them. Take for example, you want to remodel your bathroom. You ask 3 different contractors to give you estimates. Would you expect them to be identical?
About the "valuable for stakeholders outside the team..." statement. If the Scrum Team is doing a good job of including the stakeholders in the ceremonies and making everything transparent, the stakeholders should be able to come to understand things just as the Scrum Team does To me what it sounds like he is advocating is that it would be "valuable to management to compare teams to see who is under performing". And using estimates for that is a REALLY BAD idea.
The only thing I can halfway agree with in his scale is that a 13 is large. But unlike him I have seen teams pull a 13, and even a 20) into a sprint and complete them. But they also took into consideration that they would need to limit other work and swarm on the larger story. I have also seen stories estimated as a 5 turn into more work and complexity than stories that were estimated as a 20. It is all relative and fluid. Estimation gets better as the knowledge of the team on the problem space increases. It gets better as the team starts to learn the strengths and weaknesses of the members. Trying to create a "scale" to be used across the org is not something I'd advocate.
Recently we had a consultant say that we were doing story-point estimation 'all wrong'
Interesting. I mean no disrespect, but they seem to be amateurs. A good consultant wouldn't say it's all wrong - they'd be more likely to say the estimation could be improved, or that the current estimation isn't helping the business too much and come up with authoritative proof in support of the change.
Is this consultant "specialized" in Scrum or are they familiar with other, non-Scrum, concepts? If the latter, I'd be very curious to hear how they conciliate the "value based" story points with WSJF?
I am not an expert in story point based estimation myself. But here are my two cents anyway.
I believe there is no fixed method to do the estimation or assign story points. While your consultant is right to an extent, he is not completely correct. Consider the below two points.
- a 1 might be a UI change
- a 2 might be a UI and a DB change
In this case, what does UI change denote? Is swapping a logo considered UI change? What about redesigning a form with 20 fields? What about realigning 10 different UI elements? So, you see, that just the value or just the effort can't be used to estimate your story.
That said, my preference is to consider hours(approximate obviously) as the main criteria. The reason is that value is useful from user's perspective. But estimation is done by the team for the team. And from their perspective, hours matter.
That's why we say "velocity might change". Because we are estimating how much work we can do, not how much value we can generate. Since velocity is a measure of value, it will fluctuate. If we focus on just the value, then we will always be working on high value items only, while low value stories will never be considered.
To conclude, we want our estimate to always be as close to 100% as possible without going over. Value can change based on what we delivered. Sometimes, we might deliver a low value story if it is prioritized higher or is a dependency for other high value stories. So, yes, velocity will change as it should track the value we are delivering.
Again, this is just my understanding. It is possible I have not considered every scenario.
So, you see, that just the value or just the effort can't be used to estimate your story.
How about include efforts , complexity , uncertainty, risk while making an estimate. I personally find its much easier when we have the abstract & relative estimation just like T-shirt sizing - XS,S,M,L etc or story points. When we look at any task, generally we refer the quantity/size first. Its big , too much, complex changes, small change , or many small changes etc. Why not restrict the estimation here itself with the size? By using 'approximate time' , isn't it still volatile and abstract? At end of course use what best suits for your team to generate the value and deliver the Goal. I just shared my thoughts , it hardly matters how you estimate if Goal is delivered successfully.