This is crazy - not being able to forecast a rough budget estimate for the entire project upfront - needed for funding request

Enterprise does not have unlimited fund so they need to know. I find this is difficult with Scrum. (Well, Scrum says it's difficult to master :-). 

@Monir Khan, You are right, organizations may not have unlimited funds, and Scrum does not say how this should be handled. That is why it is called a framework.

The below excerpt from the Scrum Guide is potentially where some clarity regarding this topic may even be mentioned.

The Product Owner is responsible for maximizing the value of the product resulting from work of the Development Team. How this is done may vary widely across organizations, Scrum Teams, and individuals.

Notice, the last sentence. That is the key.

So wondering, how do you, as a Product Owner, come up with a high-level estimate when Scrum does not allow upfront planning for the entire project that helps coming up with the high-level funding estimate?

Now, Scrum doesn't say that you shouldn't have a high-level estimate. What we need to takeaway is that we shouldn't be spending too much time and energy trying to get an up-front budget. In complex product development, this is almost always going to be a challenge. Therefore, either use historical data (empiricism) or current information (again empiricism) and create a budget based on what you know now. I am by no means saying that this is easy and that there are no challenges to it.

Let me clarify with an example. You plan a trip to a different country. Can you truly create an accurate budget? or is it more realistic that you would create an approximate budget with some buffer? Can you say that your trip will follow exactly as you planned? What if your trip gets canceled? What if you don't have insurance? What if you have to pay for a hotel as a result of a flight cancellation? What if on the way to the airport, on this particular day there was more traffic than usual and you missed the flight? How would you plan for all these kinds of unknowns in reality? Whatever your answer might be, a similar logic will apply in this case as well.

Hope this helps.

Scrum does not allow upfront planning for the entire project that helps coming up with the high-level funding estimate?

Scrum doesn't actually forbid this. It's just that Scrum puts you in a position that doing so may be wasteful, because investment and return on investment can be emergent, as new increments of the product are developed.

The costs of owning a product should be possible to budget. This can include various costs, such as paying for hardware/software/licences/insurance; but is also going to include running teams.

Decisions can be taken throughout the lifetime of the product to increase or reduce spending; and it's common that Product Owners also need to balance this within organizational budget constraints. Total cost of ownership is an important factor to consider when managing the Product Backlog.

Depending how your organization is structured, you might need to hire people to form a new team, and you might not have a role for them if the product is unsuccessful; or these people might already work for the organization.

For instance if you have a team that are already in your organization, and able/willing to move onto a new product, and would still be able to add value elsewhere if the product is terminated, the financial burden is probably relatively low. It would theoretically be easier to commit to funding the team in the short term.

Producing a releasable increment at least every sprint would then put a Product Owner in a strong position to continuously decide whether sufficient value is being obtained that continuation of the development effort makes sense.

If after one sprint, it already becomes clear that the product will not produce an acceptable return on investment, decisions can already be taken to cease further investment.

how do you, as a Product Owner, come up with a high-level estimate when Scrum does not allow upfront planning for the entire project that helps coming up with the high-level funding estimate?

My advice is to use the same approach you ought to take for any estimates: base them on evidence and the advice of those who will do the work.

Thank you, Steve, Simon for your insight and wisdom. Very much appreciated.

So the thing I learned here that - Scrum does not forbid you doing some sort of high-level budgeting estimation up-front. We just have to be cognizant of time - not spending whole lotta time on this. I think the secret sauce is in this statement "How this is done may vary widely across organizations, Scrum Teams, and individuals.  Thanks for bringing this into the light. 

I was coming from my professional experiences where we have to come up with a high-level budget for any initiative, for the funding approval, so the fund can be secured and allocated for the initiative. This is just a high level and for that, a rough up-front plan is needed from beginning to end, and yes, so many things could be changing along the line that we have to manage as we move along. Good to know that is also allowed in Scrum (not forbidden).  

So, @Steve Matthew, in your traveling example, it would be - say, I'm going to country B from country B by SpaceX Airline in December for 2 weeks. I'm taking the business class. I'll be staying at the hotel, LaLa Palooza, close to downtown. I'll use taxies to go places around the city while am there and have my meals outside. So this would be my high-level plans, and for this, I can come up with some high-level estimates and make my case for how much I plan to spend on that business trip and get the trip approved. Yes, there are so many things that can go out of plan due to unforeseen circumstances, but I do have a high-level initial estimate. This is not empiricism by any means but it is needed for my trip to be approved.  And coming up with this high-level estimation was my concern. Now I know it is not forbidden in Scrum.

 @Simon Mayer  Yes, there are certainly ongoing maintenance, operational cost besides the upfront costs (hardware/software/licenses/ etc)  and those needs to be factored in for my initial high-level estimates so I can present a rough estimate on how much I am going to spend for the project for the next 4 years or so.  This is just a high-level number to get some understanding, so people have some rough idea on spending, and yes, so many things could be different in between that we have to manage as we move along. I do have some understanding now how to manage with Scrum.

Sincerely thanking you both for sharing your insight. 

Cheers

As others have mentioned, Scrum doesn't forbid up-front planning and high-level estimates of the necessary funding. However, I'd challenge the effectiveness of these activities in a situation where Scrum is appropriate.

First off, I'd point out that Scrum isn't always appropriate for all problem spaces. Some problems are very well-defined, tightly constrained, governed by best practices, and highly predictable. In these cases, you know a lot about the problem, the potential solutions, and it's straightforward to understand the impact of your decisions on solving the problem. In these cases, it's more likely that spending time with up-front planning and estimating can get you to a realistic plan, and you can often see several of the key risks and have mitigation strategies built into your plan. Scrum is not appropriate for these cases and would likely add overhead.

Scrum is suitable for situations that are less obvious, such as when you are dealing with a lot of unknowns. Some of these may be known unknowns, while others may be unknown unknowns. You can't effectively predict the likely outcomes of your decisions and you can't see details of your potential solutions early. Instead, your solutions will likely emerge over time and require feedback and iteration. Up-front planning and estimating don't add value due to the number of unknowns. You can't effectively enumerate and manage your risks, either. This is a case where Scrum is more appropriate.

If you're in the first category, I'd recommend stopping and thinking about why you're using Scrum. If you're in the second category, I'd want to push back on the idea of investing too much time on up-front planning and estimation. That doesn't mean you don't do any up-front planning. Some level of planning is going to be valuable, but perhaps not enough to give a meaningful estimate.

So, you're in that second case. What can you do?

You have a team. It's a small team, perhaps. A Product Owner, a Scrum Master, and 3 to 9 members of a Development Team. Given that team, you know the cost to fund that team for a unit of time. That unit of time is a Sprint. The Sprint length is going to vary based on your context - a lot of people work in 2-week Sprints, some in 3 or 4, some based on calendar months. Based on your experiences and domain knowledge, you can pick an appropriate starting Sprint length.

The stakeholders have an idea for a product or service that they need and your team has some level of experience building similar products or services. The stakeholders know how much this idea is worth if it can be fully realized. They also know how much they are willing to invest to learn. It's probably some fraction of the value of the idea.

Rather than doing an up-front plan, offer a fixed number of Sprints. After each Sprint, you can demonstrate progress and get feedback from the stakeholders. It may take a few Sprints for the team to get into a "well oiled" state, but once they do they are regularly meeting their goals and demonstrating a valuable product or service. The stakeholders get to decide if it's worth funding another Sprint or not. There are a lot of different outcomes. Perhaps they realize that their idea is much more complex than originally thought and it will take more effort to realize it than the value that can be obtained. Maybe they realize that they don't need the full scope in order to start seeing a return on their investment. Maybe they continue to revise their ideas as they have a tangible product in front of them.

Instead of spending days or weeks with up-front planning and estimating, you have something tangible in weeks or months along with a much more realistic view of the effort and complexity of the problem being solved. A collaboration between the stakeholder and the team leads to optimizing every dollar spent. The risk to the stakeholder is funding a couple of weeks to a few months of the team's time, with the ability to terminate the project at any point when the cost exceeds the value.

Personally, I'd rather fund the team for a month or two and see the actual products being developed than plans and documentation being written. It may cost a little more than the planning, but the results will be of a much greater quality.

Thank you, Ian.

I think I needed to be clear that I was coming for an estimation for launching a new initiative where there was no evidence. Yes, totally agree, if there is evidence then that's what we use for estimation (empiricism).  Thanks so much for chiming in.

Cheers

I understand where you're coming from. Whenever I've worked with ELT or Portfolio Governance, they say "yeah! Agile is great but we need budgets, timelines and milestones and we need that now for next year"

It's not a battle I've won so I've had to work with Program Managers and create rough estimates of features I can deliver on, and use the cost of the Sprint to project out.

This is not empiricism by any means but it is needed for my trip to be approved.

@Monir Khan,  I think there is some clarity required here.  Let us first look at the definition of empiricism, from the scrum guide,

Empiricism asserts that knowledge comes from experience and making decisions based on what is known.

So based on the above understanding if you go back to my example, what I was saying is that you can make certain decisions based on what is known now. You may estimate a certain amount of money, but you cannot accurately state that this is my budget, as there is always room for error. Also, in the way you were describing your trip, the reason you were able to come up with those high-level estimates is because you had a rough idea of what all those things would cost. You had the knowledge of it. If let's say you landed in a country without knowing the cost how would you know how much to spend?

When you carry out one Sprint, you may be able to get a rough idea of how much it cost and you may be able to forecast how many Sprints it may take to complete a certain amount of work. Using that info you could estimate your budget again.

Realistically, even in traditional project management, if the budget is insufficient, then either you discontinue the project or you get more money. Fundamentally, I don't think there is much of a difference. 

Thank you, Thomas. Appreciate your chiming in and sharing your insight.

I totally agree for some initiative Scrum is not appropriate, for example making a building. There's no room here for an adaptive approach.  You can't just create a portion of the building and aks people to go in do the testing by living in there. There may not be electricity, water, and so many things for living and testing.  This got to be a predictive approach. In this case, we do the upfront planning and come up with an estimation. There should not be any doubt about this.

However, on the other hand, if I have an initiative to create a mobile banking app then that definitely be better approached with the Scrum as technology changes rapidly, and they're so many things that may need to be changed on the fly as we move along and learn. And the suggestion you mention - running 2/3 sprints to get some better understanding and estimation, and this should be covered by the seed funding, and it will help coming up with a more realistic estimation for the entire estimation. This seems ideal but am still thinking what Mark Adams just mentioned and that's the realistic view in corporate worlds. I guess, we just have to learn how to navigate through all these challenges 

Thanks again...

Thank you for chiming in Mark.

Bang on. You just hit the nail on the head. That's exactly my concern - in a corporate world we need to forecast a high-level estimation up-front so portfolio/finance folks have some idea on how much they have to allocate for a specific initiative. 

Good to hear you have practical experience in managing this. Thanks for sharing. 

Thanks again, Steve. Very much appreciated.

I think, am beginning to get some grasp on how to handle this.  As was also suggested by Thomas Owens, so start out by running 2/3 sprints to get some better understanding and estimation, and that should be covered by the seed funding. I guess, we just have to learn how to navigate through all these challenges to come up with better ways to do some estimation. 

Cheers...

Check out this presentation by Robert Pieper PST delivered at PMI Philadelphia about Financing Agile Delivery with Forecasts. 

Thanks

Thank you Aditya for sharing. Will check it out..

Cheers...

So wondering, how do you, as a Product Owner, come up with a high-level estimate when Scrum does not allow upfront planning for the entire project that helps coming up with the high-level funding estimate?

I am the Scrum Master of my team, but when we talk budget with the customer, we formerly did a rough estimate, based on the requirements known at that point. And we told the customer that this is just a budgetary estimate with a certain risk.

Now as the Scrum Team is working for quite some time we know the velocity of the team. The PO will break down the PBIs and we do a Points Estimate on those, to tell how many sprints we will be working on the items and then we can give the customer an estimate on budget and timeline. Always with the note that both may change, due to the agile approach. And if we get changes in, we inform PO and customer about the changes in budget expected, again from the points we estimate.

Thank you for sharing your experience, Tim. 

I guess coming up with the Points Estimate comes from experience. Do you use any specific model/reference point for this?  Thanks 

We are doing Story Points estimation for quite some time now during Refinement and Sprint Planning. And yes, we have some reference items.