Skip to main content

Production hotfixes and Scrum

Last post 08:55 pm April 10, 2014 by Charles Bradley
16 replies
05:40 am April 7, 2014

The situation is that our production software is not bug free. Of course our sprints have to be result in shippable software, but we are just started using Scrum and I believe even when using Scrum production bugs will always exists.

So how to handle high important issues from production? These kind of issues must be fixed as soon as possible and can’t wait till the next sprint.

I think there are 2 options:

1) Hotfix bugs are placed on top of the current sprint backlog. The team will fix this issues as soon as possible.

2) Hotfix bugs are placed on a separate list. The team (or only some team members) first do the work on the hotfix list before continue the work on the sprint backlog.

In both cases there is a risk that some committed items on the sprint backlog will not be done when the sprint is over. This risk can be reduced by planning time for hotfixes during the sprint.

In my opinion option 1 is better than option 2 because option 1 is more transparent. And I think we don’t have to reserve time for hotfixes because we don’t know how much time we need.

I'm curious about other opinions and experiences.


05:47 am April 7, 2014


I would prefer to start with X amount buffer per sprint for hot fixes and over a period of time you will get to know how much effort goes into these issues and team can adjust the commitment accordingly. This will ensure the transparency too.

I am assuming that team is estimating the hot fixes in story points as soon as they are coming into sprint backlog.

Cheers
Sanjay


10:07 am April 7, 2014


Posted By Franck van der Sluijs on 07 Apr 2014 05:40 AM
Of course our sprints have to be result in shippable software, but we are just started using Scrum and I believe even when using Scrum production bugs will always exists.


Hi Franck,
this sounds to me as if you do not believe in the team's ability to ever deliver a shippable increment.
I agree with you that software is never bugfree. However, the increment has to be in sufficient quality for production. If it is not, the solution is not a bugfix release (just a workaround), but process improvement.
Try to figure out with your team what you can improve in your process so the critical bugs which are not accepted in production do not emerge. You will see that building and deploying bugfix releases is pure waste.


11:10 am April 7, 2014


Posted By Ludwig on 07 Apr 2014 10:07 AM


Posted By Franck van der Sluijs on 07 Apr 2014 05:40 AM
Of course our sprints have to be result in shippable software, but we are just started using Scrum and I believe even when using Scrum production bugs will always exists.


Hi Franck,
this sounds to me as if you do not believe in the team's ability to ever deliver a shippable increment.
I agree with you that software is never bugfree. However, the increment has to be in sufficient quality for production. If it is not, the solution is not a bugfix release (just a workaround), but process improvement.
Try to figure out with your team what you can improve in your process so the critical bugs which are not accepted in production do not emerge. You will see that building and deploying bugfix releases is pure waste.


I agree, but my question is how to deal with our current situation. We are using Scrum since a few weeks, but our product exists many years.

We are not building a new product but we are maintaining an existing product. Every release contains bug fixes and new features. Sometimes a bug fix can’t wait for the next release. That’s what we name a “Hotfix”. The discussion within my company is how to handle these hotfixes (not how to prevent making bugs).

I believe when we improve the process we create less bugs and less hotfixes are needed. But for now we have to deal with the current situation.


12:10 pm April 7, 2014

OK I understand. So maybe it is really an option to include an item "Bugfix release" in each planning, because you already know you will need it. You just don't know yet which bugs are included and when it is needed.
Another option is to play the "more is learned" card within the sprint and to re-negotiate the sprint backlog between PO and the dev team. This option is reasonable if the need is expected to arise in rare cases only, because it can be abused by the PO for quick changes. It is important he understands the consequences to productivity.


03:37 pm April 7, 2014

Hi Franck,

when we or someone else discover a bug, first we talk to Product Owner if it's hot or not.

If it isn't hot, we will add on the top on our backlog to deal with it during next sprint (or later).

If it's hot and we have to work on it immediately, we'll move the UserStory(-ies) from the backlog to the Corner of Hope (agreed with PO of course) and assign one or more people (or only for % of his/her time) to work on a bug depends on PO's decision how sever the glitch is.

The problem is a bit more complicated, because usually it's impossible to estimate bugs.

BR,
Bartek


09:31 am April 8, 2014


Posted By Xebord on 07 Apr 2014 03:37 PM
when we or someone else discover a bug, first we talk to Product Owner if it's hot or not.

If it isn't hot, we will add on the top on our backlog to deal with it during next sprint (or later).

If it's hot and we have to work on it immediately, we'll move the UserStory(-ies) from the backlog to the Corner of Hope (agreed with PO of course) and assign one or more people (or only for % of his/her time) to work on a bug depends on PO's decision how sever the glitch is.

The problem is a bit more complicated, because usually it's impossible to estimate bugs.


Thanks for the reply. What’s the “Corner of Hope”, is that part of the current sprint backlog? Or is that a separate list?


03:03 pm April 8, 2014

The "Corner of Hope" is a part of the current sprint backlog where are stored User Stores which the dev team would like to finish, but they are not promised to do it. Hence there is no problem, if the team doesn't complete them.

So if the new fault is found and needs to be solved immediately with already defined backlog, the team has to removed the lowest priority US to make space for the fault. But instead of removing it for the backlog completely, put it in to COH.

Hence the team is sending the message to PO: we are going to work on the fault, we have to move some USs to COH, but still we will try to finish them.


BR,
Bartek


03:13 pm April 8, 2014

I don't believe in the "buffer" or "separate list" strategies.

My preferred method is to do this: http://www.ScrumCrazy.com/bugs

A "shorter method" ..."If it's an business emergency, do it now, and then adjust sprint scope based on the impact." ... "Not a biz emergency, goes on the PBL and ordered by the PO just like any other PBI"


03:30 pm April 8, 2014


Posted By Charles Bradley on 08 Apr 2014 03:13 PM
I don't believe in the "buffer" or "separate list" strategies.

My preferred method is to do this: http://www.ScrumCrazy.com/bugs

A "shorter method" ..."If it's an business emergency, do it now, and then adjust sprint scope based on the impact." ... "Not a biz emergency, goes on the PBL and ordered by the PO just like any other PBI"


Nice bug chart, thanks!


06:00 pm April 8, 2014

Franck, great, hope it helps!


01:46 pm April 9, 2014

In Lean theory, a case can be made for immediately dropping all Work In Progress and fixing a defect regardless of how trivial it may be. This is because the process evidently needs to be fixed even if the product itself is not (yet) seriously compromised. After all, deferring a minor fix doesn't just mean deferring the release of a more valuable product. It also means allowing a faulty process to continue at the risk of potentially introducing more serious defects, and compounding the waste incurred, due to the shortcoming not being addressed.

Well that's the theory. In Scrum practice however, this risk of continuing with a broken process needs to be weighed against the risk of not delivering a Sprint increment at all. The management of risk, just like the delivery of scope, is time-boxed in Scrum. That's why it's important to have a strategy like Charles' for triaging defects into or out of the current Sprint, and for revising the sprint forecast if necessary. The worst thing is not to have a strategy at all.


04:39 pm April 9, 2014

> The "Corner of Hope" is a part of the current
> sprint backlog where are stored User Stores
> which the dev team would like to finish, but
> they are not promised to do it.

Just out of interest, what does that make the remaining stories? The Backlog of Despair?


03:56 pm April 10, 2014

> The Backlog of Despair?

:-DDDD

but it sounds you are a bit sceptical, doesn't it?

If so, could you elaborate a bit?

BR,
Bartek


04:31 pm April 10, 2014

Hi Charles,

nice chart, it's good that it focuses on fixing the bug.

Let me please summarize it: if it's severe bug, fix it now. If not, fix it later. Right?

But what about sprint goal? And the current User Stories? It's not covered by the chart.

Let's assume that the bug appears at the beginning of the sprint and it takes half of the sprint to solve it . What next? Cancelling the sprint?

I don't understand, why do you suggest not to track the bug (3A)?


BR,
Bartek


08:54 pm April 10, 2014

Your summary is decent, but it skips a whole bunch of other stuff about whether things go on the PBL, the SBL, or not tracked.

Sprint goal not mentioned by the chart, agreed, not meant to. If you have a biz emergency, I would expect that the SG would no longer be nearly as important. (not to mention Scrum)

Re the current PBI's, see the bubble with the asterisk(*)

Re: 3A -- if the issue is very small (immaterial/tiny, etc), then the time spent to track it is not nearly as important as just fixing it. This is to prevent us from getting caught up in red tape and paperwork waste. Much important to just fix it!


08:55 pm April 10, 2014

Also re: 3A ... More value in "responding to change" over "following a plan"... and more value in "working software" over "comprehensive documentation.


By posting on our forums you are agreeing to our Terms of Use.

Please note that the first and last name from your Scrum.org member profile will be displayed next to any topic or comment you post on the forums. For privacy concerns, we cannot allow you to post email addresses. All user-submitted content on our Forums may be subject to deletion if it is found to be in violation of our Terms of Use. Scrum.org does not endorse user-submitted content or the content of links to any third-party websites.

Terms of Use

Scrum.org may, at its discretion, remove any post that it deems unsuitable for these forums. Unsuitable post content includes, but is not limited to, Scrum.org Professional-level assessment questions and answers, profanity, insults, racism or sexually explicit content. Using our forum as a platform for the marketing and solicitation of products or services is also prohibited. Forum members who post content deemed unsuitable by Scrum.org may have their access revoked at any time, without warning. Scrum.org may, but is not obliged to, monitor submissions.