Managing a continuous flow of UAT bugs while sprinting
I’m new to this forum and can’t wait to engage in interesting discussions regarding scrum. Currently I’m the product owner of our software and even though we made some huge steps since we started adopting scrum we still have a lot of questions. I’ll try to keep this post focussed on one topic so this discussion will hopefully not go all over the place.
I have a question regarding executing sprints and managing UAT’s for several customers and dealing with bugs that come out of these UAT’s.
First some context;
- We are a small financial software company which delivers software to several local and international customers
- Our sprints are two weeks and our policy states that every 15th of each month customers can decide to get the last deployment installed on their acceptance environment for UAT purposes. Yes I know 15th of the month…. That doesn’t make any sense. This was decided by a manager who didn’t know how scrum worked at the time and is unfortunately still our policy. Will try to change this soon!
- Our software requires a combination of customer experts to test during UAT
- Our software requires a UAT because external interfaces are only available on the customers acceptance environment. Results of interface test done on acceptance are not visible to us but only to our customer. It’s impossible to test the entire software chain
- All customers have different UAT lead time varying from 2 to 6 weeks
- Our automated test coverage is very low
- Our manual test coverage is average
- Our system has a lot of legacy
During sprints where we plan 60% of our time to work on new functionality, rework and technical debt we constantly get back UAT bugs from customers that expect that we fix these bugs and carry out a new deployment when the bugs are fixed a.s.a.p.
Carrying out the UAT’s at this point is still necessary because;
- Automated test coverage and manual test coverage is still too low. In the past this has lead to several crappy releases resulting in customers that don’t trust the quality of the software and demanding to do a UAT
- External interfaces are not testable on development and test environments which results in customers needing to have to test these interfaces during UAT
- We build a lot of new functionality based on customer input. The customer first wants to test and accept this functionality (often including external and internal interfaces). Because UAT’s can take up to several weeks they don’t fit in de sprints
Obvious solutions would of course be to invest in automated testing and install stubs for external interfaces but then, at least for the next year, we will still have customers demanding UAT’s because we still have to proof we can deliver high quality software.
So… regarding everything stated above. Do you guys and girls have any advice you can give me to manage the continuous flow of UAT bugs? All tips and help is very welcome.
Geert de Jong
What can be done to get the UAT test criteria articulated ahead of development, so it can become more test-driven?
Also, what can *you* do (as PO) to improve customer representation during each Sprint, so each finished item is quality-assured in the Development environment as part of the Definition of Done?
If your definition of "done" does not include a reasonable test coverage, then this problem will remain forever. Promise!
You have to get things automated, let alone to make sure that working features are not damaged between increments.
In the sense of scrum, your increments are not delivering 'working' products - that's pretty much the core of the problem.
To get around this, testability should be your #1 priority and you should account for that in the product backlog. It's easy to justify because users can't work with crappy software - actually, the increment/release is worth nothing if no one is going to use it. Before cleaning up the technical debt, you better make sure that re-design and re-factoring doesn't screw up everything. Again, testing is key to success.
Another idea is to make the bugs high priority in the next sprints. You want to get working software out of the door, not a buggy software with more badly tested features.
Regarding stubs and external interfaces, I'd recommend to ask your customers if they can provide (better clone) their UAT environments for nightly development tests.
I hope this is of any help for you.
Thanks for your reply. I think your conclusions are spot on! Our biggest impediment at the moment is that we don't have a reasonable test coverage resulting in non working products. And if we don't adress this and free up time to work on this the problem will remain. I'm going to adress this impediment to management so they will also understand the importance so I get the means to solve this.
We are currently already in the process of getting better UAT environments from our customers so we can expand our test coverage with several key interfaces. That said, it's a slow process and will probably take several months before realized.