Is UAT part of the Sprint?

Last post 04:12 pm November 16, 2018
by Tilmann Zuper
22 replies
Author
Messages
05:05 pm October 30, 2018

At my organization we follow a schedule so there's a release to production every 4 weeks.  So the Sprint schedule is setup the following way:  Two weeks development then we go into one week QA. After QA signs off, we go into UAT and at that time the development for the next sprint starts.  So the UAT is not part of the sprint.  Is it ok to move with the above schedule if we promised our client a new release every 4 weeks?  

 

Sample Sprint

05:44 pm October 30, 2018

This goes to Definition of Done.  Is the story complete if it hasn't gone through UAT yet?  What happens to the story if UAT finds problems? Is it considered done then?  Admittedly, this will require you to think more carefully about the sizing of your stories, but that's not necessarily a bad thing.

06:16 pm October 30, 2018

I highly doubt that the item be delivered to production if it has not completed UAT.   In addition, it sounds like you just divided the sprint by phase, creating a mini-waterfall in each sprint.   This is neither Scrum nor Agile.

In my opinion, you need to change the "Focus" of your sprints, from trying to keep everyone busy (premature optimization), to actual completion of items regardless of who is busy or not.

07:03 pm October 30, 2018

It sounds as though the work in a Sprint is being moved between stations as a batch, and with reference to a schedule. There’s probably no sense in doing so if you have well-formed items that are independently valuable.

It is usually better to make schedule orthogonal to the workflow, which means limiting Work In Progress and having the team focus on one item at a time in order to complete it. In other words, think about how you can smooth the flow of value throughout a Sprint so as to deliver it earlier and more often.

10:10 am October 31, 2018

In addition to the points made above, I'd like to note that your current company has the wrong understanding of the term sprint.

At my organization we follow a schedule so there's a release to production every 4 weeks.  So the Sprint schedule is setup the following way:  Two weeks development then we go into one week QA. After QA signs off, we go into UAT and at that time the development for the next sprint starts.  So the UAT is not part of the sprint.  Is it ok to move with the above schedule if we promised our client a new release every 4 weeks?  

So you have

  • 2 weeks development, followed by
  • 1 week QA, follwed by
  • 1 week UAT

Your company seems to believe there are 3 week sprints, releases to production every 4 weeks, but the reality is quite different. 

Development, QA and UAT should go hand in hand, and daily cooperation is a must. As soon as a story is dev complete, it's QA-d and pushed for final UAT. So rather than following the traditional stages of design > development > testing > acceptance, you should put efforts into integrating all aspects of the work into a certain time frame, with a daily completion rate, and inspection the following morning.

04:47 pm November 1, 2018

You are doing waterfall or possibly Kanban but certainly not Scrum.  From the Scrum Guide

The Development Team consists of professionals who do the work of delivering a potentially releasable Increment of "Done" product at the end of each Sprint.

It sounds like your Development Team is delivering a potentially testable Increment.  

But a lot of this depends on how you have written your Definition of Done and whether your Development Teams starts working on the next iteration while the testing is occurring on the previous.  If that is the case, then you could argue that they are doing Scrum.  But I would expect that during the testing phases they are working on fixing bugs found.  That would lead me to deduce that the Increment was almost potentially releasable. 

I know this is a scrum.org forum but I suggest you take a good hard look at Kanban because it is an Agile practice that would fit better into your organizational structure. (Hope that last statement doesn't get me banned.)  If you really want to do Scrum, then your organization is going to have to change.  Remember that you can still do a 4 week release schedule as long as the teams are able to build potentially releasable increments each sprint. But you will have to incorporate testing into your sprint activities.  

07:37 am November 10, 2018

It seems like mini waterfall inside the sprint and at the end of each sprint there should be incremental release. Also I don't see sprint review or retrospective happening here.

03:14 pm November 14, 2018

As "potentially releaseable" is different from "definitely releaseable", and "release" can be different from "release to production", a UAT can be done outside the SCRUM framework.

Given a complex business software, if a feature is implemented and verified to be working without errors at the end of a scrum sprint, it does not say anything about if the sum of all features are suitable and sufficient for business operations. A UAT could be seen as a simulated production release for validation of the product's suitability for the business, in order to minimize risks when developing and releasing complex and business critical applications.

10:25 pm November 14, 2018

Tilmann, this may be true only if UAT is not a requirement for releasing to production.   Perhaps there can be a separate effort to vet the new functionality with a subset of users, to gauge suitability and sufficiency for business operations, but this only works in Scrum if this effort is not a pre-requisite for going live.

Scrum is quite clear that the goal of each sprint is to produce items of value that are suitable for production release.   Arguing that UAT can also be an end-goal for a sprint simply muddies the waters.

11:32 pm November 14, 2018

If there is any testing to be done then the work is not Done, and it remains work In progress. It doesn’t matter how close it is to production, such as a UAT environment.

This does not imply that work ceases to be tested once it is Done and released into production, rather from that point it will be in a fit state for evaluation under empirical conditions.

10:51 am November 15, 2018

As "potentially releaseable" is different from "definitely releaseable", and "release" can be different from "release to production", a UAT can be done outside the SCRUM framework.

Tilmann, your position is not really accurate. I'll copypaste from another thread where I said: "do not confuse the terms "potentially" and "definitely" in front of releasable. What I mean by that is, the increment produced by the team should be of production quality (per DoD), which means it should be of definite release quality, however the actual deployment to production environment is potential, for the PO (business) may decide to postpone release."

 

Timothy, re "this may be true only if UAT is not a requirement for releasing to production" have you (or anyone else for that matter) ever experienced a scenario where user accepted testing was not such a requirement? While I've got enough experience behind me, I don't know it all, hence my question.

11:02 am November 15, 2018

What is user acceptance?  Sometimes it's better to deliver something to the user and learn from their real-life, marketplace behaviour, than to give them the opportunity to not-accept something when they've only 'tested' it.

11:20 am November 15, 2018

Alex, I'd argue that sounds more like market feedback, not acceptance. Wouldn't you agree?

Most teams use acceptance criteria (regardless of whether these are written in a ticket, in Gherkin, etc) and these acceptance criteria have to be met in order for the story to be considered done. UAT means internal validation (not production one) and can be performed manually, by automated tools, or a combination of both. We surely don't want un-checked stories to be shipped to production and have the users UAT them, do we?

In some cases UAT is performed by the PO themselves in order to gain confidence and add an extra layer of quality security. 

11:47 am November 15, 2018

Let me rephrase my original question then.  What is the purpose of User Acceptance Testing?  Is it for the users to determine what is production ready?  Or is it to understand how the users' behaviour and validate/invalidate assumptions? 

If it's the former, does that sound fair and in line with Scrum?

If it's the latter, then yes, it does sound like market feedback.

 

01:22 pm November 15, 2018

Not sure I fully understand where you're coming from.

You do UAT internally to check whether the work is in agreement with the intended development (any variations/changes to requirements would have been handled one way or another, and result in an appropriate solution). So, from this perspective, the purpose of User Acceptance Testing is to verify whether the story, as thought of and as understood by the team, meets the criteria set forth for it to be deemed complete.

This is surely very different than having the very same story in production, subject to users' consumption - utilization - operation - call it what you want. This is the real market feedback (whether from external or internal customers).

02:07 pm November 15, 2018

So, from this perspective, the purpose of User Acceptance Testing is to verify whether the story, as thought of and as understood by the team, meets the criteria set forth for it to be deemed complete.

What separates this user acceptance to any other form of testing?  

My understanding of UAT has always been that it includes the user in some capacity.

09:54 am November 16, 2018

"What separates this user acceptance to any other form of testing?"  

Perhaps the fact it is supposed to be a rather exhaustive one, performed on behalf of (or by, in some cases) the stakeholders in order to gain confidence the development satisfies the wanted behaviour/requirements, there are (to the best of their understanding and efforts) no bugs (or if there are, they are so minor there's a chance they won't even be noticed by end-users), etc? In UAT you want to make sure that was was developed works flawlessly and the user will be able to use it (no bugs, no errors, no crashes). Therefore, User Acceptance Testing is performed, on behalf of the business, for the end-users.

As mentioned above, this is way different than end-users' feedback - hence the value that is yet to be proven.

Here's where a DoD (and a review/demo) come in handy. Would the business (PO) be confident of releasing to production a story that, although it passed dev checks and QAs, was not looked into by the stakeholders' representative (PO)? Or a BA if there is one? Or have a wide group of interested people (review) see it in action (albeit not in depth, cause UAT should be done before the review).

 

My understanding of UAT has always been that it includes the user in some capacity.

If by user you mean end-user (external customer) then the answer is that it usually doesn't. UAT is performed internally, by the business, from a end-user perspective. In performing the UAT, the business wants to gain confidence on whether the story is fit for the end-user's utilization.

____

The above represent my knowledge and understanding, and therefore constitute my practice. If someone has different views/understanding, please share your thoughts and arguments.

10:20 am November 16, 2018

Here's where a DoD (and a review/demo) come in handy. Would the business (PO) be confident of releasing to production a story that, although it passed dev checks and QAs, was not looked into by the stakeholders' representative (PO)? Or a BA if there is one? Or have a wide group of interested people (review) see it in action (albeit not in depth, cause UAT should be done before the review).

Are you not risking relinquishing the control of the Development Team to external parties by doing this? As in, if a PO and a BA or a wider group (any/all of which might fall outside of the Development Team) are needed to determine what is or what isn't Done, is it then fair to say the Dev Team has all the skill it needs to bring the Increment to a potentially releasable state?

The above represent my knowledge and understanding, and therefore constitute my practice. If someone has different views/understanding, please share your thoughts and arguments.

I might be wrong too! I guess this is why transparency over terminology is so important.  I wonder if by leaving out UAT entirely, the Scrum Guide is suggesting that such a process should be viewed from a different perspective.

 

10:41 am November 16, 2018

Are you not risking relinquishing the control of the Development Team to external parties by doing this? As in, if a PO and a BA or a wider group (any/all of which might fall outside of the Development Team) are needed to determine what is or what isn't Done, is it then fair to say the Dev Team has all the skill it needs to bring the Increment to a potentially releasable state?

I don't think so, Alex. Here's why (note I'm referring striclty to Scrum):

  • PO is part of the Scrum Team (and a BA can be, too, as long as they actively contribute to the current increment). 
  • There is no control over the DT, but an ongoing collaboration.
  • No outsiders would determine whether it's done or not. If there's a DoD in place, then that's all the guidance needed.
  • The DT can surely bring the increment to a potentially releasable state, however the PO, even if the increment satisfies the DoD and is potentially releasable, may still not be convinced of some stories in it, in which case they may delay release, ask for some stories to be rolled back from release and so on.
  • I've seen cases where everything went well up to the review, where, post demo, a senior stakeholder advised that the weekly release be put on hold due to their concerns over functionality (everything worked/looked great from our team's perspective, but said stakeholder was concerned over the functionality - financial industry btw and recommended we don't present it to customers)
  • Bottom line: UAT should be part of DoD, imo
11:10 am November 16, 2018
  • I've seen cases where everything went well up to the review, where, post demo, a senior stakeholder advised that the weekly release be put on hold due to their concerns over functionality (everything worked/looked great from our team's perspective, but said stakeholder was concerned over the functionality - financial industry btw and recommended we don't present it to customers)

This is exactly the problem I think introducing the concept of traditional UAT to a Definition of Done brings.  

By making acceptance (in this case by a senior stakeholder) something beyond the Development Team's control, they cannot fairly be held responsible for bringing something to Done, a potentially releasable state.  

I understand the collaboration has its benefits and would more than likely be useful for the Development Team in reducing the risk of creating an Increment which is not Done. If collaboration goes beyond that and becomes a formal step of approval, then the Development Team no longer has control over what's Done.

02:55 pm November 16, 2018

This is exactly the problem I think introducing the concept of traditional UAT to a Definition of Done brings.  

By making acceptance (in this case by a senior stakeholder) something beyond the Development Team's control, they cannot fairly be held responsible for bringing something to Done, a potentially releasable state.  

I actually didn't say that, Alex. If I lead you to interpret my words that way, apologies. UAT was part of a DoD, but there was no acceptance from anyone else in the above example. The functionality was done in acordance with the DoD, everything was right in every way possible. My point was that, even if a story passes UAT and is ready for release, it can still be parked (not released) for reasons outside the team's knowledge, hence the importance of not only UAT being done internally, but also of gaining as much internal feedback (through demo ideally) as possible.

 

Let me use another example to better illustrate some of the points above.

In that finance company I mentioned above, we weren't operating in a pure Scrum setup. We were very close to it, but also very far if you know what I mean. But some things we did do right. One of these was focusing on value and deliver in small bits, week after week. There was a very good collaboration between all team members and also with most outsiders. In our setup, the PO would always perform UAT against the work created by DT and reviewed by QA. The reason, in that case, was that because of the circumstances and industry specifics, we had to ensure the functionalities delivered to end-users were as neat as possible. Sure, there were bugs, there were live (production) issues, but that's not the point.

The point was that, the PO, a former trader, had to ensure the feature is fit for other traders' use, and it was as close as possible to "real-world" baptism. It surely would've been far from wise to skip UAT, and have customers do the so called alpha- or beta-testing. While some companies, in some industries, can afford to engage in a- or b-, it wasn't our case.

 

03:32 pm November 16, 2018

the PO, a former trader, had to ensure the feature is fit for other traders' use, 

And if it wasn't fit? What would occur?

04:12 pm November 16, 2018

There are many real-world examples, where SCRUM can be beneficial, but something like a UAT would not match into a sprint. 

As explained in another thread, life critical applications are a good example for this, like medical diagnosis software or airplane control systems. For a good reason, users of such systems would not rely on a few days of testing such a software. They would not give acceptance to go into production before it has been tested intensively in many simulated real-life situations, which in most cases would take much longer than a single sprint.

Getting the empirical feedback only after setting such systems into production could cost lifes in such cases.

So "potentially releaseable" of a done backlog item could mean "if released into production, it would work without errors and as specified", which in many cases is insufficient information for judging the risk of really releasing it into production.