Agile and Data Warehouse

Last post 05:50 pm December 15, 2020
by Narayan Krishnan
4 replies
Author
Messages
01:57 am December 10, 2020

Anyone here who uses on Agile on Data Warehousing projects? We use Informatica and the dev time for some of the complex workflows takes longer than what can be completed in 2 weeks. Completing it in small parts doesn't work since the whole thing needs to be completed to be able to test. Looking to see if anyone else has any experience in this area.

 

Thanks in advance

05:22 pm December 10, 2020

From the current version of the Scrum Guide's section describing the Sprint

They are fixed length events of one month or less to create consistency. 

Two week Sprints are just a common practice but there is nothing to say you can't go longer or shorter.  If Sprint length is the only thing holding you back, you have options.

05:56 pm December 10, 2020

We use Informatica and the dev time for some of the complex workflows takes longer than what can be completed in 2 weeks. Completing it in small parts doesn't work since the whole thing needs to be completed to be able to test.

Scrum is for developing and sustaining complex products, where more is unknown than is known. Each Sprint is an opportunity to mitigate uncertainty and risk. How about empirically testing some of the assumptions that go into those "complex workflows"?

03:10 pm December 11, 2020

I'd approach this situation like this:

  • First, I'd consider what Ian Mitchell said. Is Scrum appropriate for your context? Scrum is best suited for environments that have higher amounts of uncertainty, ambiguity, and risk and the practices that are part of or enabled by Scrum allow you to mitigate these issues. However, if your effort is clear, obvious, and well-defined with a lot of constraints or little room for freedom, then Scrum will likely add overhead. That doesn't mean that you can't get something out of some Scrum practices, but the framework as a whole probably isn't that useful.
  • Second, I'd consider what Daniel Wilhite suggested. Scrum allows for Sprints of up to 1 month. If Scrum is appropriate for you, can you deliver something in a month or less? Can you use that thing that you make to get feedback. I'm not of the mindset that you need to deploy your changes every Sprint, but you do need to be able to show what you're working on to stakeholders, get confirmation that what you are doing is correct, and realign on the desired outcomes and the path to get there. Scrum gives you the framework to do that.
  • Third, I'm unconvinced when someone tells me that the whole thing needs to be completed in order to test. Nearly every complex problem has process steps and you can test, demonstrate, and get feedback on steps to make sure that you are solving the right problem in the right way. Perhaps everything needs to be completed to make it worth deploying, but that doesn't mean that you can't get intermediate feedback along the way.
04:08 pm December 15, 2020

Thanks for all the feedback. I really appreciate it.