Skip to main content

Data Integration development and Scrum

Last post 07:40 pm February 6, 2023 by Kevin Kunch
6 replies
09:30 am July 26, 2019

I work as a BA within a team that predominantly works on Data Integration activities.  We have a vendor system, into which we load different types of data, from a variety of sources.  We validate the data and create a golden copy.  We export data from it, with differing content and formats,  it to a variety of target systems.

Traditionally the BA’s have written mapping documents (using Excel) to describe the data import/export requirements.  These documents are fairly tightly controlled and aligned to the software release versions.

As a team we have recently moved to a Scrum way of working and User Stories have been developed which include the mapping document.  To keep the stories small they are not entirely independent; there is a basic story and then subsequent refinements.

Not a great example but here goes:

Story 1: Send of file of all invoices to system Z.

Story 2: Only send invoices from a subset of suppliers to system Z.

Story 3: Only want to send invoices that haven’t been fully paid to system Z.

Story 4: Only send an invoices over a certain value weekly, all other are sent monthly.

You get the gist.  So the mapping document would normally include these additional criteria specified in stories 2,3 and 4 but I do not want to include these in the mapping document for Story 1 but I also do not want to have 4 versions of this mapping document that I need to keep updated (for example an error in version for story 1 would have to be corrected in four versions).

How would others approach this issue ? I’m all for following the “working software over comprehensive documentation” paradigm but our dev and test teams are difficult to budge.

03:46 pm July 26, 2019

Why do you think any of these constitute user stories? How do they promote discussion, and the emergence of scope? What is the problem with the old way-of-working that you are trying to solve, and why do you think Scrum might help?

08:03 am July 29, 2019

To be fair, the decision to use Scrum as a way of working is decision that was imposed on the team and I'm not sure exactly what problems were being targeted. Sadly with all the Agile hype it's seen as a magic bullet and also a means of furthering one's career by having a CV full of Agile projects. Having used Scrum elsewhere at another organisation, I'm not convinced that it is appropriate for this type of work.

However this is what I'm faced with and so I'll press on.

08:14 pm July 29, 2019


One of the popular guidelines to creating PBIs is that each item should be able to stand on its own (independent).   Does attaching Mapping Documents, instead of documenting the requirements and acceptance criteria in each item,  support the independence of each PBI?

I see nothing wrong with your 4-story example.   In my opinion, you seem to be creating unnecessary complexity by attaching common mapping documents (requirements) to multiple PBIs.   

Perhaps any proposed Mapping Document change should be managed by a separate PBI that pertains specifically to that change?

08:47 am February 3, 2023

So glad to hear about your team's journey towards Scrum for Data Integration. I'm also wondering where I can get the best data integration tools? Tnx!

03:45 am February 4, 2023

Hi @Peter Davis,

I don't see any problem with the way the Stories have been created by your team. What your team is actually trying to do by way of this is to build the mapping document 'incrementally' - one brick at a time. There is definitely value in doing this because team would be able to gather feedback fast on completing the first Story (a catch all Story) and then build upon that as part of subsequent Stories.

And in case there is any defect in Story 1 (which I ideally believe should be worked upon before starting work on Story 2), even if the rest of the Stories are completed, change would only need to be done in Story 1 & not the rest of them, as long as the filter criteria for all the rest of the Stories remain valid.

The important thing to consider here is that the purpose of splitting PBIs into smaller units is not only that it should be completed within a Sprint, but also to speed up the feedback cycle on each of them. By splitting the Stories in 4 parts, you're actually speeding up the process of gathering feedback in each of them individually, rather than waiting for the last one to be completed (final brick laid) and then gather feedback on the whole.

03:48 pm February 6, 2023

Great to see the team is adapting to Scrum for Data Integration development. I get the challenge of incorporating additional requirements from the User Stories into the mapping documents. Have you considered using a data integration tool to streamline this process? 

By posting on our forums you are agreeing to our Terms of Use.

Please note that the first and last name from your member profile will be displayed next to any topic or comment you post on the forums. For privacy concerns, we cannot allow you to post email addresses. All user-submitted content on our Forums may be subject to deletion if it is found to be in violation of our Terms of Use. does not endorse user-submitted content or the content of links to any third-party websites.

Terms of Use may, at its discretion, remove any post that it deems unsuitable for these forums. Unsuitable post content includes, but is not limited to, Professional-level assessment questions and answers, profanity, insults, racism or sexually explicit content. Using our forum as a platform for the marketing and solicitation of products or services is also prohibited. Forum members who post content deemed unsuitable by may have their access revoked at any time, without warning. may, but is not obliged to, monitor submissions.