Scrum for data projects

The Scrum Guide itself says:

Scrum was initially developed for managing and developing products. Starting in the early 1990s, Scrum has been used extensively, worldwide, to:

1. Research and identify viable markets, technologies, and product capabilities;
2. Develop products and enhancements;
3. Release products and enhancements, as frequently as many times per day;
4. Develop and sustain Cloud (online, secure, on-demand) and other operational environments for product use; and,
5. Sustain and renew products.

Scrum has been used to develop software, hardware, embedded software, networks of interacting function, autonomous vehicles, schools, government, marketing, managing the operation of organizations and almost everything we use in our daily lives, as individuals and societies.

So, yes. Scrum, as a framework, can be used in things other than software development.

Just based on this, it seems like you may have some misunderstandings about what is Scrum. User stories, for one example, are not part of Scrum. Scrum talks about a Product Backlog, a Sprint Backlog, and backlog items. User stories are one common representation of backlog items, but they aren't the only valid way. That's true of many of the things in Scrum - there are plenty of implementation details that are left to the team or organization employing Scrum.

I understand that user stories by itself are not part of scrum. But at some point sprint backlog items are broken to user stories for teams to work on. I feel it is much difficult to break down to user stories for data projects compared to a web application. Are there any case studies of successful data project implementations using scrum. 

Is there a product for which valuable feature-complete increments can be released to stakeholders sprint by sprint?

Hi S K,

I have been product owner of a centralized data integration solution. It is very much possible to work in a scrum / agile way in this area, but whether it is useful depends on what you want to achieve. For my time as product owner the situation was that data quality was poor, so it was important to get feedback on this quality of the data consumers (my stakeholders) as soon as possible. And of course for the most important data first. So in this way, you can define stories based on value.

Also, when requirements change regularly (in the data field, mappings from one model to another), working scrum is very useful.

A situation where more waterfally way of working could work, are situations where value is only delivered when a full data project is finished (for example a regulatory reporting thing), data governance, quality and models are sharply defined, and so it makes more sense to do detailed design, mapping, implementation, testing and releasing. But to be honost, I hardly doubt that this is a realistic situation.

Further, tooling, for managing backlog, roadmap (important for managing your stakeholders) and sprint backlog/kanban board we moved from Jira to Priooo. We released data integration jobs with release scripts to production. Automated testing partially implemented using Jenkins.