Scrum for R&D Software Development

Do you mean Research & Design or Research & Development?

I mean Research and Development for a software product we are developing. I guess there is a bit of design involved as well.

I mean Research and Development for a software product we are developing. I guess there is a bit of design involved as well.

There's no reason to assume that "Scrum does not work" in this context. Scrum may be seen as an empirical approach in which validated learning is important

When you were told that Scrum does not work for R&D, can you explain the reasoning behind this assertion, or why you suspect "modification" might be necessary?

Should work very well if not ask team why they are not satisfied, what do they expect ? Try KANBAN maybe.

I would think that Scrum works really well for Research and Development. 

• You state a problem to research. 
• You refine the story into workable experiments. 
• You start the experiments. 
• Evaluate the results as you go and at each sprint boundary.
• Determine the next steps based on what you just learned.

Isn't that pretty much the definition of Research and Development?

Research and Development is harder with scrum because the product backlog is harder to categorize and prioritize when the result of a given research project may be unclear or take unbounded amounts of time to complete. In my experience most engineering teams are pretty used to a sequential approach along the lines of: design, implement, test, measure. Each step typically has a pretty predictable amount of time to commit. Whereas research has a success vs time tradeoff; often times success is possible by committing more time to research. But this isn't very agile. Furthermore if product doesn't have a deep understanding of the research subject, it can make it hard to communicate what's possible and how to prioritize without selling exaggerations. So how do you make a product backlog when the downstream usability is unclear? Furthermore how do you make a product backlog when success is not guaranteed or the time to success is unknown? And if your product manager is not a researcher in your field, how do you decouple the researched technology from an actual product increment?

As an initial process, I've broken my R&D projects up into several steps, each one of which can result in a "don't pursue" decision:

1. Hypothesis testing: gather the initial data to suggest that your hypothesis might even be possible to validate
2. proof of concept: often times by providing an initial mock-up 
3. Scientific process: generate a hypothesis, ask a research question (probably involving company KPIs), propose an experiment, 

Step 3 is essentially a normal scrum/product process. Asking a research question can be worded in terms of a user story. This is the point where we converge from a typical research process to a typical engineering process. Once we have a proof of concept we have a better idea of how the technology works and can do our best to categorize it's utility and potential failings so that product can better understand how to 

The hardest part of this whole thing is resisting the sunk cost fallacy. Once your developers have spent so many human-hours performing steps 1 or 2 it may seem desirable to spend time on step 3 for a lack-luster product output. Don't do it. More than the typical cost-benefit analysis of hours spent for value generated, there's additional costs of technical debt of your product, and user cognitive load to having extra features that don't deliver too much value. Hopefully your engineering leads and design leads will support these last statements! This is why it's so important to make the emotionally difficult decision to not continue a research project early on if it doesn't look promising immediately. (And let's avoid that "don't be emotional" narrative too: acknowledge the emotion upfront and move on when it's prudent)

Great reply by @Stefan Sullivan,

I just signed up to thank you.