Approach Experimentation with Purpose
Scrum Teams treat feedback and data as a neutral and objective input, not a tool to confirm biases. By basing their decisions on objective results rather than optimism or personal opinion, Scrum Teams ensure they are continuously learning and adapting based on what the data actually reveals, not what they hope it reveals. This mindset reinforces empiricism, and encourages experimentation as a cycle of inspecting and adapting through deliberate testing of ideas.
Experimentation
Experimentation is an ongoing practice of trying out ideas, testing assumptions, learning from results and adapting based on evidence. This process supports continuous product improvement and product discovery. It helps reduce the uncertainty of risky assumptions about what actually works for users and a product.
Scrum Teams should define clear, measurable outcomes for their experiments. This helps to determine what kind of evidence they need to validate or invalidate a hypothesis. A hypothesis should make it clear what success and failure look like. If the data supports the hypothesis, it may be worth continuing to explore the idea further or even scale it if evidence and confidence are high. If it doesn’t, the signal is just as important. It is probably time to pivot, abandon the idea, and move on to the next important item on the Product Backlog. This empirical mindset embraces failure as part of the learning process and helps Scrum Teams avoid sunk-cost fallacy and deliver products that no customer wants to need.
Understanding What to Validate and When
Not all ideas are created equal, but without some level of experimentation, it is unclear whether an idea is valid or not. That being said, ideas do not all need the same level of experimentation. Product Ownership demands clarity on what needs to be validated, how quickly it needs to be tested, and what level of uncertainty justifies running an experiment. For example, in the early stages of a new idea, when the Scrum Team has little to no evidence, it’s often more valuable to “move fast and break things,” running lightweight, rapid experiments to get insights.
The goal is to match the level of validation to the risk involved. Riskier assumptions need more deliberate, well-designed experiments. Simpler or lower-impact ideas can often be validated with smaller tests or observational data. A great tool that might help you determine the level of experimentation is appropriate is the Truth Curve.
Defining Experiments as a Core Practice
An essential responsibility of the Product Owner is to collaborate with other Scrum Team members and stakeholders to design experiments that bring clarity to ambiguous product ideas. This involves expressing:
- What value or risk is being tested
- What assumptions are being made
- What experiment will be run to test those assumptions
- What success or failure looks like, based on measurable outcomes
By formalizing experimentation in this way, Product Owners create transparency, reduce emotional attachment to ideas, and foster a culture of learning.
Applying Appropriate Experimentation Techniques
Different product decisions call for different experimentation methods. A/B testing may be suitable for fine-tuning UI changes, while user observations and interviews may be more appropriate for exploring entirely new value propositions. Considerations when selecting experimentation include:
- The speed of getting results
- The confidence level required for decision-making
- The effort and costs needed to run the experiment
- The ethical and practical feasibility of the test
Whatever experimentation techniques, the aim should be to make product decisions with higher confidence and less waste and ultimately deliver products that are truly valuable for customers.
Resources:
