This post is for anyone who wants to inspire, change or influence others through their efforts as professionals, with emphasis on the latter.
What is the optimal size of a team? Should teams be stable over at least several years, or is it a good idea they change at will? How should leadership interact with autonomous teams? Is scaling always a bad idea? What contexts are Scrum suited to, and which not? Of all the impediments that a team faces, which is the one to focus on first?
The answers to these questions depend upon what you’ve come to believe over the years. Perhaps you’ve read many blog posts about how horrible SAFe is, or how useless estimation is, or what the optimal size of a team is. You may have the practical experience to support your beliefs. Wherever they come from, your beliefs inevitably shape your behavior and decisions. And that is a mighty important realization when you consider yourself to be an agent of change. Because those beliefs may drive you to make or block decisions or suggest changes, that unintentionally cause harm to individuals, teams, and organizations.
What if you end up applying a methodology like Scrum to a context where it isn’t suited, and people leave the company as a result? What if you block attempts at scaling because you’ve read that it's always a bad idea, or your own experience with previous cases is mediocre. What if you recommend teams to stop estimation altogether and cause financial damage compared to when you wouldn’t have recommended this?
“Wherever they come from, your beliefs inevitably shape your behavior and decisions. And that is a mighty important realization when you consider yourself to be an agent of change.”
I believe that being a professional means that you recognize that you have an ethical responsibility towards your clients and teams to “do no harm”. This responsibility should naturally drive you to seek out evidence to support or reject your beliefs and to inspect the quality of that evidence. And you should be transparent about the kind and quality of the evidence you use to base your claims on.
I know this is hard. You may feel compelled as a professional to make strong claims because that makes you look resolved. On social media, in particular, strong claims and catchy one-liners draw more likes and clicks than a more nuanced view.
This post is a guide on how to bring a more evidence-based perspective to your claims. Where and how can you find evidence? How can you determine the quality of that evidence? And what should you do when there is no clear evidence? I think you should read this post, and take it to heart when you write content that is aimed at influencing others. I think you should read it, and take it to heart when your professional work involves changing others.
Good evidence is as free of bias as possible
There are many potential sources of evidence to support or reject your beliefs. You can draw from personal experience to conclude that some practice doesn’t work because you’ve never seen it work well (or the reverse). You can also draw from the opinions of thought-leaders, other practitioners, and people you hold in high regard. All these sources offer evidence.
But the question is; how free of bias are these sources? Thought-leaders often have a stake in promoting certain beliefs, so as to bring people to their classes, books, or products. Your own experience is probably colored by your beliefs, as confirmation bias drives us to more easily remember and recall the instances that match our beliefs. The experience of other practitioners is helpful too. But what if those practitioners are all part of the same “belief bubble” — when everyone is pro-Scrum, or against SAFe or against estimation, or exclusively Scrum Masters?
All evidence is biased to some extent. We can get philosophical on this and conclude that “evidence is a social construction” and “there is no objective truth”. But it still remains true that some evidence is more biased than other evidence. It is important for a professional to be able to distinguish between high-quality and low-quality evidence because the quality of the evidence should inform the strength of our beliefs, and the decisions we base on them.
The most reliable source for evidence we have is the scientific method. The reason for this is that it aims to understand the natural world through the systematic collection of objective evidence. The most unbiased way to do this is through double-blind experimental research with control groups. But since that isn’t always possible or feasible, scientists use a wide range of methods to gather and report evidence in the most objective way possible. The scientific method also requires researchers to be very clear about what they expect to find and to be transparent in how they gather and analyze the evidence to answer their research questions.
“That sounds good”, you might think, “but that kind of high-quality research doesn’t exist for our work with Scrum teams or as change agents”. This is a remarkably common belief in our profession. Fortunately, the facts are different. There are dozens of high-quality academic works to draw from, there are dozens of academic journals dedicated to (agile) software development (ASD) and change management. I quickly found hundreds of academic papers that investigated Scrum and Agile teams. So where do you find it?
“There are dozens of high-quality academic works to draw from, there are dozens of academic journals dedicated to (agile) software development (ASD) and change management.”
Where to find relevant scientific research
Google Scholar is a good starting point because it purposefully indexes academic journals. My favorite approach is to enter a search term I’m looking for (e.g. “software estimation”), and then select “review articles” under “type” on the left. This optimizes the search results for academic articles that summarize other research, which is often a good way to get a sense of the academic consensus.
A problem you may run into here is that many academic papers are only accessible to people from academic institutions. Although I haven’t tried it, this site may help you access those papers with little effort.
References in academic papers are also a good source. Academic authors usually explicitly reference where their ideas or claims come from. It is often helpful to identify and follow these references if you’re interested in them. I always find it most helpful to start with a review article, and then investigate their references more fully.
The dedicated platform ResearchGate is incredibly helpful for finding relevant research, connecting with researchers, and accessing PDFs. Although ResearchGate is a “gated community”, you can still browse a lot of their content as a “citizen interested in research”. You can only create accounts, and access the full platform when you qualify as a researcher.
Books can be a good source of scientific research when they actually offer evidence. Books can also be extended opinion pieces, where no actual data is presented. This isn’t a problem, and these books can still inspire, but they are not in themselves evidence. It is also important to consider the aim of the author. Some authors are merely proposing an idea and may reference only supporting evidence, or not at all. I think you should treat these books with a healthy dose of skepticism, even though the message might appeal to you. Other authors take a broader view and feature both supporting and dissenting evidence. In general, I find that books that are written by academics generally offer such a broad view as they have been trained in how to find, assess, and include evidence. This includes books like Superforecasting, Teaming, The Person and the Situation, and Organizational Culture and Leadership.
There are also dedicated institutes and organizations that summarize scientific research for professionals. ScienceForWork.com tries to answer common professional questions by reviewing available scientific research. There are some publications available at AgileAlliance.Org and Scrum Inc. We also try to do so for Scrum-related questions in our own Science section here on Medium. If you know more good sources, I’d love to learn about them — there are too few for my taste.
How to read and understand scientific research
Almost all scientific publications follow a similar format. This format is designed to offer maximum transparency as to how researchers reach their conclusions, and to allow other researchers to replicate the study if they want to confirm the findings for themselves. This is the general structure:
- All scientific papers begin with an “Abstract”, which is a super-short summary of the findings. This allows readers to quickly identify whether or not a paper is relevant or not.
- The authors set the stage by reviewing existing research around a topic and identifying one or more research questions and hypotheses. This section is usually called “Introduction” or “Related Work”.
- The authors explain how they tried to answer the research questions. This usually involves the methods they used (e.g. case studies, surveys, experiments) and the statistical methods used. This section is usually called “Method”.
- The authors then share the detailed results of their study, but generally without interpreting it too much. This section is usually called “Results”.
- The authors then discuss the meaning of their results in the section called “Discussion”. Here, the authors also determine whether or not the results confirm or reject their expectations.
- Finally, the authors summarize their findings and conclusions in a section called “Conclusion”.
Academic papers can be technical, dry, and lengthy, which can make them daunting to read. Not all academics are great writers. It is less important to write deeply engaging prose than it is to be clear and transparent. Academics generally also don’t write papers for a general audience, but for academic peers. Especially the “Method” and “Results”-sections can be littered with complex formulas, charts, and statistics and are very hard to follow when you’re not familiar with the techniques used. Fortunately, you can often get what you need by reading the “Conclusion”, the “Discussion” and the “Introduction”. Sometimes the “Conclusion” alone is already enough. It might be tempting to only read the “Abstract”, but most academics know that this is also where a lot of “paper marketing” goes on, and the claims are sometimes stronger than what is in the actual paper.
How to assess the quality of scientific research
Once you dive into scientific research into Scrum, Agile, software development, and related practices, you’ll find hundreds, if not thousands, of academic publications. That is great! And it can be confusing also because not all research is of similar quality. How do you know what to look for?
Academic research is generally published through academic journals. Some of these are well-known and broad, like Nature or Science, but most are not and are highly specialized (e.g. “ACM Transactions on Parallel Computing”). What distinguishes academic journals from regular magazines is that each submitted paper goes through a lengthy peer-review process. Each paper is read and reviewed by an anonymous panel of academics that is selected for a paper by the journal editor based on matching specializations. This panel determines if a paper is of high enough quality, and generally reviews whether the methods used by the authors are sound, whether their conclusions match the results, and whether or not the study builds on existing research.
Academic journals vary in how strict their peer-review process is. Some papers published pretty much everything that is submitted, while others have incredibly high standards that only the best papers meet (like Science). One way to figure out which journals are probably more reliable is to refer to an index like this one, and particularly the “impact score”. Each academic journal has such a score, regardless of their specialization, and it tells you how often the papers in this journal are referenced in other studies. Although “impact scores” are imperfect, they do tell you if other academics take the publications in a journal seriously.
One question I always ask myself when I read a paper is: “Where was it published?”. This is usually shown in the footer or header of a paper. Some papers are never published in academic journals and are simply shared online by authors. Other papers are submitted, but not yet peer-reviewed. Although these studies can be of very high quality, they haven’t been reviewed by other academics yet.
Finally, it is always good to keep in mind that the world is a complex place. When trying to answer the same question, different studies may reach different conclusions. Perhaps the studies used different samples, and thus found different things, or the studies used different measures. This is why it is important to recognize the notion of “scientific consensus”. Over time, as more studies come in, scientists will reach a consensus about what is probably going on. A single study that fails to find something does not automatically reject the findings of a dozen other studies that did. It is hard for a layperson to understand what the scientific consensus around a topic is. But one way to find out is to look specifically for recent “review articles” or “literature review” around a topic. In these articles, the authors purposefully gather and review the research on a topic to date and determine the consensus. The screenshot I shared from Google Scholar earlier in this post shows four such review articles for the topic of estimation in Scrum, for example.
What if there is no good scientific evidence?
Scientific research is an exercise in nuance. The world is always more complex than we think, and studies with oddball findings are common. You may also discover that many questions can’t be fully answered by reviewing scientific research. Then what? Does this mean all the effort is for nothing?
The first conclusion is you should nuance any claim that can’t be backed up by strong evidence. If you feel that “All estimation is wasteful in Scrum”, but you can’t find solid evidence to support that claim, it isn’t very professional to make that claim in the first place. The same goes for strong claims like “Scrum Teams should not use a Definition of Ready” or “SAFe destroys autonomy”. You can certainly still have the opinion, but it should be bookended with nuance.
The second conclusion is that you may be able to help scientists to try to answer that question with you. Or you can even perform the research yourself. When you’ve searched for scientific research around your question, you may have found the names of scientists who’ve researched this, or related questions. Scientists are always glad to connect with professionals, as it is often very hard to collect good evidence. Perhaps you can work together, like I’ve done with Daniel Russo, Ph.D. and others, to answer questions you have. Our professional community will be all the better for it.
Closing Words
In this post, I offered guidance on how I try to find evidence to test my beliefs about Scrum, teams, and change. I also admit that it takes more effort than to just write something that matches my opinion.
Although there is value both in opinion pieces and more evidence-based pieces, I feel that the balance in the professional discourse in our community has tipped too far to left. I also feel it doesn’t honor our professional responsibility to our clients and the people we intend to influence. If we aim to “do no harm” to our clients, shouldn’t we go out of our way to test that what we believe is true? I will certainly continue trying, and I hope you will too.