"You can't predict anything. How can you be certain about anything when everything is chaos and we're not in control?"
-- John Lloyd
Introduction: From Chaos to Control
Voyagers of old, like Columbus or Marco Polo, would chart their journey in terms of the extraordinary. Theirs was a geography of punctuating events. As they navigated the deep and dark ocean, or plowed the trade routes in their caravans, it was encounters with strange lands and peoples, and new wonders and remarkable beasts, which served to demarcate their travels. We can presume that those visited were just as surprised as the visitors.
The career of a Scrum Master can also be charted in terms of its jaw-dropping moments. That is the geography to which we become accustomed. Watch the jaws drop, for example, when we talk about learning to build the right thing at the right time, rather than somehow doing projects faster and cheaper. The viziers of an organization can seem uneasy with this perspective: it threatens to upend their very understanding of common sense, and their assumptions about what agility surely ought to mean. We in turn find ourselves baffled at such a hostile reaction to the innovation they demand to master, and yet simultaneously reject as unmanageable. We bring an incompatible world view to the emperor's court.
Agile transformation involves changing an organization's culture, such that the leap-of-faith anyone must take before outcomes are evidenced is reduced. Each Scrum Sprint provides an opportunity to inspect and adapt, and rapid feedback is obtained as a matter of course. Empirical process control is thereby baked in to agile practice from the very beginning, as part of the deal.
In most companies an empirical approach is resorted to only in times of crisis and considered tolerable only once a project begins to fail. Empiricism is something to be done summa desperatione...basically, when there is no choice but to inspect and adapt your way off the hook. Ironically once the situation stabilizes, the cultural imperative is to then revert to the established practices which brought grief in the first place. There is a status quo to be maintained even in how exceptions are handled.
This post is the first in a 6 part series of organizational discovery. By considering the "punctuating events" most commonly experienced in agile practice, we will follow an employee's journey from a chaotic project environment to a new culture of measured and managed innovation. We will see how empiricism can be established under conditions of high uncertainty, and gain an appreciation of the new common sense upon which empirical practices are founded.
The feeling of chaos
Actually, I suspect you already know what chaos is, or rather what it feels like. Chaos is when you turn up to the office -- say, eight in the morning -- and before you can get your coat off they're already at you. "Can you just do this? Can you just do that?". Your whole working day is spent firefighting, playing whack-a-mole, and re-actively trying to deal with the various issues which keep cropping up. Then, maybe around 5pm, you can get back to your desk and make a start on the things you'd hoped to actually do that day.
There's no real mystery to the situation. Work previously undertaken is not in the state we thought it was. Shortcuts have been made and the necessary quality isn't there. Quality easily becomes the silent casualty of the traditional scheme where time, scope, and budget are all meant to be fixed. It might actually be possible to get away with cutting quality for a while, but eventually the problems surface. Defects will be raised or work discovered which is not truly fit for use. You might then try, re-actively, to fix the situation, but without the time or budget to do so. Further shortcuts are made in a cascade of poor remedies, and what was once a complex yet manageable situation descends into a doom loop of chaos.
Many projects can feel chaotic and overwhelming. Engineers in particular value certainty and control, and so a product team may struggle with a creative and frequently changing commission that is filled with unknowns. They sense they are in a reactive mode, dealing with shifting requirements and playing whack-a-mole with unforeseen issues. There are too many urgent decisions to make and not enough time or information to make them well. Cause and effect are unclear, and the team cannot predict which immediate remedy is most likely to succeed with stakeholders.
Recognizing chaos
The one constant experienced in a typical project is uncertainty. Every day people find themselves reacting to whatever happens, just like everyone else on their team. New problems keep surfacing seemingly out of nowhere. It becomes next to impossible to hold a sensible meeting, because the people you need keep being pulled away to handle some fresh crisis. What really proves wearing is that everything is treated as "top priority"...which in practice means nothing truly is. Progress stalls.
Here are some of the symptoms you may recognize:
- Diluted focus: Attention and effort get split across too many “important” things, so nothing receives the depth needed to be done well. Work becomes movement without meaningful progress.
- Lower productivity and quality: Constant task-switching to juggle multiple “top priority” items reduces output and increases errors; losses of 20% or more can be expected due to the switching of context.
- Chronic stress and burnout: If everything is urgent, people remain in firefighting mode, overwhelmed and exhausted, leading to burnout and high staff turnover.
- Wasted resources: Time, money, and energy get poured into tasks that they are told are urgent but don’t actually move an organization towards vision and goals. The result is “resource dilution,” where reactive initiatives are sponsored, but none are properly funded or staffed to the point that systemic improvement is possible.
- Poor decisions and loss of judgment: When importance is inflated, engineers stop asking why something matters and focus only on how fast to deliver, and quality suffers.
- Organizational chaos: Dependencies pile up, decisions stall, and managers lose visibility over what is going on. Dashboards and status reports become conflated with outcomes, and yet amount to little more than noise in the machine.
When estimation and forecasting seem impossible
The meetings held in such an environment often prove ineffective because team members are constantly pulled away to deal with emergencies. Everything is treated as a priority, and consequently there is little meaningful progress. The team cannot estimate timelines or make reliable forecasts, and so a manager is left unable to plan, lead, or provide reassurance to stakeholders. With no stable plan and no clear pattern to events, both the team and stakeholders become increasingly anxious and uncertain about outcomes.
Team members will struggle to estimate how long the work will take, or to predict any realistic dates. There is no time to actually change, or even perhaps to process and internalize the things that are going on. Employees will wonder how they can satisfy their roles and accountabilities when planning and control are effectively impossible. The sands are shifting with no clear pattern or direction. Rumors replace data, and politics overtakes engineering. How can clients be reassured when there is no certainty to offer? How can the company expect the team to commit to outcomes when they can’t build a plan that survives even one day?
Managing complexities of exactly this nature is the sweet-spot of Scrum. Enough needs to be understood and base-lined to frame a modest commitment each Sprint, and enough stability is then maintained for the commitment to be met. Volatility continues to swirl around...but it can be handled. Contingency is planned in, but it lies in revising scope rather than in cutting quality. The completed work will always be Done and in a usable state. The gravity well of doom is skirted around and avoided.
Next
In upcoming posts we will follow the journey of a project manager from chaos to control. We'll see how a forecast of work is made to meet a Sprint Goal, and how Done increments are built that validate our assumptions under conditions of high uncertainty. We'll see how to commit to goals rather than to forecasts of work and how to handle the matter of contingency, revising our forecasts as we learn, and as moles continue to be whacked.