Unit 3

Summary of Key Arguments in this Unit

Argument 1 - Historically, governments’ approach to work has favored detailed up-front project management, and has pursued delivery that does not vary from the plan

For over a century most public servants in Western bureaucracies have been trained to plan their projects and budgets in clearly delineated project phases, with clear timelines and clear lines of accountability. These approaches to planning are often referred to as 'waterfall', even though that is a term primarily of criticism, rather than a badge many practitioners apply to their own work.

The ideals of control and clarity in government planning have been part of public management theory for a long time, sometimes baked invisibly into the command and control structures of bureaucracies, sometimes explicitly celebrated and encouraged through named initiatives such as POSDCORB or PRINCE2. Governments were driven towards these methods by historic failures that were ascribed to insufficient control over and discipline of public servants and their projects. Max Weber, for instance, emphasized the importance of individual public servants being assigned clear work packages, so that personal accountability for failure could not be avoided.

Waterfall approaches to public sector planning do not just emerge from public administration theory. Governments also have a history as key drivers of civil and military engineering and urban planning. The building of roads or structures demands and rewards extremely detailed up-front planning by engineers. Combining this engineering history with a theoretical tradition that rewards control and predictability, and then add in the pressures that arise from electoral competition, and it is easy to see why waterfall-like planning techniques have been the normal approach to project management in governments until very recently, when the riskiest decisions about technology projects (for example, type of technology, impact on people) cannot be made upfront as it requires empirical data. Furthermore, public sector planning in the Waterfall process is largely a form-filling exercise that can now easily be gamed (for example  by using AI).

Key reading for argument 1: Agile Methods: Fact or Fiction - Matt Ganis

Argument 2 - The waterfall tradition causes major delivery problems to almost all digital government activity in which citizens are key users.

Most government services are now critically dependent on software. This includes not just visibly 'online' services, like COVID-19 tracking apps or government websites, but also seemingly face-to-face services, like the provision of nursing or waste collection, all of which are facilitated by a vast array of digital systems.

Starting as early as the 1950s governments started to use digital technologies to support the provision of all kinds of public services. Their default model for developing and rolling out new services was waterfall, with large amounts of planning performed before the first line of software was written. So long as these systems were built to operate in very controlled and constrained environments, these traditional project planning mechanisms could work well - for instance in the provision of software for air traffic control systems, or space rockets.

By the 2000s, the public was starting to interact directly with government systems via the internet. At this point cracks began to show. Software projects in both the public and private sectors failed partially or entirely at a rate unknown to traditional built-infrastructure projects. At the same time, citizen expectations of fluid and easy to use systems were in stark contrast to the brittle and confusing interfaces that emerged from waterfall processes.

Ultimately the detailed up-front planning methods became fundamentally incompatible with the unpredictable complexities introduced by both complex computing systems, and by unpredictable human beings. A crisis point was reached.

Key reading for argument 2: The CHAOS Report 2015 - Standish Group

Argument 3 - Iterative approaches succeed more often in citizen-facing services because they accept uncertainty, rather than trying to eliminate it

Throughout the 1990s and 2000s governments suffered a relentless, embarrassing and costly series of 'IT disasters' that tarnished their reputations for competence, and provoked considerable introspection.

Looking for ways out of this bind, governments started to pay attention to the revolution in project management thinking that had started to take place within the software industry during the 1990s. The Manifesto for Agile Software Development, published in 2001 was not the first statement of new, fundamentally iterative approaches to project management, but it was catalytic in moving the ideas from niche parts of the private sector to a wider audience.

The key difference between iterative project management and waterfall project management is that the former expects projects to take place in highly uncertain environments (like an emergency room), and the latter in quite controlled, predictable environments (like a statistical office). This ability to learn and improve as projects go along turns out to be fundamental to most citizen-facing public services, where unpredictable problems are routine and unavoidable.

Argument 4 - Waterfall approaches assume that projects will be completed, and that development will then cease. Agile treats projects as if they will always be continually improved.

One of the most fundamental assumptions of government project management is that projects are initiated, delivered and then closed. This assumption is deeply baked into work that is delivered using waterfall approaches. For example within PRINCE2 (a popular project management methodology from the waterfall tradition) project closure is a specific phase.

Agile project management is based on the idea that most good, user focused services are never in any meaningful sense ‘done’. They are delivered on the assumption that there are always further ways that a service can be better for its users, and there is always the threat of obsolesce from rapidly changing digital technologies, and from shifting user expectations.  Consider how an ordinary website from 2010 is largely not fit for the smartphone users of the 2020s to see how rapidly this obsolescence happens.

This aspect of agile can be one of the most common ways in which iterative approaches can come into conflict with traditional public service practices. For example, budgeting regimes will often treat the costs of public service improvements as something that happens within a fixed period of time, with a beginning and an end. Shifting the norms from one-time, large projects improvement to continuous improvement may significantly challenge bureaucracies that do not have strong digital era leadership.