The defenses of bureaucracies as "rational processes" being posted here are, I have to say, somewhere between disingenuous and naive. The cynics are right here, and are in fact underestimating things.

Even with new digital governance tools and 2.0 organizational models, it is likely impossible to change things, but just because bureaucracy may turn out to be theoretically inevitable (in the "death and taxes" sense) does not mean we should condone it or rationalize it into something it isn't.

And this has nothing to do with whether you have an idealist or tragic (i.e. perfectibility vs. corruptibility of man) views. Bureaucracies are structurally pathological from the point of view of both the bureaucrats and those they govern, for very good reasons. The only party for whom a bureaucracy is a rational thing is the party that is above it, and "owns" it in some sense. In the worst cases, it benefits nobody at all, because it escapes the control of even those who designed it to serve them, becoming a non-human beast.

Here's how this works.

A bureaucracy is an institutionalized way to load-balance moral hazard, designed to give some people authority without responsibility, by taking it away from others. The net result is that the overall amount of moral hazard in the system goes up at "above the bureaucracy" loci, but the individual maximum goes down within the bureaucracy and among the governed, goes down. It is an amplified water-bed effect. The main side effect is that people who DO want to take responsibility for things within their scope of authority are discouraged from doing so.

At the risk of over-simplifying, the system is designed to let the C-suite legally get away with massive crimes by policing your ability to commit minor ones.

Another way to think of it is in terms of the fundamental object in bureaucracy: forms. A form is designed to protect those who must process it, not serve those who depend on it. A second layer of protection is added by locating the authority (and competence) to override every form in a third "above" location, and making access to over-ride processes punitively cumbersome a priori.

The nominal reason is to triage the natural variety of "cases" a process must handle. The actual reason is to enforce Procrustean uniformity on the natural variety rather than sort it, and only recognize the exceptional nature of things when they absolutely cannot be ignored. By defining normal variation as exceptional, and making exception-handling resources scarce, the unserved "exceptions" are reduced to a narrower, unnatural "normal."

Illusory business cases are made, to prove the cost savings of uniformity and standardization. These rarely hold up to scrutiny and true cost analysis. In some cases the argument is even interestingly circular: there are no actual economies of scale once the governance costs are thrown in, so it becomes a case of "we need governance to ensure uniformity, and we need uniformity to give the governance process something to do."

But in more complex cases, such circularity is invisible because a mature bureaucracy evolves to be opaque even to those within it (the technical term is "illegible" in the sense of James Scott, from Seeing Like a State). Governance costs are so diffusely spread over so much illegible messiness that computing the value of a single process becomes impossible. This is by design (though the design is not due to the bureaucrats themselves, except in rare cases, they are pawns who think they are actually adding value).

But you don't need to penetrate the opacity to understand things, because even a cursory analysis of organizational incentives reveals the true reason bureaucracies exist: it is to create a monopoly over risk taking at the very top.

For those at the top to have this monopoly, the rest of the system must behave predictably, even if sub-optimally, and at far higher cost. This is what I mean by the moral-hazard waterbed effect with amplification. If P is the total moral hazard in a system without a bureaucracy, and comprises P1 at the top and (P-P1) in the rest of the system, and Q (comprising Q1 and (Q-Q1)) is the state after putting in a bureaucracy, Q>P, but (Q-Q1) < (P-P1). You could say (Q-P) is a measure of the oppressiveness of the system.

To create this monopoly, certain risks are vastly exaggerated, and the systems and processes over-engineered to protect against them.

For example, nominally, bureaucracies are put in place as a check and balance against individual excess at the rank-and-file (such as individuals being nepotistic in hiring or corporate purchasing). Higher-ups are nearly always exempt, or expected to police themselves. There are two ways to police such rank-and-file behavior:

1. Trust by default, detect and punish offenders AFTER the fact. This is the sane way to the degree that the system is non-oppressive. The "innocent until proven guilty" model.
2. Distrust by default, punish everybody with burdens just enough that most offenses are prevented before they happen. The "guilty until proven innocent" model.

(this is of course the famous Theory X/Theory Y distinction originally proposed by Douglas McGregor).

To the degree that Q>P and Q1>P1, you must shift to model 2.

This just scratches the surface. I could (and someday might) write a book about this stuff. I haven't talked about why it is structurally impossible for both universities and corporations to study this stuff, creating a systemic self-blindness/incapacity for introspection. I haven't talked about how the assumption of a certain propensity to break rules becomes a self-fulfilling prophecy. I haven't talked about "work to rule" examples which show that nothing would get done if everybody actually played by the rules.

This is red pill/blue pill stuff. The deeper you dig, the more you are amazed at what you find. If you can step back from the frustrations to admire it, this stuff is really beautiful in some ways. Sometimes I feel like applauding. I actually have the outline of a mathematical model somewhere, that I haven't had time/funding to work on.

I won't say more... I've already cannibalized too much of a blog post in the works that many of my readers are waiting for.