The Gatekeeper Blog
Insights into the chief of staff role

Adam Smith’s “invisible hand” famously guides market economies, in which unplanned transactions by various agents lead to collectively beneficial outcomes. This logic governs most interactions between enterprises and consumers and between enterprises today. But work within enterprises is organized mostly according to the very different concept of bureaucracy, the principles of which were codified and popularized by a later philosopher, Max Weber. In this model, activities are planned, mandated, and guided by the very visible hand of organizational hierarchy.

HOW BUREAUCRACY WORKS

According to the transaction cost theory of the firm, companies exist because they can eliminate certain transaction costs that would otherwise be incurred in free markets, such as the effort required to discover prices or renegotiate contracts. As a result, companies can often better coordinate tasks that require a high degree of alignment between multiple parties or involve nuance or context. For example, car manufacturers typically outsource the production of car parts, because the exact shape and functionality of each part can be specified in advance. In contrast, car assembly typically happens within a firm, because the assembly process has many highly interdependent steps and can therefore be error-prone and require nuanced judgment.

Bureaucracy, according to Weber, emerged as an organizational form to create stability and predictability, making enterprises more efficient. In particular, he identified six essential characteristics of bureaucracy that set it apart from other forms of organization:

1. Division of labor with clearly defined roles
2. Hierarchical management structure with clear lines of authority
3. Documentation that specifies required decisions and actions
4. Specialized training and meritocratic selection for each role
5. Full-time managers appointed to operate the organization
6. Static, depersonalized rules that exhaustively guide management

Most large companies today fundamentally fit Weber’s definition of a bureaucracy. They task employees with job descriptions to facilitate the division of labor and arrange them in multilayered hierarchies through which directives are transmitted, executed, and monitored. Hiring and promotion decisions are based on objective criteria specific to each role. And extensive and relatively static rule sets authorize some actions and forbid others, giving only limited leeway for judgement and initiative based on circumstances.

This organizational form thrived in the stable, predictable environments in which most businesses historically developed. Hierarchical management enabled detailed planning at all levels of the organization; clearly defined roles allowed tasks to be decomposed and executed efficiently; and comprehensive, unchanging rules allowed new hires or resources to be easily integrated into the organization in pursuit of economies of scale.

 

BUREAUCRACY’S STAYING POWER

Despite the model’s popularity, the effectiveness of bureaucracy has increasingly been called into question because of structural changes in the business environment. Uncertainty is rising, disruption is increasing, and what it takes to succeed is changing faster than ever. As a result, companies increasingly must compete on adaptiveness, learning, and innovation. But bureaucracies are inherently ill-suited for these new imperatives: static organizational rules inhibit adaptation; top-down hierarchies are predicated on forecasting and planning rather than experimentation and learning; and highly codified tasks often do not leave sufficient room for discretion or imagination. Indeed, bureaucracy’s demise has been predicted for decades: “The conditions of our modern industrial world will bring about the death of bureaucracy,” wrote leadership scholar Warren Bennis in 1966.

Why, then, has bureaucracy remained the dominant organizational paradigm? The bureaucratic model is simple and well-codified, allowing it to be easily understood and applied by managers in any industry. The things it does well—such as planning what needs to be done, decomposing this objective into specific actions, and coordinating those actions among employees—are still valuable in many situations. Bureaucracy is also a convenient mental model that is often employed even if the actual network of the organization departs somewhat from the formal model.

Some companies have boldly experimented with alternative organizational forms, moving away from the planned, top-down nature of bureaucracy. (See the exhibit, “Alternative Organizational Models.”)

These philosophies share some characteristics, including a focus on avoiding top-down hierarchy and allowing teams to self-manage, with the aims of increasing adaptiveness through decentralized action and increasing innovation through greater employee engagement and motivation. Zappos has gone further than most companies in breaking free from the restrictions of bureaucracy. By fully empowering each team to determine the value propositions it offers and make its own investment decisions, the company effectively brings market forces inside the organization. (See the sidebar, “Zappos’ Journey to Bring the Market into the Organization.”) However, such alternative organizational models are still not proven, scaled, or codified to the degree necessary to make them accepted alternatives for mainstream companies today.

So is bureaucracy’s dominance here to stay, in spite of its frequently heralded demise? We believe not, as further changes in the business environment will likely push it past its breaking point. Uncertainty and dynamism continue to rise, which puts an even higher priority on learning, resilience, adaptiveness, and innovation. In addition, AI’s rapidly increasing power presents new challenges, because organizational models must encompass both human and algorithmic decision making. Technology also expands the range of possible organizational solutions, because algorithmic decision making allows activities to be coordinated on much shorter timescales and across a much wider set of participants (within and beyond the company).

Leaders must therefore reconceive the organization—from a static hierarchy and rule set to a continuously evolving model; from being human-centric to encompassing both humans and algorithms; and from being contained within company boundaries to encompassing connections and activities with external partners. They also need to reframe how their enterprises compete—from serving a relatively standardized and static set of offerings more efficiently to competing on the rate of learning, in order to discover and act on new opportunities. For that they will need a new generation of organizational models, which we collectively call the “hybrid learning organization.”

Although we don’t have a blueprint for the hybrid learning organization, we can already identify several emerging design principles:

• Integrated data systems, data communications, and decision engines that allow routine decisions to be made at algorithmic speed
• New human ways of working that foster imagination and higher-level cognition, rather than rote decisions and actions, which will increasingly be handled by AI and automation
• Data-driven feedback loops to facilitate learning on multiple levels—from product offerings, to culture and organizational models, to business models, to the learning approach itself
• The ability to operate and learn at multiple clock speeds—from split-second algorithmic timescales to the decadal timescales of social and ecological change
• The ability to be ambidextrous, balancing exploitation of current models with exploration for new ones, developing a mosaic firm that uses bureaucracy when appropriate but also employs more dynamic organizational forms when needed
• Seamless coordination with both internal and external stakeholders, through digital platforms and multicompany ecosystems
• A dynamic organizational model that is continuously adapted and refined based on the context

Perhaps not surprisingly, the first glimpses of organizations that embody such principles can be found in some technologically advanced companies. Some of the new ways of working can be seen at Zappos, as described earlier. Alibaba has developed the “self-tuning enterprise” concept, in which algorithmic learning principles are applied throughout the organization. Its decision engines autonomously experiment, modulate, and improve over time, relying on information from the company’s vast data ecosystems, and decentralized teams are free to trigger new initiatives when they see market potential. Whereas most organizations have a static vision and organizational structure, Alibaba continuously evolves its vision and organization design. Amazon (Zappos’ parent company) demonstrates similar elements, such as data systems that are interconnected throughout the organization, allowing all parts of the company to react to new information without guidance from a centralized authority. Employees often take a “hands off the wheel” approach, validating algorithmic decision processes and setting guardrails rather than intervening directly, which allows them to focus on more creative efforts such as imagining new businesses and business models.

 

The challenges facing businesses today are evolving rapidly, yet most firms are still organized along the entrenched 19th century paradigm of bureaucracy. Recognizing the new imperatives they face, winning organizations will embrace bureaucracy where useful but also boldly experiment with new organizational models that harness both technology and human ingenuity where needed. The exact shape of these new models is still undetermined, but enterprising leaders are currently developing them. In some of the companies described in this article we can see hints of a revolution that will likely affect all companies and sectors eventually.

The authors thank Tony Hsieh and his colleagues at Zappos for allowing them to closely observe and discuss the market-based dynamics model. The views in this article are the authors’ own.