Lakhasly

Foundations of Collaborative EA 6
CONTENT
Reflections on Complexity 139
Beyond Threshing Machines 139
Structure and Behavior of Complex Phenomena 140
Principles of Managing Complexity 141
Management Capabilities of Hierarchies and Networks 146
The EA Dashboard as a Yardstick for EA Effectiveness 152

Despite the well-thought-out repertoire of frameworks and best practices set out in the previous two chapters, enterprise architecture in many organizations is far from having the sustainable and sweeping effect that it promises. Our caricatures way back in Chapter 1 outlined the extremes to which the idea of managing EA often deteriorates. But how can we do better and avoid these extremes?
The key success factor is to what extent a collaboration on EA between all stakeholders comes alive. The often-recited mantra—that the executive board’s commitment to an EA initiative is the de- ciding momentum—falls short of the mark. If an EA initiative is running into a dead end, it is mostly the breach between strategic vision and ground-level reality that makes it go astray.
The commitment of the CEO is a necessity, but it’s not a sufficient condition for closing this breach. It fails to be a substitute for a living, self-dependent collaboration. With commanded EA, people must collaborate, because the chief is saying so. But as soon as high-level management attention goes away, EA again becomes stale unless it is valued and supported by the ground-level personnel involved in designing, developing, operating, or simply using IT. Hence collaboration is the leitmotif for a better way to practice EA.
But how can we elicit collaboration and apply it as an antidote to a stale EA? Our antidote has three guiding principles as ingredients, as listed in Figure 6-1.
These three guidelines can be mapped to the toolkits set out in the subtitle of this book: lean, agile, and Enterprise 2.0, respectively.
• Lean opens our eyes to all kinds of waste, such as piles of unread specifications or extra processing due to following bureaucratic governance. This gives us a systematic way to eliminate waste to

Collaborative Enterprise Architecture
© 2012 Elsevier Inc. All rights reserved.

137

138 CHAPTER 6 Foundations of Collaborative EA

FIGURE 6-1

The three guidelines of collaborative EA.

come to a streamlined, demand-driven delivery process. Pragmatism, reduction of bureaucracy, and lightweight processes are what lean techniques subscribe to.
• Agile teaches us how to approach a problem in an iterative manner. It allows us to learn with a managed trial-and-error approach that has short-term feedback cycles, which is characteristic of an evolutionary problem-solving strategy. The feedback cycles bring all stakeholders together with a constant heartbeat, thereby guaranteeing collaboration.
• From Enterprise 2.0, we import insights and practices that help give birth to a living community of knowledge workers. This in particular avoids the notorious breach between high-level vision and ground-level reality.
But what makes us believe that the three guidelines will do the job? The rationale comes from both foundational and hands-on arguments. One foundational argument is that any enterprise architecture is beyond doubt a complex being and must not be managed as though it were simple. Applying manage- ment practices that rely on predictability, up-front construction, or detailed plans of action is bound to fail when they’re applied to complex systems.
Furthermore, we’ll discover that a network of peers under suitable circumstances has a higher ca- pability to shape and control a complex system than a command-based hierarchy (we will come back to both terms and discuss them in greater detail). This is the abstract root reason that a well-organized collaboration on EA matters in a network of stakeholders, has a deeper impact on the enterprise archi- tecture, and can shape it more purposefully than a hierarchical top-down approach.
The hands-on argument rests on our practical building blocks introduced in Chapter 1. They are subsumed under the headlines of the three guiding principles in respective chapters. Chapter 7, “Toward Pragmatism: Lean and Agile EA,” for example, proposes a systematic sieve that is inspired by lean methods and filters out typical waste in an architecture work stream. The gains of our building blocks are demonstrated by an EA Dashboard that indicates the key traits of success or failure.
We have repeatedly stressed that EA is about managing a complex system without explaining what the trail of complexity actually entails. Let’s leave the path of practical reasoning for a moment and consider complexity from a general perspective. We invite the impatient practitioner to lean back,

Reflections on Complexity 139

and remember a statement by the economist Friedrich August von Hayek: “To prefer something practically applicable means to relinquish the power our thinking provides us with.”1

Reflections on complexity
Beyond threshing machines
Complex systems have characteristic properties. Their structure and behavior are fundamentally dif- ferent from simple systems such as the threshing machine depicted in Figure 6-2.
The threshing machine has a firm composition consisting of a steam engine transforming heat into mechanical force and a coach that is driven by this force via a belt. The coach takes cereal ears as input to the chute on the roof, threshes the corn off the ears, and throws the straw out at the left end. Though the apparatus comprises quite a few components, it is not complex in the proper sense. There’s just one rather limited thing it is capable of doing when put into a suitable environment, and managing it simply means operating a few levers at the steam engine. Pushing the lever makes the wheels and belts run faster so that more corn and straw is produced.
Now compare this to the landscape of information systems in today’s enterprises! Envision the in- credible zoo of information systems, applications, and interfaces: Even when we limit the level of detail to the elephants in this zoo, a printout of the landscape map would easily fill a wall. The map would show a snapshot of an expanding cosmos of systems where messages are echoing in a network that gets denser all the time. And mark this: Each single information system is already a highly compound being in itself, with its numerous hardware components, drivers, and the deeply layered software stack it operates.

FIGURE 6-2

Model of a threshing machine.
German Historical Museum, used with permission

1See Hayek (1969, p. 49); translation by the authors.

140 CHAPTER 6 Foundations of Collaborative EA

The transparent causality and predictability of a threshing machine are lost in this picture. The num- ber of components and interdependencies has grown to an extent that puts the IT landscape into a dif- ferent category from the threshing machine. We are not making an ontological statement here: A Laplace demon2 with the intellectual capability of predicting the movement of all atoms in the universe would probably see no difference between the information systems compared to the threshing machine. But with our more modest intellectual powers, humans cannot understand IT in the same way as the threshing machine, and likewise we should not attempt to manage it along the same lines, which is the important point.
In Miyazaki’s Sen and Chihiro Spirited Away,3 the six-armed mechanical Kamaji is steering a giant bathhouse by swiftly operating an entanglement of wheels, levers, and triggers. It looks fascinating, but it is certainly not a role model for operating an enterprise’s IT. Managing a landscape of information systems feels a lot more like keeping a garden at equilibrium and cultivating its shape than handling a giant threshing machine with hundreds of gauges and levers.
But we haven’t reached the peak yet. Enterprise architecture management is not simply concerned with a landscape of information systems. It primarily looks at how people utilize information technol- ogy, how daily routines and work processes are or should be interwoven with information systems, how human capabilities and automated functions interoperate, and so forth. We are not looking at a tech- nical system but at a sociotechnical one. But with humans entering the scene, we beyond doubt have the first “system components” in our scope, for which the behavior and interactions must not be regarded as “simple.” This is true in particular because today’s workplaces are no longer cutting down human degrees of freedom in the way Henry Ford’s production lines did.
Hence we are better off at this point with accepting that the system EA is dealing with in fact is complex, to understand what this entails, and to decide which management approaches are appropriate and which are not.

Structure and behavior of complex phenomena
Let’s move on from an “Uh-oh, this might be complex” gut feeling to a clearer understanding of com- plex phenomena. There is no general definition of complexity; different branches of science have cre- ated their own stipulations. Some jesters mock that the definition of complexity is best exemplified by the definition of complexity. But fortunately we do not depend on a sharp delineation of complexity. What we’re after is an overall understanding of the phenomenon and a derivation of what it means for

2The French mathematician and astronomer Pierre-Simon de Laplace (1749–1827) invented the thought-experiment of an intellect with unlimited computational capability to illustrate his deterministic view of the world. He believed that if such an intellect “would know all forces that set nature in motion, and all positions of all items of which nature is composed,” it would be able to compute the whole future and past out of the laws of nature. “For such an intellect nothing would be uncertain and the future just like the past would be present before its eyes” (Laplace, 1814). Though this view of the world is somewhat outdated today, the “Laplace demon” is still a concept used in many “what if we weren’t limited by our intellectual powers?” thought experiments.
3Sen and Chihiro Spirited Away is the famous animated fantasy adventure by the Japanese artist Hayao Miyazaki. It won the Academy Award for Best Animated Feature in 2001. Kamaji is one of the fantastic creatures helping Chihiro in her struggle against the witch Yubaba. He is the chief technology mechanic of Yubaba’s bathhouse and looks like a cross between a human and a spider.

Reflections on Complexity 141

management. A first step toward understanding is the following list of structural characteristics of com- plex systems:
• A complex system is composed of many components.
• The behavior of one component influences the behavior of many other components.
• The system has an internal state, and its behavior does not depend solely on input but also on the state.
• The variety of the system, that is, the number of states it can have, and the differences these states show are high.
• The system has many channels of interaction with the environment, and the variety of inputs and outputs is high.
This list remains a fuzzy characterization. One may ask what the quantifiers many or high mean, and you could argue whether any of these bullet points can be dropped in one or the other case. But since there’s no doubt that the sociotechnical system EA is concerned with satisfies all these criteria, it would be a pointless discussion. That’s why we claimed that we don’t depend on a sharp delineation.
Next, what differences in the behavior can we expect from complex systems in comparison to sim- ple ones?
Table 6-1 gives some hints at the differences.

Table 6-1 Simple and Complex Behavior
Simple System Complex System
Shows the same behavior under the same Behaves differently even if the external conditions are external conditions (strong causality). exactly the same. Has its “own will.”
Shows similar behavior under similar external Small deviations in the external conditions can conditions (weak causality). sometimes lead to a completely different behavior.
Variations in the external conditions lead to The relationship between conditions and behavior is proportional variations in behavior (linearity). nonlinear, sometimes discontinuous.
Works as anticipated along its preconstructed Shows emergent structures and generates spontaneous lines. Shows no life of its own. order and regularities.
Is accurately fitting into a certain environment but Is sufficiently well integrated into the current environment likely to become unstable in case of changes. but capable of absorbing vast changes (viability).
Is essentially bound to a particular environment Is able to learn and adapt to new environments and must be reconstructed if this ceases to exist. (adaptability).
Has a delimited interface with the environment Has permeable boundaries and tends to alter the behavior and can be isolated from observers. when being observed (second-order cybernetics).

Principles of managing complexity
Now, what have we learned from this brief comparison with regard to mastering the complex socio- technical system EA is concerned with? From general considerations like this we certainly cannot deduce hands-on recipes that will make enterprise architects roll up their sleeves to give them a try tomorrow morning. It is more a set of principles guiding our expectations and attitudes with respect to the EA game that comes out of our reflections.
Principle 1. Stay at coarse granularity in analyzing complex systems (such as enterprise architectures) and predicting their future.

142 CHAPTER 6 Foundations of Collaborative EA

The large number of influential factors and the inevitable discontinuities in the behavior of complex systems limit our ability to explain and predict things to coarse grains. Detailed descriptions and ex- planations of their causality as well as single-point predictions are exceptional cases. In the words of an IT saying, coarseness of grains is not a bug but rather a feature of managing complexity. Finer gran- ularities cause an exponentially increasing effort and are likely to end up in a quixotic project. What is achievable in the domain of complex phenomena is4:
• A general understanding of how things work in principle
• A more or less fuzzy prediction of a range of future states
Furthermore, our scientific education elicits a certain bias for positive propositions. But in complex areas, negative explanations of what has not happened or predictions of what will definitely not be the case can sometimes be as useful as positive ones.
As an enterprise architect, your CIO might ask you what the target application landscape will be three years from now. The most appropriate response might be: “No clue, but I can list fifteen percent of the current applications that will definitely be discontinued in this timeframe.” It might not be recommended to answer this way, though, because you immediately have a nasty political problem on your plate. But it is important to treat this as a political problem and not a problem of how you do application landscaping.
Principle 2. Strengthening a complex system’s ability to change wins over optimizing the status quo.
A complex system shows a certain “life of its own” or has its “own will.” It shows a notorious re- calcitrance to stay close to the drawn lines and generates emergent, not anticipated, structures. Con- structing such a system ex ante is, therefore, a thankless if not useless task.
Architects who have led large software development programs know how it feels when “the best blueprint and architecture principles ever” very soon are subject to “creative interpretation” by the im- plementers on the ground. The initial architecture is transformed (the planner might say distorted) by the thought process of the hundred brains involved in the program. Architects sometimes feel like pull- ing their hair out and shouting for stronger governance.
But it is more insightful to acknowledge that such change is not only inevitable but is in fact a nec- essary ingredient of success. The constructed information system is interwoven with the people devel- oping, maintaining, and using it, and this conglomerate certainly is a complex system in the sense characterized previously. Such a system is viable to the extent that it can preserve its identity under chang- ing conditions; a necessary prerequisite of this ability is to form variations responding to changes or late insights. A rigid corset, on the contrary, is likely to break down on the bumpy road of a two-year program.
What holds true for the initial state of a system also applies to later phases of the life cycle. We have a certain tendency to dream of optimal states. Take, for example, a sermon like this:
When we have eventually transformed our legacy applications into a SOA landscape, our IT problems will be easy to solve. We shall eventually be able to adapt our IT landscape to the pressing business demands in an agile fashion.
Most professionals would hesitate to explicitly subscribe to such a na¨ıve vision, but it is embar- rassing how such “happy salvation” beliefs hover as tacit assumptions over so many endeavors.

4Cf. Malik (2008, p. 182 ff).

Reflections on Complexity 143

With regard to the nature of complex systems, it is not advisable to search for the “optimal enter- prise architecture.” Optimality is always relative to an environment, and the frame conditions are likely to change. Hence it is better to accept certain ugly spots of the IT landscape and strengthen the viability of the whole, which means developing and using capabilities such as being able to learn, solve prob- lems, adapt, or absorb disruptions.
There is also another argument in favor of this principle. An enterprise resembles a self-transforming, adaptive organism rather than an extremely complex (but ultimately static) machine. It is a sociotechnical system captured in permanent evolution. More precisely, it coevolves alongside the other enterprises in its market segment in an everlasting struggle against displacement and extinction.
“Now here, you see, it takes all the running you can do, to keep in the same place,” says the Red Queen in Lewis Carroll’s Through the Looking Glass.5 In a competitive world, some constant amount of improvement is required to merely defend your own position. All players in the market operate in a “fitness landscape,” where enterprises (similar to biological species) constantly perform minor or major transformations in the pursuit of fitness peaks in that landscape.
The principle behind this phenomenon is called coevolution; by transforming itself and its business processes, the enterprise changes the survival game not only for itself but also, through changing the market conditions, for its competitors. This makes the fitness landscape dynamic, and the peaks of optimal fitness are changing at each evolution step. An EA that tries global optimization for an enterprise is, in terms of the coevolution process, searching for the global fitness peak (or peaks)— for the global optimum in competitiveness. By a relatively simple thought experiment6 it can be shown that the complexity of this search is in exponential order of the enterprise’s complexity itself. “Therefore, finding the global peak or one of a few excellent peaks is a completely intractable problem,” writes Kauffman (1995). Such a search is an NP hard task, which basically means that there is no simpler search algorithm than to test all combinations—resulting in a combinatorial explosion of the number of options.
Opposed to the futile search for global optima, strengthening the ability to change enables the enter- prise to flexibly adapt itself to an ever-changing fitness landscape. Principle 2, “Strengthening a complex system’s ability to change wins over optimizing the status quo,” expresses that this strategy yields better results—eventually. We will see a bit later, in the section “Benchmarking Hierarchies versus Networks of Managers,” what a suitable organizational structure valuing this principle can look like.
Principles 1 and 2 can be directly derived from Table 6-1. To get further advice, we now turn to a theorem about systems in general, not only complex ones. It is the Law of Requisite

5The biologist Leigh van Valen used this quote in postulating his Red Queen Hypothesis (van Valen, 1973), which sees the evolutionary ecosystem as a kind of zero-sum game whereby the gain of one species equals the loss (or even extinction) of another one. This leads to an “arms race” in which each species must constantly improve its fitness to keep pace with the newly developed competitive advantages of others.
6Kauffman (1995) chose a very simplified model for a participant in a coevolution process. Kauffman’s reasoning deals with biological species, but his argumentation applies to enterprises as well. An abstracted genome defining system traits is mod- eled as a set of N Boolean variables, each representing a gene. For every gene, a fitness function is defined that delivers a value between 0 and 1. The function value depends on the state of that gene plus contributing inputs from K other genes (with
K < N). The overall fitness is then defined as the mean value of all fitness functions. Even with such a very simple model,
the number of local optima in a fitness landscape explodes when K approaches N. The search for a global optimum is an NP hard problem under these conditions. (NP hard is a term from computation complexity theory.)

144 CHAPTER 6 Foundations of Collaborative EA

Variety, formulated and proven by William Ross Ashby, one of the fathers of cybernetics in (Ashby, 1958)7:
The larger the variety of actions available to a control system, the larger the variety of perturbations it is able to compensate.
Ashby himself shortened this statement to a more memorable slogan:
Only variety can absorb variety.
The general scenery of this law is sketched in Figure 6-3: A controller is managing a system by acting on some control variables based on observations taken from the system. These adjustments are neces- sary to cope with perturbations stemming from somewhere inside or outside the system and to keep the system in line with the goals that are also pre-given from somewhere. One usually has gauge values such as temperature and thermostat position in mind when thinking about observed and control variables, respectively.
With regard to sociotechnical systems, however, there are more options. Take, for example, the traffic system of a country. Some ordinary gauge values are available, such as the number of traffic injuries per year. But control variables, for instance, include rules the drivers and pedestrians must obey. It is important for the applicability of the model that the actions of the controller are not confined to such simple things as turning a thermostat, because enterprise architects typically exercise their con- trol more by designing blueprints or setting rules than by turning some gauge value up or down.
One further term needs explanation before we can draw some conclusions—namely, the term variety. The variety of a system denotes the number and variance of states it can attain and is a measure of the system’s complexity. In short we could say that the essence of control or management is reducing the variety of the system and stabilizing it in the boundaries of the states that are admissible according to the goals.
FIGURE 6-3

A cybernetic control system.

7We recommend Heylighen (2001) as a more readable source of information.

Reflections on Complexity 145

Now, what does Ashby’s law tell us? It claims that there is a certain relationship among the fol- lowing factors:
• The complexity of the environment (measured by the variety of the perturbations or influential factors)
• The complexity of the controller (measured by the variety of the repertoire of actions available to the controller)
• Finally, the variety of the system itself
If we denote the varieties in this order by V(E), V(C), and V(S), respectively, the relationship can be expressed by a proportional equation as follows:
VðSÞ= VðEÞ— VðCÞ
Assume that it is our task to design the system’s control C. What are the variables in this equation that can be influenced? V(E) must be regarded as fixed; the environment won’t do us a favor and morph into something simpler. V(S), on the contrary, is not altogether pre-given. It has two ingredients: On one hand, it is an inherent construction property of the system, something we cannot alter right away. But on the other hand it reflects the accuracy of control required by our management ambitions, and depends on how rigidly and precisely we want to determine what is going on.
Therefore, how simple the management of a system can be depends on:
• How complex the system is in itself
• How widespread the environmental factors are
• How fine-grained the goal settings are
These background considerations pave the way for further guiding principles.
Principle 3. Management, in particular EA, has about the same complexity as the system it manages.
This principle should be regarded as a caveat against “EA made easy as a pie” promises. New- comers to EA frameworks such as TOGAF are sometimes daunted by the complexity of such frame- works and start grumbling about the difficulties in grasping it. But with an eye on the complexity of the system EA is managing, our principle sets the expectations right. Neither the newcomer nor stakeholders like the CIO should expect that EA is simple as 1-2-3.
Principle 4. Complex systems (such as enterprises) cannot be managed at an object level but only at a meta level (management by rules).
Managing a system at the object level means intentionally positioning its elements, planning and constraining their interactions, and giving detailed instructions as to how they must behave and interact in each instance. We learned already from Table 6-1 that complex systems show a certain reluctance to stay faithful to their blueprints and generate emergent structures. Influence at the object level therefore tends to be similar to herding cats.
Ashby’s law also gives us a hint as to how complex such a managing instance would have to be. An EA office acting as a direct controller to the IT and enterprise architecture of an organization is far beyond these capabilities, even if it employed the most brilliant brains that money can buy.
Certain levels of indirection and abstraction are inevitable. A viable EA therefore confines itself to cultivating emergent structures by means of abstract rules and relies on the local knowledge of

146 CHAPTER 6 Foundations of Collaborative EA

autonomous subunits. Concretizing the rules down to code, server configurations, and other interven- tions at the object level is left to subunits. The use of abstract rules does not render simpler the overall management of a system—Principle 4 still applies—but it is the only reasonable approach from the top down. In a sense, it leverages the self-management capabilities of subordinated levels to do the job.
Following P. Weill and J. W. Ross (2004), who promote a similar management approach, we call Principle 4 the Management by Rules principle.
The final principle we want to set out is based on research by Kauffmann (1993, 1995):
Principle 5. Complex systems (such as enterprises) are best managed at the edge of chaos.
The term edge of chaos was coined by Kauffmann and is, simply speaking, the fine line between too little and too much control over a system. Kauffman’s work indicates that systems show a sudden tran- sition from a rigid, “frozen” state to chaotic behavior when certain control parameters are increased. Only in between, along the edge-of-chaos line, the systems show an efficient, structured behavior.
Brown and Eisenhardt (1998) have conducted a couple of empirical studies, which indicate that the same principle applies to the way enterprises structure their internal processes. The most efficient EA, it seems, exercises just so much control that the organization operates at the edge of chaos— structured enough not to let the IT slip into anarchy but not so rigid that it is locked into bureaucratic permafrost.

Management capabilities of hierarchies and networks
Ashby’s Law of Requisite Variety states that “only variety can absorb variety” and thus gives us a lower bound for the complexity of a management organization in relation to the complexity of the managed system and the required accuracy of control. But this management organization can of course be much more complex than needed; it can be an exuberant entanglement of roles, communication channels, and so forth on top of an innocent tiny system.
Furthermore, Ashby’s law gives no indication which organizational form of the management ap- paratus deals most economically with complexity. To fill this gap, we end our reflections on complexity with a comparison of the two most prevalent organizational forms: hierarchical organizations that rely on a top-down information flow, and network organizations that exchange information in a network. The latter structure is based on an exchange of information between peers on equal terms.
How can we estimate their respective capabilities, their parameters of influence, and how they com- pare? These considerations give a hint of how to position and organize an EA office and are the foun- dation for our building blocks described in Chapters 7 and 8.
Since we are asking about the capability to manage complexity, we will benchmark our organiza- tional forms against a system with unlimited complexity: Only then can we examine how far they can go. In a system with limited complexity, the capability of one manager to shape and control things sometimes appears as a threat to other managers, since it diminishes their piece of the cake, their de- grees of freedom to shape and control things.
Let’s put this managerial jealousy aside by envisioning a system with an unlimited need for man- agement, where the capability of one manager is welcomed by other managers as an opportunity to even shape and control more things. Enterprise architects with a notoriously understaffed team will not find it difficult to envision such a system. In this happy situation, the IT landscape anyway seems

Reflections on Complexity 147

FIGURE 6-4

Illustrations of the concepts manager and management capability.

like a Hydra8: If you get one part nicely in shape and under control, immediately two other parts pop up that need taming. There’s no end to the management capability you need.
What we mean by manager here is explained in Figure 6-4. It is an entity that gives design and control directions while taking into account observations from the managed system, local or global knowledge about the environment, and directions from other managers. In this general sense, an en- terprise architect must certainly be considered a manager.
The capability of a manager is her ability to constrain the current and future state of affairs. It is her power to shape the system and keep it in certain bounds, as depicted in drawing (b) of Figure 6-4. A manager’s capability is proportional to the variety of her design and control activities—namely their breadth and effectiveness.
With these definitions in mind, we now take a look at the first prevalent organizational form, the hierarchy. Figure 6-5 shows a simple balanced hierarchy with its major construction parameters, the height and branching.
A characteristic feature of a hierarchical management organization is that the management capa- bility depends on the level in the hierarchy—it is a function C(l) of the level. For simplicity let’s assume that the capability drops proportionally from one level to the next. In formulas, this can be expressed as:
Cðl þ 1Þ ¼ a • CðlÞ; where 0 ≤ a ≤ 1
This implies that C(l) a C , where C denotes the capability of the top manager. We interpret the parameter a as the degree of autonomy in the hierarchy. In autocratic hierarchies, this parameter will be close to zero. For example, you might think of an old-fashioned, patriarchic handicraft business

8Hydra is a monster from Greek mythology. It possesses many heads; for each head that is cut off, the beast is able to grow two new ones.

148 CHAPTER 6 Foundations of Collaborative EA

FIGURE 6-5

A simple balanced hierarchy.

where all decisions must eventually be made by the grand old craftsman and owner of the establish- ment. If you address someone at the journeyman (apprentice) level with a question, he shrugs his shoul- ders and replies that he first has to ask the master. The subordinates might add a pinch of local knowledge and interpretation to the master’s commands, but that’s about all of their autonomy.
In modern organizations and more complex enterprises, however, the parameter a will be consid- erably higher than zero. An abstract reason for giving more autonomy to the subordinated levels is hidden in the formula that adds up the overall management capability of the whole organization:
2 2 h h h l l

CðHÞ ¼ C þ b • a • C þ b

• a • C þ .. . þ b • a • C ¼ C • Sl¼0 b a

This sum is the best that can be expected from a hierarchy. It makes certain idealistic assumptions, namely that the capability of level l is not diminished by the lower-level l 1 and that the different branches of the hierarchy do not issue contradictory directions.
But we learn from this formula that autonomy pays off; moreover, it pays off exponentially.
Figure 6-6 illustrates this concept nicely for seven hierarchy levels and a top capability C 1.
As we can see, the capability of an autocratic organization to manage complexity is almost reduced to the mastermind’s capability to do so. More local autonomy, on the other hand, soon greatly increases this capability.
But what happens if we make a shift in paradigm from hierarchies to networks of managers? It isa popular bias that networks somehow excel at hierarchies. Professionals are, for instance, stunned by the success of loosely organized open-source projects in managing and shaping highly complex software architectures. In these projects—so the argument continues—the design and control are in the hands of the crowd, the network of contributors. Still, they manage a complexity that easily overcharges many rigidly organized companies. Isn’t this the magic of the mysterious network effect?
Even if we concede that open-source projects are not as loosely organized as people tend to believe, this is a question worth looking at. But an appeal to gut feelings about the omnipotence of social net- works, which often is the basis of such discussions, certainly is not sufficient for our purpose. In what

Reflections on Complexity 149

FIGURE 6-6

Exponential growth of the management capability.

follows, we strive for an abstract model of the network effect with regard to management capabilities and identify the parameters it depends on.
What is the management capability of, for instance, a simple network of managers like the one shown in Figure 6-7?
We can only give an abstract answer to this question if we make idealistic assumptions as we did in the case of hierarchies. We assume a well-designed, working network, where none of the

FIGURE 6-7

A simple network.

150 CHAPTER 6 Foundations of Collaborative EA

participating managers is messing up the other manager’s business, for example, by issuing contra- dictory directions.
If there is no network at all and the managers work independently without overlapping responsibilities and without insight into what others are doing, the overall capability simply adds up to C(N) n C. But on the basis of well-functioning communication channels, managers can synchronize their own directions with what they know about the other’s intentions.
On this ground, it is conclusive that a manager’s capability to design and control a system is in- creased by the awareness of what the other managers aim at and plan. Furthermore, it is not altogether unrealistic to assume that this increase is proportional to the number of peers a manager is able to synch up with. There might be an upper limit for how much information about others’ intentions any one manager can absorb, but let’s set this aside for a moment.
The capability of a single manager can therefore plausibly be quantified by a term C (n 1) a C, where the parameter a this time does not reflect the local autonomy, as in the case of hierarchies, but how well aligned are the different nodes of the network. The total capability of the network would then add up to:
CðNÞ ¼ n • ðC þ ðn — 1Þ • a • CÞ ¼ n • C þ n • ðn — 1Þ • C • a
The second term of the formula, n (n 1) C a, quantifies the network effect and is just a variant of Metcalfe’s law. This law, named after the inventor of the Ethernet protocol, Robert Metcalfe, states that the value of a network increases by n2 n (n 1) with the number n of members. With these two formulas at hand, we can conclude:
The capability of a hierarchy of managers to cope with complexity is significantly lower than that of a corresponding network.

The argument can be given in a single line, keeping in mind that h
l¼0

bl:

CðHÞ ¼ C • Sh blal < C • n < n • C þ n • ðn — 1Þ • C • a ¼ CðNÞ
Metcalfe’s law was a hot topic during the Internet bubble because investment gamblers misinter- preted it as a “natural law” predicting quadratic revenue growth from linearly increasing investments. Recent criticism points out that the value of a network grows more modestly by n log (n) (Briscoe, Odlyzko, and Tilly, 2006).
The argument is based on yet another law, Zipf’s law. The linguist George Kingsley Zipf sorted English language words by frequency of occurrence and discovered that frequency approximately de- creases in a harmonic sequence 1, 1/2, 1/3, 1/4, 1/5, and so forth. This observation turned out to be transferrable to many other statistical phenomena: Whenever we sort a collection by size or value, the distribution is stunningly close to some variant of the harmonic sequence.
This transfer might work for the contact list of a networker, too: It is indeed plausible that not all partners on this list provide equal value to the networker, and Briscoe and his coauthors claim that sorting the list from very important persons (VIPs) to marginal random contacts resembles the har- monic distribution. Hence, if we accept this hypothesis, the network effect is no longer measured by the quadratic term n (n 1) C a but by a more moderately growing, logarithmic formula, namely:9

9This is because the harmonic series S1

1 converges to log(n) þ E, where E ¼ 0.5772.. . is the Euler constant.

Reflections on Complexity 151

n • Sn—1 c • a = n • ðlogðnÞþ 0:5Þ • C • a
But even this damped growth outperforms the hierarchical capability C(H), and the same argument works for all kinds of positive network effects. The comparison theorem therefore holds true unless we assume that networking diminishes a manager’s capability C to influence the state of affairs or that a hierarchy in fact increases this capability.
There are other indications still that networks might be more efficient than hierarchies. Let’s go back to the notion of an enterprise as a participant in a coevolution process, competing for market share (just as species compete for habitats to ensure their survival).
The long-term survival of an enterprise is determined by its ability to continuously search for peaks in the volatile fitness landscape formed by itself and its competitors. This search means nothing else but a constant adaptation of business portfolio, organization structure, and strategy in the coevolution pro- cess. The enterprise’s survival in the market follows the same basic laws that govern the survival or extinction of a species.
One of the key factors determining a system’s adaptability seems to be the “connectivity” within the system’s internal structure (Kauffman, 1995; Bak, 1996; Lucas, 2005). If the agents within the system are insufficiently connected, the system is too cool and static. At the other end of the spectrum, with too much connectivity the system overheats and slips into chaos.
The agents within an enterprise are essentially its employees. A network structure between them allows for a flexible tweaking of the connectivity, depending on the problem at hand. In comparison, in a strictly hierarchical organizational structure the connectivity cannot be varied and adapted so easily.
There is yet another, although related, angle to this idea. Hierarchical organizations are efficient at implementing a centrally devised strategy. In evolutionary terms, where adaptation by self- transformation is paramount, this is often only the second best option. Kauffman (1995) has shown that an internal organizational structure in “patches,” each independently optimizing its fitness, usually works better for the whole system when the fitness landscape is rugged and volatile. A network struc- ture is a more natural implementation of such patches than a strict hierarchy. We will deal more closely with concepts for such “local independence” in Chapter 7, “Toward Pragmatism: Lean and Agile EA.” It is time to draw a bottom line to this notable dose of theory and formula work. Did we say that an organization of architects should have a network structure rather than a hierarchy? Not quite, since a network has considerable downsides in comparison to a hierarchy. A hierarchy, for instance, has the advantage that management directions reliably reach the ground after passing all hierarchical levels. In a network, directions can oscillate between managers for an undetermined time. That’s the reason that
systems in which reaction times are paramount are better off with hierarchies.
Furthermore, working networks are apparently more difficult to build. With a hierarchy, it seems to be a lot easier to reach a noncontradictory, nonoverlapping set of roles and responsibilities; at least the track record of organizing firms hierarchical is longer and has more proven practices at hand.
This list of downsides can probably be continued, and you may find good reasons in this list to decide for a hierarchical EA office. But eventually we have to concede that there indisputably remains one important advantage of a network: It is more capable of managing complexity. Lots of people share a gut feeling that if it gets really complex, we have to employ networks to do the magic. This is more than a gut feeling, as we have seen in the above mathematical deduction.

152 CHAPTER 6 Foundations of Collaborative EA

The EA dashboard as a yardstick for EA effectiveness
“Everything exists”: That is one extreme. “Everything doesn’t exist”: That is a second extreme. Avoiding these two extremes, the Buddha teaches the Dhamma via the middle path.
—Buddha
In the following two chapters, we will lay out our set of building blocks to strengthen collaborative elements in EA. In preparation, let’s consider how to gauge the applicability of our measures. We have no case studies available to prove their effectiveness, but we can offer a simple yardstick against which each proposal can be valued.
Reducing a long list of potential EA problems to merely a handful of criteria will make it easier to judge if and how a specific measure might have an effect. For this purpose, we go back to the four dimensions of EA complexity introduced in Chapter 1. To recap, the four dimensions are Perspective, Governance, Strategy, and Transformation.
In each of these dimensions, there is an interplay between chaos and order. The most effective mode of operation is to strike a balance between the two extremes and to move along the edge of chaos (see Figure 6-8) that separates them. The underlying assumption is that EA will work best if the enterprise architects find the optimal middle ground. This is the best compromise between a top-down, long-term-oriented EA style and a collaborative bottom-up, evolutionary mode.
This assumption is backed up by research results from management theory. The advent of the first Internet boom, around the millennium, has triggered a lot of research in this area. Markets and competition in the IT area changed more quickly than ever before. Companies rose from startups to multibillion-dollar operations within a decade, or virtually disappeared in the same period, or under- went dramatic transitions to survive. Strategy had evolved into a deciding asset.
Our concept is primarily based on the ideas of Brown and Eisenhardt (1998).10 They took up the edge of chaos notion introduced by the evolutionary biologist Kauffman (1995) and applied it

FIGURE 6-8

Maneuvering at the edge of chaos.

10Many authors express similar concepts—for instance, Schwartz (1996) or Freedman (2000). The guiding theme of these books is that successful strategies in a complex, volatile, and unforeseeable environment should allow for a certain amount of chaos in favor of increased flexibility.

The EA Dashboard as a Yardstick for EA Effectiveness 153

to the business world. Brown and Eisenhardt’s basic message, derived from their research on selected industry case studies, is (somewhat pointedly) that an organization fares best when it ac- cepts a certain amount of—but not too much—disorder in various dimensions of business activities.
The EA Dashboard as depicted in Figure 6-9 has been inspired by their taxonomy, with the original categories adapted to the more specialized terms of EA. The structural dimensions Perspective and Governance are summed up under Edge of Chaos, whereas the more time-oriented dimensions Strategy and Transformation are subsumed under Edge of Time. This is a related term introduced by Brown and Eisenhardt to describe an organization’s take on change processes such as modernization and future visions. Otherwise, there is no semantic difference between edge of chaos and edge of time. In both cases, the extreme positions in the Dashboard’s gauges are exactly the EA caricatures introduced in Chapter 1.
Being at the edge of chaos involves staying clear of the extremes. Structure and processes are strong enough to provide guidance, whereas on the other hand they are still slight enough to have flexibility and not to consume too much management attention. In a similar fashion, navigating at the edge of time involves having the right pace of change for transformations and the right amount of foresight. EA reaches a state of optimal efficiency when it operates in the unstable equilibrium of the “edge of X” middle-ground position.
In the subsequent Tables 6.2 through 6.5, the criteria for judging an EA organization in each dimension are listed in detail. Taking these criteria, the individual position of the gauge hand can be determined.

FIGURE 6-9

The EA Dashboard.

154 CHAPTER 6 Foundations of Collaborative EA

Table 6-2 EA Dashboard: Criteria in the Perspective Dimension
Perspective
Too Low About Right Too High
In the Chief Mechanic’s Workshop Broad Yet Detailed View Living in Cloud Cuckoo Land

Focus Technology focused, no broad vision Right balance between big picture and technical reality on
the ground Ivory tower of concepts and strategy
Organizational setup Directly part of the IT organization, or at least strongly connected with it Independent organizational unit, short reporting line to CIO Anywhere
Network Primarily among IT crowd, few network ties with business Equally well connected with the business, IT, and higher management Main connections to higher- level management
Team composition Technical experts Mix of experienced business experts,
IT architects, and
go-betweens at home in both worlds Management consultants

The EA Dashboard as a Yardstick for EA Effectiveness 155

Table 6-3 EA Dashboard: Criteria in the Governance Dimension
Governance
Too Weak About Right Too Rigid
The Overstrained Technical Advisors Control Where Needed The Guardians of Wisdom

Rules “Break rules” culture in architecture, standards are ignored Concentration on a few guiding architecture principles that are kept at all times Many standards, blindly followed
Processes Very few or no EA processes in place A few core EA processes that are universally accepted and continuously revised and updated Comprehensive and rigid EA process framework
Communication Random communication— everybody talking to everybody Continuous commu- nication, formal and informal, between stakeholders (business to EA, EA to projects, etc.) Communication only along narrow and strictly defined channels
Role of the enterprise architect Mere advisor, only formal but little de-facto authority, ownership, or accountability Guide, mentor, auditor, with healthy amount of formal authority Intimidating enforcer of rules

156 CHAPTER 6 Foundations of Collaborative EA

Table 6-4 EA Dashboard: Criteria in the Strategy Dimension
Strategy
Too Myopic About Right Too Far-Reaching
Sweeping Up the Change Requests Balanced and Flexible A Deep Look into the Crystal Ball

Vision No vision at all Multiple parallel scenarios to remain flexible, with many inexpensive pilots as part of the strategy One single and firm vision for the future, based on belief and speculation rather than research
Planning No planning for application portfolio and strategic initiatives at all Planning for foreseeable time horizon only, continuously revised Meticulous planning over a long time horizon, with little deviation from plan
Focus Reactive mode, focus entirely on today’s problems Balancing attention between present and future Proactive mode, intense focus on future, ignoring the present

The EA Dashboard as a Yardstick for EA Effectiveness 157

Table 6-5 EA Dashboard: Criteria in the Transformation Dimension
Transformation
Too Slow About Right Too Fast
The Ever-Growing Backlog Steady Evolution The Permanent Construction Site

Renewal Mainly maintenance of existing systems Applications are replaced according to consistent criteria, but without haste Applications thrown out in rapid succession
Enhancements Functional enhancements mainly by layering existing applications Design of new capabilities decided case by case, either by new applications or by enhancement of existing ones Functional enhancements mainly by creating new systems
Outsourcing New systems are developed and run externally, own IT staff concentrates on existing systems Outsourcing is spread across old and new applications Own IT staff does only “fancy” new development, existing systems are outsourced
Readiness to take risks Conservative, focus on ensuring operability Moderate, cautiously taking risks when needed High, aggressively taking risks

The EA Dashboard uses findings from management theory to provide a measuring instrument for EA groups and the building blocks of collaborative EA. In subsequent chapters we will indicate by the Dashboard in which dimension of EA complexity the respective proposal can have an effect. This way we will provide an easy-to-use application guide for our building blocks.