Integrating Layer
May 2004 Summary: This chapter describes the different strategies for designing an integration layer and the tradeoffs involved in choosing an alternative. An integration layer can automate complex business processes or provide unified access to information that is scattered across many systems. ContentsChoosing an Integration Layer Type "Any problem in computer science can be solved with another layer of indirection."—David Wheeler (chief programmer for the EDSAC project in the early 1950s) As described in Chapter 1, connecting multiple systems requires an infrastructure that moves data between the systems. However, you often want the solution to do more than just pass data around. You want to add an additional layer of functionality on top of the functional assets that reside inside the existing applications. This layer can automate complex business processes or provide unified access to information that is currently scattered across many systems. How should such an integrating layer be designed, and what choices do you have? Unfortunately, there is no single right answer for all enterprise architectures. This chapter discusses some of the key considerations to help you to understand the tradeoffs associated with the various alternatives and to find the most suitable approach. Level of AutomationA fully integrated enterprise seems to be any CIO's idea of perfection. Complex interactions between systems are orchestrated through precisely modeled business process definitions. Any data format inconsistencies are resolved through the integration layer. Relevant summary data is presented to executive dashboards with up-to-the-minute accuracy. Such visions are surely enticing, but should every enterprise set out to build such a comprehensive and inherently complicated solution? It is not likely that anyone would build such a solution because of the time and money it takes to build a complex enterprise integration solution. Therefore, deciding how far to go is an important step when planning an integration solution. Automation brings efficiency and can help eliminate inconsistencies in current business practices. However, complete automation can require an enormous upfront effort that delays the tangible benefits to the business. A simpler solution might only achieve a portion of the business benefit, but it achieves that benefit much sooner and with much less risk. For example, displaying information from multiple systems in a single screen that is divided into individual panels can increase productivity, and it does not require the system to resolve all the differences between existing systems. Level of AbstractionObject-oriented design principles teach the benefits of abstraction. Abstracting one system's internals from other systems allows you to change one system without affecting the other systems. This ability to limit the propagation of changes is a key consideration for integration solutions where connections can be plentiful and making changes to applications can be very difficult. You achieve abstraction in a number of ways. For example, you can achieve abstraction by passing self-contained documents between systems. Passing a document does not instruct another system to perform a specific function, but it leaves that choice to the receiving system. Thus, if the receiving application processes the document in a different way, the originating system is not affected. Maintaining StateMost integration solutions aim to streamline business processes such as placing an order or making a payment. These processes rely on the coordinated completion of a series of steps. When an order arrives, the order has to be validated, the inventory has to be checked, the sales tax has to be computed, and an invoice has to be generated. This type of integration requires some mechanism to track the current state of an order. For example, the mechanism might indicate whether the order has been validated yet and which step should be completed next. This state mechanism can be implemented in three ways:
CouplingLoose coupling has become the standard of distributed applications. Loose coupling makes integrated systems less dependent on each other and allows them to evolve independently. However, loose coupling is not a panacea. Loosely coupled systems can often be difficult to understand and debug. Semantic DissonanceOne of the key difficulties in building integration solutions is the notion of semantic dissonance. This term describes a situation where data that appears to be the same may not necessarily mean the same thing. For example, one system might treat a monetary figure for annual revenues as if it includes sales tax, while another system treats the monetary figure as if it does not include any taxes. Likewise, one system might define the Western Region, in the context of revenue breakdown, to include the state of Colorado. Meanwhile, another system may define Colorado as part of the Central Region. These types of semantic dissonance can be very difficult to detect and reconcile. Some rather elegant types of integration rely on the complete resolution of semantic dissonance, which might not be feasible in real-life situations. Other simpler forms of integration, such as Portal Integration, can accommodate ambiguity, but they do it at the expense of precision. For an in-depth discussion of data modeling and the perils of semantic dissonance, see Data and Reality, by William Kent [Kent00]. Choosing an Integration Layer TypeBased on these considerations, this guide identifies three approaches towards integrating layers. Each approach is presented as a pattern (see Figure 1):
Figure 1. Three integration approaches The following paragraphs briefly describe each approach to integration, in order from least to most complex. Portal IntegrationPortal Integration connects systems to provide a unified view to a user. Portal Integration can dramatically increase productivity because users no longer have to access multiple applications independently. Instead, they receive a comprehensive view across all systems in a visually consistent format that is illustrated in Figure 2. More sophisticated versions of Portal Integration also allow the user to make updates to individual systems or even across multiple systems.
Figure 2. Portal Integration provides a unified view across systems Portal Integration is often easier to implement than other, more automated alternatives. The disadvantage of the simplicity is that much of the business process and many of the rules still reside in a user's head as opposed to in a system. Nevertheless, Portal Integration is often a good first step towards integration. Entity AggregationThe main limitation of Portal Integration is that it aggregates information only for end users but not for applications. Entity Aggregation addresses this shortcoming by providing a unified data view to applications, as illustrated in Figure 3.
Figure 3. Entity Aggregation provides unified data access across systems Entity Aggregation provides a logical representation of unified data entities across multiple data stores. Applications can interact with this representation as if it were a single data source. Thus, Entity Aggregation greatly simplifies the development of applications that need to access data across multiple data stores. The tradeoff for this increased integration is that the integration layer must now resolve any instances of semantic dissonance between the individual systems. Resolving such semantic dissonance and creating a unified data layer can be a major undertaking. Process IntegrationProcess Integration focuses on the orchestration of interactions between multiple systems, as illustrated in Figure 4. As mentioned earlier, automated business processes such as straight-through processing often drive integration initiatives. It is often advisable to model these processes outside of the applications to avoid coupling the applications to each other. You can achieve this modeling by adding a Process Integration layer that manages the interaction across applications. This layer tracks the state of each business process instance and also enables centralized reporting. For example, this layer enables the Business Activity Monitoring (BAM) feature of Microsoft BizTalk® Server.
Figure 4. Process Integration orchestrates activities across systems The three integration layer types are not mutually exclusive. An integration solution might use Portal Integration for some portions of the solution and Process Integration for others. Management reporting functions may be accomplished by using Entity Aggregation. All the approaches may be used in parallel. Integration Layers PatternsThe following table summarizes the three patterns just discussed and shows the corresponding implementation patterns. Table 1: Integration Layers Patterns
|
