Process Integration

 

Integration Patterns

Start | Previous | Next

Contents

Context

Problem

Forces

Solution

Example

Resulting Context

Testing Considerations

Related Patterns

Acknowledgments

Context

You have multiple disparate systems, and each of those systems is part of an overall business function. For example, a business function such as processing an incoming order may require the participation of the customer management system, the inventory system, the shipping system, and one or more financial systems. The business could operate more efficiently if the systems could be integrated so that the business function could be completed automatically.

Problem

How do you coordinate the execution of a long-running business function that spans multiple disparate applications?

Forces

To solve this problem, you have to balance the following considerations and forces:

• To implement the overall business function, you could have one system directly call the system that performs the next step, as defined in the business function. However, this approach encodes the sequence of interactions into the individual systems. Creating this dependency in each system makes changing the sequence more error-prone and also limits your ability to reuse systems in multiple contexts.
• The change and maintenance cycle of a complex business function is likely to be different from the change cycle of the individual business functions that reside inside the applications. For example, financial functions such as computing sales tax or posting revenues are typically subject to infrequent, but mandatory, legal or regulatory changes. In contrast, the overall execution of a business function might be changed much more frequently based on business and marketing strategies.
• Complex business functions can often take days or weeks to complete. However, most functions that are available in existing applications are synchronous; that is, the caller has to wait while the application performs the requested function. This type of synchronous interaction is not well-suited to long-running business functions because one application could spend a significant amount of time waiting for another application to complete the requested function.
• A longer time span of execution increases the likelihood that an application might fail during the execution of the business function. If one application fails, portions of the overall function may have to be reverted so that all systems can be returned to a consistent state.
• Because a complex business function can take a long time to execute, it is likely that a new request will arrive while the previous one is still being serviced. To improve response times, the solution should be able to handle multiple concurrent executions of the function. Many applications are not designed for this type of concurrent execution.
• Modeling business processes inside a software component can be difficult if the processes are not well understood or documented. In many businesses, users make decisions based on experience rather than documented business rules.

Solution

Define a business process model that describes the individual steps that make up the complex business function. Create a separate process manager component that can interpret multiple concurrent instances of this model and that can interact with the existing applications to perform the individual steps of the process.

Figure 1. Process Integration with a process manager component directing applications according to a process model

For each incoming request for the business function, the process manager creates a new process instance based on the process model. Each instance maintains the current state of the process plus any additional information that is needed for the business process to continue. Creating individual process instances allows the process manager to execute many business functions in parallel.

After one application completes its business function, the process manager determines which function to execute next based on the state of the process instance. Therefore, each participating application can operate individually without any knowledge of the sequence of steps defined in the process model.

The process manager maintains this state even if a specific application is temporarily unavailable. As a result, the overall execution of the business function is not stopped if a specific application is temporarily unavailable. Instead, the business function can continue after the application is back online.

The process manager interacts with the individual applications by way of Data Integration, Functional Integration, or Presentation Integration, depending on the nature of the interaction and the architecture of the application. Therefore, Process Integration resembles a composite application built on top of existing business functions that reside in existing applications.

Process Integration provides a clean separation between the definition of the process in the process model, the execution of the process in the process manager, and the implementation of the individual functions in the applications. This separation allows the application functions to be reused in many different processes.

The separation of process model and process manager also allows the process manager to be domain independent because it only has to interpret the basic constructs that form a process model. These constructs manage the parallel execution of multiple steps, and they typically comprise sequences, decision points, forks, and joins. Using these constructs, enterprises can create a model at the business level without having to understand the internals of the process manager or the individual applications.

Process Integration involves collaboration between the components that are described in Table 1.

Table 1: Process Integration Components

Component	Responsibilities	Collaborations
Process model	Defines the sequence of steps that make up the business function.	The process manager interprets the process model in separate process instances.
Process manager	- Manages multiple instances of the process model and their current state. - Maps the steps defined in the process model to the business functions that reside in the applications.	- Interprets the process model. - Directs applications through steps according to the process model.
Application	Executes a specific business function.	None.

 

The process manager typically exposes an external interface so that the business process defined by the process model can be initiated by a user, by business partners, or by another application. Correspondingly, the interface can take the form of a user interface or the form of an application programming interface (API) that can be made available to other applications by using Functional Integration, making it essentially indistinguishable from a simple application. Therefore, this business process can be used as part of another overarching business process that is defined using a higher-level process model. As a result, process managers and applications can be linked in a recursive fashion. Figure 2 shows a hierarchy of process managers and applications.

Figure 2. Hierarchy of process managers and applications

Implementation Details

At first glance, the process manager resembles a traditional application that implements business logic. The only difference is that the process manager takes advantage of functions implemented in existing systems. However, the fact that the execution of the business function can take hours or days places specific requirements on the process manager. Specifically, a process manager typically must be able to:

• Correlate messages from external systems with the instance of the business process they are intended for.
• Support long-running transactions.
• Handle exceptions raised by individual steps in the business process, and take appropriate action in the event of these exceptions.
• Provide compensation logic to undo actions committed earlier in the business process if a failure occurs in the business process.

Correlating Messages and Process Instances

Orchestration of processes involves messages sent to external systems and received from external systems. These external systems implement the actions that make up the business process. At any time, there are likely to be many instances of the business process running at once, and any message that is received must be correlated with the correct instance of the business process that it was intended for.

Transactions

Transactions provide encapsulation of individual atomic actions. A transaction is an action or a series of actions that transform a system from one consistent state to another. When dealing with long-running business functions, it is useful to distinguish between two kinds of transactions: atomic transactions and long-running transactions. When you use Process Integration, your design must support both kinds of transactions.

Atomic transactions are the same kind of transactions as those found in databases. They adhere to a set of properties, commonly called the Atomicity, Consistency, Isolation, and Durability (ACID) properties. This type of transaction requires transactional resources, such as a database update or the sending of a message, that permit you to automatically roll back changes if the transaction is stopped. Atomic transactions are suited only for short-running portions of the overall process.

Long-running transactions are used when one or more of the following conditions are true:

• The transactions must occur over an extended time period, and the transaction cannot be supported by atomic transactions.
• The actions involved in the transaction are not themselves transactional, but you want to group these actions into more granular atomic units of work that can exist across time, organizations, and applications.
• Other transactions are grouped within a transaction. Both atomic and long-running transactions may be grouped in a long-running transaction, but other transactions cannot be grouped in an atomic transaction.

For long-running transactions, you must allow the user to stop the process mid-stream. When the process stops, the system will persist data regarding the completed actions and the intermediate states. After the process restarts, the application will reload this intermediate state to allow the process to continue from the point where it stopped.

Handling Exceptions and Compensating Transactions

When external applications are invoked to implement a specific action, a variety of errors can occur. These errors cause error codes to be sent in messages that are returned by the external systems, and these errors cause exceptions to be thrown.

Status code errors can be handled by conditional logic inside the process model. This approach is natural, but can lead to an unwieldy control flow if all possible error conditions have to be covered. Errors in the form of exceptions can be handled by exception handlers attached to the scope in which the exception occurred. This scope can be the entire process model, or it can be a specific subsection. If an exception occurs within this scope, the exception handler is automatically invoked without the process designer having to explicitly model each individual error condition.

Some of these errors require the application to revert the transaction state to the state before the long-running transaction started. When these errors occur, the application issues a compensating transaction.

Example

Process Integration is commonly used to streamline the execution of a sequence of tasks. One popular application in the financial industry is the notion of straight-through processing (STP). STP describes the automated end-to-end processing of transactions such as trades from inception to settlement.

As described in the solution section, Process Integration can also be used to provide an aggregate service to other applications. For example, consider a service that makes concurrent updates to multiple data sources as defined in Entity Aggregation. The implementation of such a service requires Process Integration internally to manage transactional integrity, partial failure situations, and compensation actions.

Process Integration is such a common need that standards committees and working groups are defining standard languages to describe process models. Examples of such languages are:

• Business Process Modeling Language (BPML)
• Business Process Execution Language (BPEL)
• Web Services Choreography Interface (WSCI)

Resulting Context

Process Integration results in the following benefits and liabilities:

Benefits

• Maintainability. Creating a separate process integration layer allows users to define and maintain the business process independent from the implementation of the individual functions. This increases maintainability and reduces the skill set requirements for the personnel who maintain the process definition.
• Reusability. Because the existing applications are not dependent on the process management layer, the functions inside these applications can be reused in multiple process definitions.
• Flexibility. The process manager can support a variety of configurations that would be difficult to implement in many traditional programming models. For example, parallel execution of multiple tasks, synchronization points, timeouts, and escalations are all configurations that would be difficult to implement in a traditional programming model. Supporting a variety of configurations gives the process manager the flexibility to adapt to many different business requirements.
• Reporting capability. Because the process instances are maintained centrally, it becomes feasible to extract statistical information that spans multiple process steps and process instances. Such reports can provide answers to questions such as "How long does it take on average to fulfill an order?" or "How many orders are on hold due to insufficient inventory?" This type of reporting can be instrumental in making informed business decisions.

Liabilities

• Potential bottleneck. Managing a large number of process instances in a central process manager component can present a run-time bottleneck. However, in most cases, the parallel execution of multiple process manager instances can alleviate this threat while maintaining the benefit of central configurability.
• Temptation to overuse. This liability is the flip side of the flexibility inherent in Process Integration solutions. Because Process Integration can be used to solve nearly any integration problem, some people take this as an indication that each and every integration problem should be solved using Process Integration. This is not true. Using Process Integration frivolously can lead to overarchitected and sometimes inefficient solutions—for example, in cases where a Message Broker would be perfectly appropriate to address the requirements.
• Complexity. A generic process manager can be difficult to build because it has to deal with concurrency, checkpoints, and recoverability. For these reasons, a commercial process manager should be used whenever possible. If a commercial component is not available, the cost of building the process manager should be carefully weighed against the cost of making changes to the process model.

Testing Considerations

Separating the process definition from the functions provided by the applications allows you to test each function individually by creating a simple test driver that only calls one specific function and that verifies the results, as shown in Figure 3.

Figure 3. Test driver

Likewise, you can test the process manager by creating placeholder implementations, or stubs, of the actual services while providing test triggers to the process manager (see Figure 4).

Figure 4. Using stubs to test the process manager

Related Patterns

For more information, see the Process Manager pattern [Hohpe04]. Process Manager describes the internal function and implementation of the process manager component.

Acknowledgments

[Hohpe04] Hohpe, Gregor; Bobby Woolf, Enterprise Integration Patterns: Designing, Building and Deploying Messaging Solutions. Addison-Wesley, 2004.

[Ruh01] Ruh, William. Enterprise Application Integration. A Wiley Tech Brief. Wiley, 2001.

Start | Previous | Next