Introducing the Orchestration Engine
Orchestration picks up where messaging leaves off. The abstraction between messaging and orchestration is not coincidental. Whereas messaging is concerned with the details of sending, receiving, routing and consuming messages, orchestration addresses the non-technology business domain, keeping plumbing implementation details out of the business process. Orchestration then, is all about taking a business process and scheduling, or orchestrating how a series of activities should take place into a workflow.
|BTS 2006 takes messaging far beyond SOAP and HTTP.|
The Orchestration Engine, supported by the beautifully rich and intuitive Orchestration Designer, is one of the most important aspects of BizTalk Server. The ability to model business process while avoiding the plumbing code required to elegantly coordinate sophisticated business workflows while abstracting messaging minutia is perhaps the most compelling reason to consider BizTalk Server for your application integration projects.
Orchestration is all about coordinating the actions and responding to events of constituent services or components that work together to support a business process. As such, orchestration instances consist of one or more transactions (or a series of which can be nested). It is common to refer to orchestrations as schedules. The term is apt, because orchestrations are not very code-heavy. Orchestration merely schedules and coordinates the work of other services and components to which the same guidelines for object-oriented, component-oriented, and service-oriented programming apply. The ability to represent, communicate, and maintain these complex transactions as visual workflows is extremely powerful, as is the messaging paradigm on top of which the Orchestration Engine is built.
Orchestrations, or schedules, are compiled and deployed to a BizTalk host and can be thought of as a class or component for which instances are created on demand. When a message hits the BizTalk MessageBox database, a list of subscribers (typically orchestrations) are checked and the subscribing orchestration fires based on subscription criteria specified within the orchestration itself. This publish-subscribe model provides a very elegant approach to loosely coupled messaging. Figure 3 provides a conceptual illustration of messaging and orchestration in action.
|Figure 3. Key Components: A conceptual view of key components of messaging and orchestration in action (courtesy of the Microsoft BizTalk Server 2006 documentation team).|
It is important to remember that all messages that hit the MessageBox
are serialized to XML. Therefore, when a message activates an orchestration instance, the message itself becomes available to the orchestration as XML. To interrogate elements or fields in a message instance within an orchestration, XSD schema (or serialization equivalent) must be used to mark the elements as either Promoted Properties or Distinguished Fields. Identifying requisite fields using one of the two options creates metadata that will allow the Orchestration Engine to expose these fields as accessible properties to the runtime. The decision between using Promoted Properties over Distinguished Fields has to do with message routing. Of course, since the message itself is XML, you can write custom XPath expressions against the message instance using the Expression Editor (covered under Developer Tools). Note, however, that using Promoted Properties and Distinguished Fields can incur a performance penalty. Instead of going too much deeper into this topic here, I will demonstrate the use of each during the implementation of an EAI solution.
Within orchestration, messages are mapped to message or variable containers. You use shapes
to represent various activities for interacting with messages including expressions and flow control to model common programming semantics such as loops, message assignment, and variable initialization. Because orchestration is largely visual, you'll use shapes and properties to do most of the work. One exception is the Expression Shape, which supports a very lightweight version of the C# syntax. You use Expression Shapes to work with .NET types and make calls to .NET components. I'll provide more information on common shapes along with practical examples of several shapes later in the article.
Because orchestration coordinates the work represented by a business process, naturally, an orchestration can include one or more transactions. Unlike traditional programming, BizTalk Server supports two distinct types of transactions: atomic and long-running. You can set an orchestration with the transaction type at the orchestration level or, if you need a finer level of granularity for configuring transaction behavior, you can define a construct known as a scope to wrap a unit of work within a transaction boundary.
In order to understand the two different types of transactions in BizTalk, let me review traditional (or classic) transaction concepts. Traditional transactions possess several characteristics that can be described by using the ACID acronym:
- Atomic. All activities that are part of the transaction boundary complete as a unit of work.
- Consistent. Any work resulting in changes to data (presumably within a database or in memory objects) must be left in a consistent state regardless of transaction outcome. Should the transaction abort, the data is returned to the state it was in prior to the transaction.
- Isolated. Changes to the state of objects are isolated within the unit of work such that concurrent access to the data being operated on is unambiguous.
- Durable. The changes as a result of a committed transaction persist even in the event of a system failure.
ACID transactions have long been the staple of reliable database programming. Component-oriented technologies such as COM+ and .NET bring ACID transactions to the middle tier courtesy of the Microsoft Distributed Transaction Coordinator (DTC) and the System.Transactions namespace in .NET 2.0. As any experienced database or system developer knows, you must take great care in instrumenting ACID transactions. The unit of work within an ACID transaction must happen as quickly as possible due to the fact that maintaining isolation within transactions is both resource intensive and often requires exclusive read and/or write access to the underlying objects or data structures. That said, ACID transactions are a critical requirement for applications that require any degree of reliability and data integrity.
In BizTalk, you can design orchestrations to support ACID transactions by configuring the scope (or the entire orchestration itself) as atomic. Any variables, messages, etc., within an atomic scope are isolated within the transaction and changes to their state are not visible until after the transaction commits. Atomic transactions are married with the appropriate isolation level. Choices include serializable
(object-level locking), read committed
(only committed data is readable) and repeatable read
(row-level locking occurs to prevent "dirty" reads).
It is important to understand that if an orchestration is marked as atomic, the initiating message will not be removed from the MessageBox until the transaction commits. In addition, any work performed within an atomic scope by .NET components will not participate in the transaction unless the .NET component inherits from System.EnterpriseServices.ServicedComponent and is transaction-aware. In this case, the .NET component will be enlisted as part of a COM+ DTC-style transaction and will participate as expected in a two-phased commit. This distinction is very important because the work performed by vanilla .NET components will be persisted regardless of the outcome of the transaction.
As discussed, one area in which ACID transactions are simply not appropriate is when the work to be done cannot be performed in a very short period of time. This condition is pervasive in modern Service-Oriented Architecture (SOA) applications where the unit of work to be performed can span multiple service endpoints. Maintaining highly disciplined services that are resilient to deadlocking issues is a challenge in itself, let alone when you have little or no control of the service being consumed itself.
Another common scenario is a business process that might take seconds, minutes, or even days or weeks to complete. This is when a long-running transaction is more suitable. The best way to explain the need for the concept of a long-running transaction is with an example.
Consider an airline booking system. The system reserves seats for passengers pending a credit card authorization. Given the tendency of airlines to overbook flights, the would-be passenger's ticket may no longer be available on the day of the actual flight. In such a case, the airline might have to refund fees for the fare of the flight and provide additional compensation such as adding a number of miles or points to the disgruntled customer's "rewards" balance. This scenario is not ideal but is likely to occur during peak travel times.
An ACID transaction would clearly be inappropriate for this transaction. It would likely be impossible to maintain isolation over the course of the days, weeks or months leading up to the actual flight. It wouldn't be long before locking issues brought the booking system to its knees. Atomicity, in the classic sense, is also not tenable because the outcome of the transaction is non-deterministic-there are variables (such as whether the flight will be oversold, or if the passenger will arrive at the gate on time), that simply aren't available to the transaction at the time the tickets are reserved.
All is not lost, however. In such an example you can preserve the classic transactional properties of durability and consistency. By persisting the state of the transaction to the MessageBox after the reservation takes place, the transaction itself becomes durable. As soon as the customer boards the plane (triggering an event that notifies the booking system), the reservation system can resume the transaction and the business process completes any remaining activities such as entering the passenger's information in the flight manifest and ensuring that the passenger's seat cannot be released. Such is the case for a long-running transaction.