Introducing BizTalk Server 2006
Microsoft BizTalk Server 2006 (the fourth release of Microsoft's flagship messaging and business process management platform) provides a compelling messaging solution and complementary toolset for addressing many of the design goals typical in common EAI scenarios.
|Design goals such as availability and reliability often fall beyond the reach of traditional Web services.|
Built entirely on the .NET Framework, Microsoft BizTalk Server 2006 (BTS 2006) provides the foundation for business process automation and messaging, two key areas that are common within many integration scenarios. Through its support of messaging and workflow, BTS 2006 is a powerful addition to the EAI space, introducing revolutionary advances in multi-protocol messaging, business process design, management, and debugging. The beautifully intuitive Orchestration Designer (installed as part of Visual Studio 2005) provides a glimpse into the future of building modern, workflow-modeled software.
At its core, BTS 2006 is comprised of the Messaging Engine and Orchestration Engine. Both components are mutually exclusive and each addresses a very specific, yet complementary aspect of application integration. BizTalk Server 2006 features much more than messaging and orchestration, however, for the purpose of this article, I will focus on these important concepts.
While certainly not exhaustive; I will give you a high-level overview of the various messaging and orchestration concepts and corresponding development and deployment-time artifacts before I walk you through an application integration project with BTS 2006.
Much More than Just Messaging
One of the most confusing (or at least misunderstood) aspects of BizTalk is messaging. While messaging is certainly the bedrock of BizTalk, there is a common misconception that BizTalk's messaging engine is just a proprietary way to develop and deploy Web services. This is a false assumption, and as a matter of fact, Web services (SOAP, for example) are only one of a myriad of messaging choices at your disposal when choosing BTS 2006 to attack your application integration chores.
Messaging, as the name implies, is concerned with how data (the payload within a message) is sent, received, routed and consumed. In BizTalk, the Messaging Engine deals with low-level protocols and file formats as well as design goals such as reliability and availability. People commonly think of messaging in terms of Web services, where you represent the data or the message as XML, and you place this XML in a SOAP envelope and transmit it over HTTP. While the analogy is a valid one, BTS 2006 takes messaging far beyond SOAP and HTTP.
Despite its popularity, XML is still a rather new file format and many legacy applications cannot consume or conduct messaging in XML. To address this disconnect, BTS 2006 can work with many file formats. BizTalk Adapters support the use of common "flat" file formats such as comma separated (CSV), delimited, or positional files. These Adapters are pluggable software components written in either .NET or COM, that provide broad messaging support and the glue that helps integrate both new and legacy applications with BizTalk services.
In addition to supporting various file formats, the Messaging Engine is extremely robust and helps to address many of the shortcomings of hosting traditional Web services. With typical Web services, a client consumes a service. The service is exposed by an endpoint, typically a URL. When a client invokes a Web service, a series of serialization and deserialization steps take place. An in-memory object on the client side is converted using serialization to an XML format and placed in a SOAP envelope so that it may be sent over an HTTP connection. The service (behind the endpoint), receives a request, processes it, and optionally sends a response. More often than not, this process should occur very quickly because the client calling the service does so in a synchronous (or blocking) manner. That is, the client invokes the service and then patiently waits for a response.
For well-disciplined services, where calls are short and efficient, the synchronous pattern can be a very effective way of managing reliability. Unfortunately, this calling pattern is not always ideal. Service orientation has evolved far past RPC-style services in which a caller asks a component to do some simple work and expects a quick and trivial response. For example, you can expose complex business workflows such as inventory, point of sale, customer relationship management, and supply chain management, via service endpoints in order to leverage these services in an autonomous location-transparent manner. The service(s) behind these endpoints may consist of a long-running process that is either standalone and monolithic or distributed elegantly across other services (because services are autonomous, these details are transparent to the client). In this scenario, the expectation for the client to wait several seconds, minutes (or even hours and days) is unreasonable, if not impossible.
One possible solution to this dilemma is not a new one. In such scenarios where you have a long-running process, you might choose to invoke the service asynchronously (sometimes also referred to as "fire and forget"). Using this approach, a client invokes a service, only this time, instead of consuming client resources waiting for a response, the service returns immediately to the caller while carrying out the work in its own process.
Consider the implications of a transaction being hosted by a service in which a message is sent to a service which consists of several potential steps. How does the client know if the messages are received? What happens if the server that hosts the service goes down in the middle of a complicated business process? In the event that step three, say, in a five-step order fulfillment workflow fails (i.e., the service endpoint it calls is unavailable or returns an error), how will the transaction behave? Can it be resumed?
|The ability to model business processes while avoiding the plumbing code required to elegantly coordinate sophisticated business processes while abstracting messaging minutia is perhaps the most compelling reason to consider BizTalk Server for your application integration projects.|
This problem domain is by no means limited to services exposed via SOAP over HTTP, however, the vast majority of services today are deployed as such and are simply much too fragile to handle such complex workflows. Implementing some of the more advanced specifications and recommendations, commonly referred to as WS-STAR, is cumbersome at best and the result is that design goals such as availability and reliability often fall beyond the reach of traditional Web services. On a side note, Windows Communication Foundation is a huge leap forward in this regard and provides turnkey support for WS-ReliableMessaging, WS-Coordination and WS-AtomicTransaction specifications to name a few.
Fortunately, BTS 2006 addresses many of these design aspects using a highly effective messaging model that leverages Microsoft SQL Server as a reliable, scalable, and highly performant persistence mechanism. The BTS 2006 Messaging Engine is not only fully capable of communicating via SOAP and HTTP, but it supports additional transport and application protocols such as MSMQ, WebSphere MQ, FTP, SMTP, and file drops (to name a few).
Figure 1 shows workflow components such as those provided by BizTalk orchestration. Figure 2 presents a high-level conceptual diagram to summarize adapter support in BTS 2006. I will cover these important aspects of messaging in the sections that follow.
Adapters and Pipelines
|Figure 1. Workflow Components: Workflow components such as those provided by BizTalk orchestration are fundamental to .NET application architectures (courtesy of Microsoft Patterns & Practices).||
|Figure 2. BizTalk Messaging Engine: The BizTalk Messaging Engine supports several adapters natively, including SOAP, MSMQ, and File.||
Adapters and pipelines are fundamental aspects of BizTalk Server messaging.
Adapters are pluggable software components (written in either .NET or COM) that support the integration between BizTalk and various applications (custom and/or ERPs), protocols, platforms, and technologies. Table 1, taken from the official BizTalk Server 2006 Documentation (see Additional Resources) provides a summary of native adapters available natively with BTS 2006.
Table1: Native adapters that ship directly with BizTalk Server 2006 (courtesy of the Microsoft BizTalk Server 2006 documentation team).
|Web services Adapter
||Supports sending and receiving messages using SOAP over HTTP. Because SOAP is the core protocol for Web services, this adapter is critical for BizTalk Server 2006's ability to interact in a service-oriented world. As usual with Web services, URLs are used to identify the sending and receiving systems.
||Supports reading from and writing to files in the Windows file system. Because the applications involved in a business process can often access the same file system, either locally or across a network, exchanging messages through files can be a convenient option.
||Supports sending and receiving information between a File Transport Protocol (FTP) Server and BizTalk Server.
||Supports sending and receiving information using HTTP. The BizTalk Server 2006 engine exposes one or more URLs to allow other applications to send data to it, and it can use this adapter to send data to other URLs.
||Supports sending and receiving messages using Microsoft Message Queuing (MSMQ).
||Supports sending and receiving messages using BizTalk Message Queuing (MSMQT). MSMQT is an implementation of the MSMQ protocol that can receive and send MSMQ Messages into the MessageBox. Although BizTalk Server 2006 still includes this adapter, developers should use the MSMQ adapter instead.
|WebSphere MQ Adapter
||Supports sending and receiving messages using IBM's WebSphere MQ (formerly known as MQSeries).
||Supports sending messages using SMTP. Standard e-mail addresses are used to identify the parties.
||Supports receiving e-mail messages and their attachments using version three of the Post Office Protocol (POP3).
||Supports accessing and publishing documents stored in Microsoft Windows SharePoint document libraries.
||Supports reading and writing information to a SQL Server database.
|Base EDI Adapter
||Supports sending and receiving messages in the electronic data interchange (EDI) format.
In addition to this broad offering, Microsoft generously makes the Adapter Framework-a set of interfaces and SDKs for building custom adapters-freely available to the public. This is the same API on which native and commercial offerings are built and provides endless possibilities for connectivity.
Essentially, receive adapters move external data that arrives to the appropriate receive pipeline (covered shortly) and send adapters take prepared data from send pipelines and dispatch it to the configured destination. Adapters use various patterns to communicate, and you can organize these patterns directionally into Send and Receive:
|Since the advent of component technologies, the Internet has provided unprecedented levels of interconnectivity between systems and networks, accelerating globalization at an alarming rate.|
- One Way Receive. Receives a message asynchronously and passes it to a corresponding receive pipeline.
- One Way Send. Performs the complimentary action of the One Way Receive pattern, presenting an outbound message to the configured destination in an asynchronous manner.
- Request Response. Can perform both receive and send operations (as the service) typical in synchronous SOAP communication models.
- Solicit Response. Sends a message, such as with a synchronous SOAP call, and the adapter blocks until a complementary response is received.
A pipeline is analogous to an assembly line in which messages are either composed (when sending a payload) or decomposed (when receiving a payload). The pipeline primarily takes non-XML input from a corresponding adapter and converts it to XML so that the Messaging Engine can persist it to the database. In addition to processing received data, pipelines also take XML and compose it back to the native adapter format. As such, pipelines can be thought of as foreign language interpreters that translate data from foreign formats into native messages that can be understood by BTS (and vice versa).
Like adapters, you can extend existing pipelines and you can create new custom pipelines using the same API and interfaces used to build the pipelines that come out of the box. You may want to build a custom pipeline for several reasons, including: custom authentication, authorization, logging, or message validation using XSD schema.