BizTalk and Application Integration, cont.

BizTalk Messaging Services

Many of the low-level messaging services performed by BizTalk Server, such as sending, receiving, parsing, and tracking message documents; and data mapping, are grouped together and termed (unsurprisingly) BizTalk Messaging Services. In this section we are going to learn more about how BizTalk Server implements messaging.

As we progress through this section, bear in mind that in order to achieve application integration, all messages must pass through BizTalk Server before they reach their destination. Remember that BizTalk Server is a centralized means of routing and translating documents. Therefore a message transfer actually consists of two steps:

1.         A message is sent to BizTalk Server

2.         BizTalk Server sends the message on to another application

We shall start this section by introducing a few BizTalk Server-specific definitions associated with messaging. These will be used in later chapters. Then we will look at how BizTalk Server actually sends and receives messages, and we will finish by discussing the advantages of queueing messages.

Organizations

An organization in BizTalk is a fairly broad concept. It can encompass a whole business, or a distinct entity within a business. Remember that earlier in the chapter we described a business process (workflow) as a series of islands of automation linked by the exchange of documents? If we continue with this theme, then each island will be represented in BizTalk as an organization.

The Home organization is the object that represents your business in BizTalk; it is created for you by BizTalk Server automatically. You can rename it to the actual name of your business to make it easy to identify in the workflow.In a simple model where a company is sending a document to a trading partner, the partners and organizations exactly correspond. In more complex situations involving multiple applications within a single company, it is better to think of each entity in control of each application as an organization.

BizTalk can select a channel (explained below) for message delivery based on the organizations involved, together with the type of the document being sent. You must be sure that your definition of each organization is sufficiently detailed to uniquely identify the proper receiving party. The organization is therefore the point of contact for the exchange of messages.

Document Definitions

The document definition refers to the type of message document that is processed by BizTalk Server. It provides a pointer to a particular document specification, which is data about what the file can contain - hence it is meta data (data about data). Note that any particular specification can be used in any number of document definitions, and any particular document definition can be used in any number of channels.

BizTalk supports a variety of message types, such as XML and EDI. We shall see later that the navigation pane of the Document definitions section of Messaging Manager contains a listing of all the document specifications known to a particular BizTalk Server installation. This information is maintained in the BizTalk database.

Envelopes

Envelopes are conceptually similar to real-world, paper postal envelopes. When you want to send a document to a partner, you will specify an envelope in which the document travels. Unlike real-world envelopes however, BizTalk associates a particular envelope with a particular type of document.

You should note that envelopes are not strictly required for XML-format messages. Because we tag data in XML, BizTalk can look at the root-level tag to determine document type. However, they are critical in flatfile formats: the envelope contains information about the document that BizTalk needs to route the message. If you submit a message with routing information based on the sending and receiving organizations, Messaging Services must find a channel to use based on the organizations and the document type. Without an envelope, BizTalk would be unable to determine the document type for non-XML formats.

The Envelopes section of Messaging Manager permits management and creation of envelopes for BizTalk documents.

Ports and Distribution Lists

A port is a set of properties that directs how documents are enveloped, secured, and transported to a designated destination organization or application.

By specifying a port to use, two applications can exchange a message without either of them needing to know how the message is transmitted (protocol) or where the message is delivered to (the end location). The applications do not need to involve themselves in actual message delivery either; that is the responsibility of Messaging Services. At runtime, Messaging Services consults the properties of the named port to determine how and where to send the message it has been given. By removing responsibility for message transmission from the applications themselves, the applications can remain unchanged even when the end location or protocol changes.

For example, an application might start out using MSMQ - naming a specific queue in the process - and later move to HTTP when it is web-enabled. The application's port would initially specify MSMQ and the designated port, then would shift to HTTP and, say, a particular ASP page on a particular server as the new endpoint. So long as the port definition is updated in the Messaging Manager, the sending application would not notice the change.

There will be occasions when we need to broadcast a particular message to several applications. Each application or organization that should receive a message needs a port designated for message delivery. It would be unwieldy to have to configure one-to-one relationships with each recipient, so Messaging Manager allows you to select a collection of ports and give them a name. This named collection is known as a distribution list.

Distribution lists are also termed port groups in the configuration object model that we will explore later in this book.

At runtime, Messaging Services notes the use of a distribution list and sends the submitted message to each port in the list. Note that each port may potentially use a different protocol, so sending a message to a distribution list can result in multiple transmissions over differing protocols.

Channels

A channel is a set of properties that designates the source of documents, and also defines specific processing steps that must be performed by BizTalk Messaging Services before a document is delivered to the destination.

The destination is designated by the messaging port. By themselves, ports and message specifications are not specific to any particular message exchange in the workflow, and so they can be reused whenever necessary. By contrast, when associated with a particular port and message specification, channels can be used to implement a specific message exchange within the workflow.

Channels are an important link between the low-level configuration details of Messaging Services, and the high-level workflows of the XLANG Scheduler. Specific actions within a XLANG schedule are designated by naming a channel. At runtime, when XLANG Scheduler attempts to execute a particular action, it invokes the channel, thereby bringing Messaging Services into play to actually send the message.Transactions

A transaction is a set of actions that are grouped together into one discrete unit. When the transaction is implemented at runtime, if all the actions are carried out successfully, the transaction succeeds. If any of the actions cannot be carried out, the transaction fails and the data flow is "rolled back" to the beginning of the transaction.

Why should we need to employ such a system? Well, there are many occasions where the failure of one action has an important knock-on effect to other actions. For example, say we had a system where money was being transferred from one bank account to another. Here we would need (a minimum of) two actions:

1.         Debit the original account

2.         Credit the new account

We could place these actions in the workflow, but consider what would happen if the attempt to credit the new account failed. We would be left with a situation where the original account had been debited but the new account has not been credited: money is lost because of the error.

On the other hand, if we tie the two actions together in a transaction, the transfer will only occur if both actions are successful; otherwise the workflow will be rolled back to the start of the transaction. In both cases no money is lost. We will learn more about transactions in Chapter 2.

Message Transfer

BizTalk Server uses two techniques for transferring messages between applications:

q                      Open Internet protocols

q                      Windows-specific communications

Let's now explore both of these options more closely.

Open Protocols

The Web relies on open (public) protocols for communication between applications. Any product interested in performing application integration over the public Internet, therefore, must support these protocols in order to support common strategies for implementing Internet applications. Intranet applications are becoming common in business process integration situations, and they often use these protocols too.

It should not be surprising, therefore, that BizTalk Server supports the major web protocols for the transport of data interchange documents between applications. These are the protocols BizTalk Server supports for communications:

q                      Simple Mail Transport Protocol (SMTP) - communications via e-mail

q                      Hypertext Transfer Protocol (HTTP) - the communications mechanism used by web pages and applications (ASP, CGI, and so on)

q                      Secure Hypertext Transfer Protocol (HTTPS)- a version of HTTP that uses the Secure Sockets Layer (SSL) to encrypt traffic passing between the client and server

Of the three, HTTP and HTTPS permit the most interactive form of communication. It is possible to implement an application as an ASP page or CGI script and connect it directly to BizTalk Server. When a message is received for the HTTP-based application, BizTalk Server makes a request of the designated page or script. The results of executing the script can, in turn, be a reply message sent through BizTalk Server to the sender.

SMTP requires that a script be run upon receipt of the e-mail message. This script can perform the desired application functions or use other mechanisms to invoke applications. Because of the nature of e-mail, SMTP communications may not be immediately responsive.Message Security: Active Directory

The Active Directory is a directory service that Windows 2000 uses as a central, hierarchical store of system configuration information. This information includes security principles as well as digital certificates. Certificates are used to authenticate message interchanges, as well as optionally performing encryption on the message contents. BizTalk Server also supports the use of secure Internet protocols such as HTTPS, SSL, and secure Multipurpose Internet Mail Extensions (S/MIME), which can be used for the encryption of messages passing between partners. As mentioned above HTTPS and SSL work together to protect HTTP transfers and form the basis for most commercial Internet purchasing pages. All of these use digital certificates to authenticate the partners to a conversation and perform encryption.

Active Directory is accessible through a variety of administrative tools and is capable of communicating with any client that supports the Lightweight Directory Access Protocol (LDAP), an open standard protocol for directory access.

Windows-Specific Communications

There will be times when you want to use communications that are specific to the Windows operating system. BizTalk supports the following proprietary protocols:

q                      Microsoft Message Queue (MSMQ)

q                      Reading and writing files to the local file system (file transport)

q                      COM+

MSMQ is a feature of the operating system that implements messaging middleware. You can use MSMQ to transfer chunks of data, termed messages, to other computers using a queuing system. The queues implemented in MSMQ provide various levels of reliability and recovery. The entire queuing system, in addition, includes features designed to work around partial communications outages in the network. There are also products that enable MSMQ to communicate with IBM MQ Series messaging middleware.

Many legacy applications exchange data with the outside world solely through files on disk. They take in data by reading a file and send data to other programs by writing a file. BizTalk treats such files as messages by providing a service that can monitor a designated folder for the arrival of files, then read those files and submit them to Messaging Services, where they may be handled like any other message. This provides a valuable link to a wide range of applications that have no other communications protocol.

COM+ is the core implementation technology for Windows. Major subsystems and supporting services are implemented and exposed to applications through COM components. The recommended application design philosophy for Windows applications has long been component software using COM or COM+ as the component technology. It should not be surprising, then, that BizTalk Server is implemented using COM+, and applications often integrate most closely with BizTalk Server using COM+ interfaces.Even when you integrate two applications using some other protocol, for example HTTP, you will end up using COM+ at some point in the process. BizTalk Server provides COM+ components to look for inbound documents on other protocols, and complementary components for forwarding documents on other protocols. Integration using non-COM+ protocols is a matter of configuring receive and transport COM+ components through the Messaging Manager tool we discuss later on in the chapter. Core Messaging Services deals with these COM+ interfaces.

You can extend BizTalk's capabilities by building custom COM+ components or using BizTalk-related COM+ interfaces. For example, you can use the COM+ interfaces offered by BizTalk to perform all the management tasks you can perform manually through the Messaging Manager. You can also take advantage of BizTalk COM+ interfaces to monitor and dynamically manipulate XLANG schedule files (which describe the workflow of a business process).

If an application is written with BizTalk in mind, a tighter degree of integration than using open protocols can be performed. Applications that that wish to exchange BizTalk messages can directly invoke methods that belong to COM+ interfaces. Let's take a closer look at the COM+ interfaces most commonly encountered in BizTalk messaging.

Sending Messages to BizTalk Server

Documents are sent to BizTalk Server by invoking the Submit() or SubmitSync() methods of the IInterchange COM+ interface.

An application can call either the Submit() or SubmitSync() method directly, passing the document or the file path across to the method. However, this direct form of messaging is only available if the application is designed to support direct calls to BizTalk Server.

If the application is not capable of invoking COM+ methods on BizTalk Server directly, there is another option available. We can use receive functions to submit messages to BizTalk Server. Receive functions are configured to monitor specific directories on a server where message documents are placed. When it detects a new message, the receive function submits it to BizTalk Server. Receive functions can also be configured to monitor message queues (we will discuss these later) and submit messages from the queue to BizTalk Server.

Receiving Messages from BizTalk Server

BizTalk Server uses COM+ components known as Application Integration Components (AICs) to deliver data to an application. BizTalk Server calls, and passes the message data to, the AIC, which in turn passes the data on to the receiving application. The first question that springs to mind is: why bother? Why not just let BizTalk Server send the data directly to the application?

The answer is that, by doing this, we remove the need for BizTalk Server to understand how to communicate with the application. There are many applications that can only be communicated with using a non-standard proprietary protocol. So, by placing an AIC between the Server and the application, we can let BizTalk Server communicate with the AIC using a standard protocol, and let the AIC worry about how to transfer the message data to the application. If we need to communicate with a system that doesn't use a standard protocol, instead of having to add functionality to BizTalk Server, we produce a custom AIC that can talk to the application. This concept is illustrated below:

 

For example, SAP is a (non-XML based) Enterprise Resource Management System, which is used to manage the supply chain and inventory of very large companies. SAP provides an AIC that allows documents from BizTalk to flow directly into SAP's server.

For BizTalk to communicate with the AIC, the AIC needs to implement one of two interfaces:

q             IBTSAppIntegration

q             IPipelineComponent

Components that implement IBTSAppIntegration are known as "lightweight", because the interface does not allow BizTalk to configure the AIC. This interface contains a single method, ProcessMessage()through which a message document is passed to the interface.

A different approach is to create what is known as a pipeline COM+ component to communicate with the application. The idea of a pipeline derives from Microsoft Commerce Server, where a pipeline is a chain of custom COM+ components arranged by the site designer to execute in a particular order. A pipeline is designed to implement some multi-step task, for example purchasing goods. Each component is responsible for implementing a single step - sales tax calculations on purchases for instance. Commerce Server passes a Dictionary object holding the information built up during prior steps in a pipeline to the component. In this way, Commerce Server is able to support the construction and execution of arbitrary functionality.

For more on Commerce Server and pipelines, see Professional Site Server 3.0 Commerce Edition (Wrox Press, ISBN 1-861002-50-5).

The point of pipeline components is to be able to produce custom COM+ functionality. On the other hand, all pipeline components must have a common framework so that the Server knows how to communicate with them. Therefore all pipeline components must implement an interface called IPipelineComponent, which contains several methods, one of which, Execute(), enables the functionality of the component. The Dictionary object is one of the parameters passed to Execute(). The IPipelineComponent interface permits properties whose nature is not known until runtime to be passed to the component.

BizTalk Server also supports pipeline components; it invokes the Execute() method to pass the data to the component. Using pipeline components, BizTalk can configure and invoke arbitrary custom pipeline processing, and Commerce Server can make use of BizTalk to enhance the capabilities of commerce pipelines. It also gives BizTalk Server a potential pool of custom developers in the existing community of Commerce Server component builders.

Message Queueing

BizTalk Server places messages in queues. Message queuing is important for two main reasons:

q                      reliability

q                      scalability

Messages are buffered in queues on the sending and receiving machines. This promotes reliability because loss of connectivity between the sending and receiving machines will not lead to loss of a message. Queued messages are then delivered when connectivity is restored.

Queuing also promotes scalability because receiving applications do not have to process the peak volume of message traffic, only the average volume. The sending application can send a message and then go about other tasks even if the receiving application is experiencing a backlog. The message is not lost; it remains in a queue until the receiving application can clear the backlog.

The default messaging middleware product for BizTalk Server is Microsoft Message Queue (MSMQ), which allows applications to send messages between queues on one or more machines. In addition to the features we mentioned earlier in passing - queuing, buffering, network reliability - MSMQ can also perform message routing, encryption/decryption, and all of the other services needed for proper delivery of messages. In a sense, MSMQ is "e-mail for applications".

Note that although MSMQ is a major messaging middleware product for the Windows platform, it is far from being the only one. Messaging middleware has a long history in mainframe processing. For example. IBM MQSeries is another product with a long history of use on various platforms. MSMQ supports MQSeries on Windows via the MSMQ-MQSeries Bridge from Level 8 Systems, and MQSeries on other platforms via Level 8's FalconMQ product.

Obviously, this is a tremendous simplification of messaging middleware in general and MSMQ in particular. MSMQ is covered in depth in Professional MTS & MSMQ Programming with VB and ASP (Wrox Press, ISBN 1-861001-46-0). More details on MSMQ and MQSeries interoperability may be found at www.level8.com/falcon.htm.