We now conclude this chapter with a closer look at the tools which
BizTalk Server provides for implementing application integration. The
idea of this section is to give you a taste of each tool, before we
use them extensively in later chapters.
Defining the Business Workflow: Orchestration Designer
The whole point of BizTalk Server is application integration, and
the reason why we want to integrate applications in the first place
is to implement some workflow. A business supply chain is an example,
specifying the interrelationships of customer orders, manufacturing
scheduling, inventory management and purchasing software.
Consider how workflows are usually designed. They are designed at
the highest level, which a manager or business analyst understands,
although at a deeper level they involve specific technical issues
such as transactions. Ideally, such a system is implemented by
starting at the highest level too, drawing a flowchart of the
individual activities needed to define and implement the workflow.
Then the specific details of integrating each activity are added
in.
Note that the activities themselves, however, are accomplished by
the applications you wish to integrate. It may be necessary to write
the odd utility or component, but by and large the idea is to
integrate existing applications.
The process of designing and implementing a workflow is known as
orchestration. Orchestration Designer is the tool provided by
BizTalk Server for accomplishing the design task. It is highly
graphical - in fact, it consists of VBA on top of Microsoft Visio (a
package used for business diagram creation).
Analysts and programmers start the workflow design by drawing a
flowchart of the business process. Systems in BizTalk are built by
exchanging messages between applications, so the next step is to
specify the messages to be sent or received at each step, and the
locations from which messages are received or to which they are
sent.Finally, a designer provides Orchestration Designer with a
high-level view of the workflow through the system. This is more
important than it sounds at first. Since BizTalk Server uses a
messaging model for applications, the workflow itself is viewed by
BizTalk as a series of transformations or modifications of data. The
flow of data, then, is the application. An example workflow
flowchart is shown below. The data flow, not visible in the view
below, details how data enters the system in a message, then flows to
other messages which are sent out.
|
|
In fact, you will see a variation of the flowchart above in the
next chapter when we begin our sample application. This flowchart
will explain how a field agent sends a message describing a
residential property back to a BizTalk Server at the home office.
Rules in the flowchart are applied to the message to determine
whether certain services should be ordered on behalf of the
homeowner. If they are to be ordered, the flowchart directs messages
be sent to the business partners whose supply those services.
Orchestration Designer permits you to specify message exchanges
with Microsoft Message Queue, COM or script components, or its own
Messaging Services. It supports transactions (several actions that
must succeed or fail as a whole) and concurrency (two or more
processes occurring at the same time). These features are so
important to building systems with orchestration that we come back to
them later, devoting Chapter 6 to their study.When you have finished
specifying a workflow, Orchestration Designer exports an XML document
that defines the workflow. This document, termed a XLANG
schedule, is later used by XLANG Scheduler to run your distributed
system by controlling how and when your applications are sent
messages.
It should be stressed that Orchestration Designer only specifies
the high-level semantics of workflows. It orchestrates the activities
of lower-level functions, chiefly Messaging Services and your
individual applications and components, to compose a complete,
distributed business application.
Orchestration Designer and the fundamentals of XLANG schedules are
covered in Chapter 2, while advanced features of schedules are
discussed in Chapter 6.
XLANG Scheduler
BizTalk Server installs a default COM+ application called XLANG
Scheduler. Its function is to host the running instances of
XLANG Scheduler Engine. In turn, this Engine is a service that
manages and runs instances of XLANG schedules.This somewhat confusing
dichotomy - a COM+ application and a service with a similar name -
allows any new COM+ application to host the Scheduler Engine. This
permits great flexibility in how BizTalk systems are deployed,
managed, and run, because writing a custom COM+ application that
hosts the scheduling services allows you to take control of various
aspects of scheduling. However, in general, and in this book, you
will be using default XLANG Scheduler and its default support for the
scheduling services.
The XLANG Scheduler Engine loads XLANG schedules - the XML
documents produced by Orchestration Designer - and executes them. At
its most basic, this involves calling Messaging Services in response
to actions specified in the schedule.
There is a less obvious function of the Scheduler, however, and
that is managing the resource utilization of schedules. Schedules can
require a comparatively long time to execute. For example, executing
the standard protocols of the Web over the public Internet can often
require seconds or minutes to complete an operation. BizTalk
schedules can wait for the arrival of a message, but since these
messages are coming from other applications, even other organizations
across the Internet, there is the potential to wait minutes, hours,
or even days for a message. You might also be running hundreds of
schedules in a production environment. Clearly, the scheduling
services must ensure system resources are not consumed by schedules
that are not actively executing a scheduled action.
To do this, XLANG Scheduler uses a process called hydration. A
schedule is dehydrated when it is blocked waiting for an
incoming message. The current state of the entire schedule is
serialized to a database. When the Scheduler Engine detects the
arrival of the message, unblocking the schedule, the schedule is
rehydrated. Its serialized state is read and the schedule resumes
where it left off.
An application causes XLANG Scheduler to load an XLANG schedule
file by passing it a moniker. The moniker is a name that
represents a particular schedule file or schedule instance. A moniker
may be thought of roughly (very roughly) as being analogous to a URL.
It is a sometimes arcane-looking string that locates a resource in
the COM+ system. Here is a sample of a schedule moniker used by XLANG
Scheduler:
sked://ibex!XLANG
Scheduler/{770C13A4-AB4C-4470-821F-D1FAE5235AE0}/WaitForReply
XLANG Scheduler passes this off to the Scheduler Engine, which
then produces an instance of the schedule. Since a schedule can be
instantiated many times, on different servers, the moniker syntax not
only identifies the schedule document but also the location and
instance of the running schedule. This allows programmers to access
any particular instance of the schedule for custom purposes.
Defining Message Specifications: BizTalk Editor
The process of providing the low-level details needed to implement
a workflow starts at the bottom. Remember, the key to application
integration using BizTalk is the exchange of structured messages.
Whether those messages use XML or some other parsed text scheme,
programmers must be able to specify the structure of a message file -
the specification. This is the function of BizTalk
Editor.
The Editor is a graphical tool that uses a tree-structure metaphor
to build message specification files, or simply the specifications
themselves. Editor is influenced by database terminology, which is a
reasonable approach given that much of the information used to
compose messages will come from databases anyway. Database
programmers work with records and fields. A record
corresponds to an object or entity, while fields are equivalent to
the properties of the object. This works nicely with a database-style
table structure in which the rows are individual objects, and the
objects are described by the values contained in the row's
fields.
Unlike a relational table, however, a specification can nest
records within other records. This allows us to describe parent-child
relationships like those represented by a database join. The tree
that represents a specification in Editor is also analogous to a file
directory structure, with records standing in for directories and
fields replacing files.
In the Editor window shown below, the name of the specification is
SiteServiceDescription. The pane on the left shows the tree structure
of the specification, while the pane on the right displays some
properties of SiteServiceDescription.
When writing an XML-based specification, a programmer may describe
the records and fields in terms of the basic XML data types, as well
as the derived types permitted under the XML - Data Reduced (XML-DR)
Schemas technology implemented in the XML parser supplied with
Internet Explorer 5.0. All the options available in a schema are
available to specification builders. This includes being able to
specify the cardinality (repetition) of specific records and
fields.
In Chapter 3, we will use Editor to create three message
specifications for our sample application. One will be XML. The other
two, simulating legacy formats, will be non-XML in nature. These
specifications will be used by BizTalk to tell the Server how to read
our messages.
XML appears to be replacing other text formats for a variety of
reasons. Editor permits programmers to take full advantage of XML
without forcing them to become XML experts. Because Editor allows the
use of formats other than XML, the interface is carefully constructed
so that XML terminology does not dominate.
A complete discussion of XML, including its advantages with
respect to other text file formats, is found in Professional XML
(Wrox Press, 2000, ISBN 1-861003-11-0).
Right out of the box, BizTalk Server and Editor can handle the
following text formats:
q
XML
q
Delimited flatfiles
q
Positional flatfiles
q
Hybrid delimited and positional flatfiles
q
EDI (X12 and UN/EDIFACT)
With some custom coding, BizTalk can be made to understand your
own custom formats. This may potentially include binary formats.
Using XML does not force programmers to start from scratch when
defining specifications. For example, there are initiatives underway
to transform EDI commerce message formats and Health Level 7 medical
message formats to XML. BizTalk Server includes a collection of basic
XML template specifications for common business messages such as
purchase orders and invoices. ADO, the mainstream Microsoft database
access technology, now allows database recordsets to be persisted as
XML. Programmers can use this feature to model their specifications
on existing database table schemas.
Many legacy message formats are in forms other than XML. Both EDI
formats, X12 and EDIFACT, use delimited message formats. Versions of
Health Level 7 prior to version three also used non-XML formats.
Messages built around mainframe systems are frequently written in
character delimited or fixed position formats. Editor supports the
use of these formats; indeed, BizTalk includes specifications for
both X12 and EDIFACT. A programmer can load these specifications and
modify them as needed. When working with EDI formats, the Code List
tab in the right-hand pane of the Editor window contains a list of
all possible fields and records in the type of EDI message that is
loaded.More importantly, Editor permits programmers to specify all
aspects of delimited and positional flatfile formats. Programmers may
specify how to handle end-of-record delimiters in such files.
Delimiting characters may be specified. Field sizes may be specified
for positional files. This information is retained and used by
BizTalk server to properly parse an incoming message.
The primary file format of a specification produced by Editor is
an XML document that includes an XML-DR schema, but also has elements
added by the Editor to record specification-specific information.
These extra elements are especially important when working with
non-XML specifications. For example, XML-DR schemas cannot capture
delimiter and field position information. It is the extra elements
that capture this information, and they are proprietary to
BizTalk.
The BizTalk Framework schema repository works with XML-DR schemas,
not BizTalk specifications. For this reason, Editor can export XML-DR
schemas. A programmer can design a specification in the Editor, then
export a schema and submit it to a repository. Going in the other
direction, Editor can import an instance of a well-formed XML message
and build a specification and schema from it.
The capabilities of BizTalk Editor, along with the specific syntax
of BizTalk Server message specifications, are described in depth in
Chapter 3. Microsoft has pledged to migrate to W3C standard XML
schemas when they are approved as a formal W3C Recommendation.
Schemas have now reached Candidate Recommendation status, one step
away from formal approval.
WebDAV
Web Distributed Authoring and Versioning (WebDAV) is a
standard of the Internet Engineering Task Force (IETF) for
collaborative document editing over the Web. It permits someone to
retrieve and work with a document, and then submit it to a repository
using extensions to HTTP. BizTalk Editor and Mapper use WebDAV to
allow administrators to create, edit, and save document
specifications and maps.Why should we want to store documents in the
WebDAV repository? Well, if you are working with BizTalk tools
running on the BizTalk Server machine, you could use normal file
access to open, save, and close documents. However, to do so from
another computer, you would need security privileges to the folders
where BizTalk keeps its documents. Obviously, you would not want to
do implement this scheme - it would permit almost anyone to
manipulate or delete files without going through Messaging
Manager.
WebDAV, by contrast, exposes your documents and folders through
HTTP. If you are working with document specifications through WebDAV,
the BizTalk Editor will only show you folders devoted to document
specifications. These are actually folders on the server, but this
information is not exposed and you cannot navigate away from the
specification repository. There is also a XLANG schedule file
repository.
By using WebDAV, BizTalk lets administrators edit documents from
any computer with web access to the BizTalk Server. Users do not need
to be aware of the folder structure of the repository to find the
documents they need. WebDAV also locks documents to prevent users
from accidentally overwriting each other's changes. As well as this,
it enables users to share and work with server-based documents,
regardless of their authoring tools, platforms, or the type of web
servers on which the files are stored.
Translating Between Message Specifications:
BizTalk Mapper
Defining how one message format translates - or maps - to
another, is a critical task in application integration, as we have
seen. The Mapper tool is supplied with BizTalk Server to help
programmers perform this task. It is a graphical editor that displays
two specifications, the source and the destination, and
then permits a programmer to specify how the records and fields map
to one another. Mapper allows not only one-to-one mapping
relationships, but also one-to-many, or many-to-one relationships. It
even facilitates data processing during mapping - for example, you
might wanted to multiply a source field value by a number before
mapping it to the destination field. As we will see, some of the
processing that can occur during a mapping is quite complicated,
potentially including iteration through similar fields, or database
access.
The Mapper interface, an example of which is shown below, consists
of two regions.
At the top we have a mapping diagram showing three panes: the left
and right panes contain tree representations of the source and
destination message specifications respectively, while the middle
pane is a mapping grid, which displays lines linking source fields to
destination fields. Simple relationships between fields are mapped by
dragging a record or field in one specification to the appropriate
record or field in the other, thereby establishing a link. In such
cases, BizTalk Server performs the translation by simply copying the
content of a record or field in the incoming message into the mapped
record or field in the other message specification.
Sometimes, though, there is no simple translation. The contents of
several source fields may need to be combined to form the contents of
a destination field, or some data processing may need to be performed
on the contents of the source specification field to produce the
required contents of the destination field. In Mapper, there are two
ways to introduce intermediate data processing during the mapping
process:
q
Functoids: functional objects that perform simple predefined
operations (for example, string manipulation, or mathematical
operations)
q
Scripts: Short user-written scripts, executed by the script
functoid, which allow for more complex data processing.
If you look again at the screenshot of the Mapper, you will see
that the grid contains icons, like the box containing A. This is how
functoids and scriptlets are represented in the mapping grid.
When the desired mapping is complete, a programmer compiles the
map using the Mapper. This results in an XML transformation
(XSLT) stylesheet. Then, when mapping between message
specifications is required, the runtime server process can apply the
appropriate stylesheet to perform the appropriate message mapping.
This process, for mapping between different XML specifications, is
depicted in the diagram below:
You should note that the transformation process for mappings
involving flatfile formats has several additional steps.
The region at the bottom of the Mapper interface contains useful
information, such as lists of attributes of the fields/records
selected in each specification, and any compilation errors.
We will discuss the Mapper in greater detail in Chapter 4.
In the sample Site Managers application that we will develop
throughout the book, we will need two maps. Each will take the
message format describing the site and translate a subset of its
information into one of the other message formats. That way, we use
our own message format, and use BizTalk mapping to send out messages
in the formats used by our business partners. One of the maps will
use a functoid that we will create specially for this purpose. We
will examine this more closely in Chapter 4.
Configuring Messaging Services: BizTalk
Messaging Manager
Messaging Manager is an application that allows you to configure
and manage how messages are passed through BizTalk Messaging
Services. It can be used to specify the organizations and
applications that are the components of your workflow. From there,
you define how they are integrated by specifying the format of
messages that pass between organizations, and what should happen when
a message arrives at the server from a particular source, bound for a
particular destination. In short, Messaging Manager lets you manage
the business processing relationships that define message
exchange.
Messaging Manager consists of two main panes: a navigation pane on
the left that outlines the different items under management, and a
detail pane to the right in which search results are presented.
Editing an item in the detail pane invokes a wizard, that guides you
through the steps required to fully configure that item. A sample
Messaging Manager window is shown below, which details the channels
under management:
BizTalk message transfer is completely defined by the major
entities managed through Messaging Manager. These areas are:
q
Ports
q
Distribution Lists
q
Channels
q
Organizations
q
Document Definitions
q
Envelopes
You should be familiar with these terms from our earlier
discussion of BizTalk messaging. Application integration with BizTalk
involves configuring message documents, organizations, ports, and
channels. Only a thin layer of code is required, in most cases, to
make an application BizTalk-aware. In some cases, no custom code is
written at all. The Messaging Manager is therefore fundamental to
using BizTalk. You cannot use any runtime aspect of BizTalk Server
(except, perhaps, an extremely limited example of orchestration)
without configuring some entities in the Messaging Manager.We will
cover Messaging Manager and messaging configuration in depth in
Chapter 5, although we will also be seeing different aspects of using
Messaging Manager throughout this book as we configure our sample
applications.
When we cover the Messaging Manager in detail, we'll put it to
practical use by configuring the details of the messaging that must
take place in the orchestration schedule we build for our sample
application. In other words, in Chapter 2 we will produce a sample
schedule - a high-level view of the process. Then the low-level
messaging functions that make it work will be specified through
Messaging Manager in Chapter 5, to complete the sample
application.
Viewing Run-time Messages: Document Tracking
Once you have a BizTalk Server installation up and running, it
becomes important to be able to track individual messages transiting
the messaging system. Recall that Orchestration Designer and XLANG
Scheduler view systems in terms of the flow of messages. Viewing the
status of messages, therefore, is viewing the state of the system.
Without the ability to track messages, you cannot monitor and
troubleshoot the system as a whole. In the absence of message
tracking, you could only study individual applications and guess at
the problem; if you were exchanging messages with an external
partner, you couldn't even do that.
Thankfully, BizTalk Server does have the ability to track
messages. Every message interchange that passes through Messaging
Services is recorded by default in the Document Tracking and Activity
(DTA) database. When you configure messaging, you can change these
defaults. You can even record specific fields from particular classes
of messages, fields that play an important role in your system.
For example, one of our applications might assign a tracking
number to an order. This number might be used throughout the business
to monitor the status of the order. Since this datum has no meaning
to BizTalk, it would not be retained by default. Since it is of
critical importance to our business, however, we can direct the
tracking process to record it in the DTA database so that we can
readily report on messaging according to this number. Clearly, there
is a lot of message information available for you to mine.
While you could access the underlying SQL Server database
directly, it is convenient to have a default interface that
understands the layout of the DTA database. This is the function of
the Document Tracking application that comes with BizTalk.
Document Tracking is a web-based application. It makes extensive use
of ActiveX controls, however, so don't expect to use it in practice
to provide self-help tracking information to external partners. This
is, practically speaking, an internal troubleshooting tool.
The basic controls allow you to filter messages based on:
q
Date Range (the date and time the message transited Messaging
Services)
q
Message Source
q
Message Destination
q
Message Type
For example, say you send a particular type of message document to
several partners, but one of them requires a transformation. You
could single out documents bound for that partner by filtering based
on the destination organization. Actually, this is a common scenario.
If there is a fault in the message map, that partner would be the
only one to experience difficulty. Looking at the status of the
messages intended for that partner would help you track down the
problem.
You can further refine your criteria when dealing with messages
sent within the Home Organization, because you can look inside the
Organization and track the messages sent between applications.
As you can see from the screenshot below, the document definitions
known to BizTalk are displayed in the list box in the lower right
corner of the Document Tracking query window, which makes it
relatively simple to filter on message type. In addition to
filtering, you can sort your results based on the same criteria by
which you were able to filter. You may specify up to six levels of
sorting.
By default, BizTalk Server does not single out any fields in your
messages for retention. You can tell it to retain copies of all
original messages through the Server Administration MMC snap-in, or
you can use the Messaging Manager to be more selective in retaining
information. If you want to track messages based on specific fields,
you can tell BizTalk to capture the values of those fields in the
database and then write queries using expressions based on those
values.
In the Messaging Manager, you can select fields in either Document
Definitions properties or Channel properties. The former captures the
designated fields for every instance of that message type seen. The
latter captures the fields when a message of that type transits
BizTalk Server as part of the specific channel selected.
Once you have fields captured in the database, you can use the
Advanced Query group in Document Tracking to build expression-based
queries. Clicking New in that group leads you to a dialog that
permits you to select a captured field and build a logical expression
around it. This expression becomes part of the selection criteria for
your search.Regardless of your query parameters, the results of a
query are displayed in a new browser window that looks like this:
|
|
Note that you can expand any single document instance to view its
status information. Any fields you have asked BizTalk to save to the
DTA database will be displayed. Clicking the icon in the Data column
pops up a window showing the body of the message, assuming you
configured Messaging Services to log message bodies when you
configured the channel for this interchange. Clicking on the Schedule
icon displays any available information for the schedule (if any)
that executed the interchange. While basic, the Document Tracking
application is a good first stop for troubleshooting and tracking. It
enables someone unfamiliar with the DTA schema to query the database
and obtain reports on BizTalk messaging activity.
Document Tracking is discussed in detail in Chapter 12.
Managing BizTalk Server: Server Administration
Not surprisingly, all administration of BizTalk Server is
performed through the Microsoft Management Console (MMC). BizTalk
Server instances are organized into BizTalk Server Groups for
scalability. A Server Group is a collection of identically configured
BizTalk Servers that share a common view of administrative
configuration and document interchange. This means that any server in
the group can be used to process a particular interchange of
documents. They all share the same view of organizations, documents,
and agreements. This is accomplished by having all the Servers in a
group share several things:
q
the Shared Queue (SQ) SQL database
q
the Document Tracking and Activity (DTA) SQL database
q
receive functions for file or MSMQ message transport
q
parsers and serializers for message processing
These concepts, and how we manage them, are the topic of Chapter
9. A typical Server Administration window is shown below:
The SQ database is where incoming and outgoing messages are
stored as they pass through the Server. The database consists of the
following logical queues:
q
Work - messages currently in process
q
Scheduled - messages processed and awaiting delivery to a
distribution list
q
Retry - messages that have failed transmission and are queued
awaiting a retransmission attempt
q
Suspended - messages that could not be transmitted
The DTA database logs message interchange activity for the server
group. Administrators use it to run reports on activity within the
group using the Document Tracking application, as we have just
seen.
The BizTalk Server Administration MMC snap-in allows an
administrator to add new groups to the enterprise and configure them
in terms of their shared resources. Administrators can also use the
MMC to add a server to a group. Within a server group, the MMC
provides a view of messages in the various queues.
It also lists the receive functions configured for that group.
These are specific MSMQ queues or directory folders that Messaging
Services will monitor for incoming messages.
If you open the Properties dialog for the Server group, you can
also configure the tracking behavior and database location for
message activity on that group. Each group also shares a collection
of parsers that BizTalk uses to handle various message formats. Under
certain circumstances, an administrator may wish to change the order
in which BizTalk tries these components on an unknown incoming
message. This occurs mainly when programmers have created custom
parsers to extend BizTalk's features and the administrator wants to
ensure the custom parser is tried first.
Note that the WebDAV repository is treated differently from the
other resources we have discussed so far. It is shared among all
BizTalk Server groups in an enterprise. This is reasonable on several
counts:
q
Document specifications and maps are likely to be common to the
entire organization, not just one working group.
q
Access to the WebDAV repository is going to occur much less
frequently than message interchanges.
Clearly, you develop a specification and then use it repeatedly in
the course of normal processing. The WebDAV repository, then, has
much lower demands on scalability than the shared queues and DTA
databases.
The Server Administration MMC snap-in focuses on the maintenance
of the servers and the components they require. Maintenance of the
business logic embedded in the workflow is handled through the
Messaging Manager and Orchestration Designer. These tools are
complementary in terms of managing a BizTalk installation.Server
Administration and Configuration is taken up in Chapters 9 and
13.
Email TopXML