BizTalk and Application Integration, cont.

EAI and B2B Scenarios and Architectures

Enterprise Application Integration can occur in any type of project. Some classes of project involve integration by their very nature. But what are these projects and what kind of integration is involved?

In this section I'm going to briefly describe four types of projects, focusing particularly on those that you might encounter in enterprise-scale web applications. These project classes are:

q                      Business process integration

q                      Trading partner integration

q                      Automated procurement

q                      Business to Business (B2B) aggregation

All four of these classes can involve web technology, but not necessarily like the retail e-commerce sites with which we are all familiar. Business process integration, in particular, is almost exclusively conducted via an intranet; but it also happens that internet technologies - protocols and servers - are well suited to implementing this kind of integration as well. The reason for this stems largely from the fact that the Web is inherently loosely coupled. In other words, servers and clients have little (or no) prior knowledge of one another, so each has provisions for dealing with a failure on the part of the other. They all speak HTTP, but there is a great deal of flexibility in data formats (HTML and other MIME types). A tightly coupled system, by contrast, has explicit and closely-integrated data formats and communications. It is easier to add reliability to loosely coupled systems than to break apart tightly coupled systems to gain the flexibility and responsiveness we will need.

Business Process Integration

Application integration is important even when we stay inside the boundaries of a single company. Each functional or organizational unit begins by automating its core process: Human Relations might roll out a benefits application or personnel directory; the Sales organization tracks orders and leads; Inventory control and supply chain systems are introduced.

Before long, islands of automation have appeared throughout the company. Each contains some valuable store of data focused on some process of compelling interest to the company. Quite simply, the company could not function competitively without these systems.

This has been the pattern throughout companies for some years. It does have the advantage of permitting a company to focus its efforts on getting one important function right, instead of trying to automate every process at once. But a company shouldn't be a collection of islands; it should be an integrated and interrelated set of people and processes, which work together to achieve a goal.

Business Process Integration is the task of connecting these isolated applications in such a way that it mirrors the idealized workflow of a company. It is typically used to gain some competitive advantage by cutting costs and the time needed to move an order from placement to completion (the cycle time).

At first sight, you might have thought that integrating processes within a business should be a great deal easier than integrating the systems of two distinct companies. After all, it should be much easier for integration teams within the same company to work together, to share source code, and reach consensus. In practice, however, we actually encounter the same problems of dissimilar data formats and technologies when integrating within companies as we do when integrating applications owned by different businesses. Why should this be so? There are several reasons:

1.         Different approaches to the same data from different organizations

2.         Different data processing needs for different organizations

3.         Documentation can also be lost, or even never created, so programmers cannot count on crossing boundaries to fix applications

Because each organization within the company has its own goals and focus, this focus is reflected in the software and hardware it selects. This means that data tends to be specialized to the tasks demanded of the organization, even when that information is broadly applicable throughout the company.

For instance, any effective company is interested in tracking its customer from pre-sales to post-sales customer support. But the view of the customer held by the Sales Force is often quite different from that held by the Customer Support or Manufacturing divisions, so specialized information of interest only to Sales gets mixed together with general information about the customer, which is also of interest to Customer Support and Manufacturing.

The way data is processed may differ as well. For example, a Sales Force automation system will typically perform bulk synchronization once a day, when individual salesmen connect from their remote locations. Service representatives in a customer support call center, by contrast, need to be able to update customer records as they receive information from individual customers. Programmers assigned to integrate dissimilar systems will therefore also have to overcome these differences.

Strangely enough, these integration problems may also be beneficial to the development process, as they force programmers to deal with interfaces in a rigorous way. The discipline of designing, documenting, and implementing interfaces leads to more robust systems. Assumptions are tested with running code instead of paper models and anecdotes.

Business process integration sometimes has a reputation for involving a lot of legacy systems. Although probably true for established companies that adopted automation early on, modern applications are increasingly built using web-based intranet applications: these are easier to access throughout the company, and require less coordination and support to distribute. Every employee who possesses a web browser is a potential user of the application, while the details of technology and hardware platforms are hidden by the intranet.

We can expect business process integration to increasingly become a matter of harnessing internal web applications, which is good news for programmers seeking new technical challenges. Their companies benefit, as well: although web applications do not eliminate the problems of application integration, they do diminish the cost of solving them. Web applications are inherently distributed across a network and loosely coupled, meaning that integration programming teams can work independently, crafting and exploiting interfaces between business organizations.

Trading Partner Integration

It is becoming increasingly common in business for two or more companies to share information more closely, and at a higher level of trust than they would with other firms. These companies are known as trading partners because they have established a mutually trustworthy reputation by regularly doing business with one another. As a result, they share more information, require fewer formal contracts, and link their internal systems more closely than usual, for mutual benefit. The process of connecting the two electronically is called trading partner integration.

Trading partners may have regular agreements in place for recurring orders, or may simply place orders as needed. The benefits from a trading partners scheme are low overheads and rapid response, due to the close integration of the partners' systems. The tighter the integration, the lower the cost of doing business.

Trading partners must be able to express their needs in a recognizable format, and their partners must be able to respond appropriately. There must also be accountability at each step. The identity of the trading partners must be securely verified, with agreements in place so that each partner can be confident that they will get paid. One effort to put this discovery process on a formal basis is the proposed UDDI standard. UDDI, which stands for Universal Description, Discovery, and Integration of Business for the Web, is a coordinated set of XML data structures, SOAP messages, and APIs for describing and discovering goods and services offered for commerce over the Web.

UDDI's formal structure is described in documents hosted at www.uddi.org. You may also register there as a potential trading partner and describe your offerings and interests.

Electronic Data Interchange (EDI)

EDI is an existing message standard used by trading partners. It comes in two forms, both of which specify long lists of message types designed to facilitate business-to-business e-commerce:

q                      United Nations Electronic Data Interchange for Administration, Commerce, and Transport (UN/EDIFACT)

q                      ANSI X12 (used largely in North America)

However, two problems have largely hampered widespread adoption of these standards:q                     EDI has classically involved the use of proprietary wide-area networks.

q                      The formats used to exchange data were proprietary and difficult to read.Both of these factors increased the cost of implementing an EDI system.

In recent years however, an effort to implement EDI on the public Internet, using XML as the basis for developing message formats has sprung up. By using common network transport, enabled by open protocols for security and reliability, the hardware cost of EDI is greatly diminished. The use of XML decreases the cost of software because XML tools are readily available from third parties, and the self-describing nature of XML makes it easier to develop and debug software for specific EDI message vocabularies.

While we can expect to see more and more XML implementations of EDI, particularly in smaller businesses, we must still expect to have to integrate X12, EDIFACT, XML EDI, and other proprietary formats when dealing with trading partner integration. We will therefore still need to map data from one format to another.

Thus, while trading partner integration offers exciting opportunities for programmers to reshape entire businesses, it still means that many of the classic problems of application integration must be overcome. In fact, the challenge is greater than for business process integration, because in this case the two parties exchanging information are distinct legal and organizational entities (and they may be geographically distant as well). Integration must occur at the interfaces between the partners (we cannot rely on being able to get inside the source code for the partner's respective systems).

Automated Procurement

This is a variation on the trading partner integration scenario. Here, the relationship is formalized and so close that the partners involved are sometimes said to form a virtual company. A company may entirely outsource some critical function to a procurement partner because it is cheaper or more reliable to do so.

Obviously, integration like this has to be close and effective. A partner's systems must seem like an extension of one's own systems; this often involves integrating some internal systems so that information may flow directly from one partner's system to another.

The problems of trading partner integration exist here in magnified form. We are no longer talking about one partner among several, but about a partner that may be a company's sole source for a critical good or service. The benefits, however, are vast: a company that can quickly satisfy a prospective partner's concerns about the reliability and speed with which it can integrate systems can lock in a substantial volume of business on a recurring basis. The cost of obtaining follow-on business then drops markedly.

Systems integrators are crucial to the success of automated procurement. An effective set of methods and technology for implementing integration is a critical differentiating factor for would-be partners. Poor integration, by contrast, can sink one or more businesses.

Just-In-Time Ordering

Consider the following process, which is a classic example of automated procurement known as just-in-time ordering.

A large manufacturing company negotiates special terms with a limited number of suppliers. When supplies fall below a predetermined level, the inventory system at the manufacturing company automatically places an order with the appropriate supplier. The order goes directly into the scheduling system of the supplier, thereby saving time and money for the supplier.

Because the process is completely automated, the supplier can offer lower rates as well as a guaranteed, prompt delivery time. The manufacturing firm also benefits, because it can count on having supplies arrive as they are needed, avoiding the need to carry large inventories and therefore lowering costs.

The entire process relies on several things. First, both partners have to have a comprehensive supply chain system: they need to track their inventories and orders automatically and in real time. Second, the systems must be closely integrated. The benefits of trading partner integration will be lost if manual intervention is needed to convert an order from the manufacturer's system into an order in the supplier's system. This means that the two partners must be connected via some network.

Finally, issues of security and workflow must be agreed upon ahead of time. Both partners must know who can approve orders and how orders are to flow between the partners. These issues will be settled offline prior to implementing the partnership, but the details must be captured online so that the integrated system can respond appropriately at every step.

Business to Business Aggregation

Until recently, integration between distinct businesses, in terms of automated procurement or trading partner integration, has largely been a matter of large, established firms engaging in long-duration integration projects. This has limited the benefits of business-to-business electronic commerce to large, wealthy companies. However, as application integration becomes more routine, and particularly as Internet technologies come to the fore, smaller companies should increasingly be able to take advantage of electronic commerce.

The problem of finding a suitable supplier or partner has long been a big problem for smaller companies, since they do not have wide experience in finding and dealing with vendors outside their immediate area. This is a business problem that cries out for a technical solution. The result is the emergence of business-to-business portals, or B2B aggregators.

A B2B aggregator acts as a trusted intermediary, helping partners to:

q                      find one another

q                      establish trading agreements

q                      integrate their trading processes

These are challenges that the previously mentioned UDDI proposal seeks to address. Regardless of how you go about it, however, you must establish some common mechanism for trading. By establishing a high profile in the business community, aggregators draw potential partners to a common ground. These partners are then able to verify the credentials of the aggregator and examine their processes, before deciding whether the aggregator can be trusted to certify other vendors. Aggregators frequently specialize in a particular vertical market, further helping to focus their trading community.

If aggregators simply provided a partner-finding service, leaving agreements and integration details to be worked out offline, they would be of little real value to businesses. Instead, they provide common mechanisms for publishing product and service catalogs. They also provide standard mechanisms that allow partners to establish agreements and exchange messages that implement some commercial workflow between the two partners. The relationship between business partners and an aggregator is illustrated opposite above.

 

By taking on some of the infrastructure of trading partner integration or automated procurement, the aggregator facilitates electronic commerce for small- and medium-sized businesses. In return, an aggregator charges some fee for its service, typically a per-transaction charge. An aggregator web site typically takes the form of a portal, providing information and services to a vertical market. Its service to market participants may be as simple as offering a library of message formats, or as complex as hosting e-commerce fulfillment and settlement software.

Aggregators have an interest in open (public) protocols and internet technologies, and push for standard protocols and Web-enabled commerce systems. This is because the businesses most likely to seek their services cannot afford custom solutions and networks. These businesses are likely to be equipped with third party, packaged software operating over the public Internet; they resort to custom arrangements only when they cannot avoid it - usually when it involves some unique facet of their business.

Before we get into the details of BizTalk Server, let's take a look at the BizTalk Framework initiative. The Framework is one effort to address some of the issues we have just seen. Looking at this will give us a better appreciation of what must take place to do B2B integration and prepare us for the specifics of BizTalk Server.