Washington, September 15-18, 1999 – London, November 21-24, 1999

Creating a backend framework using Site Server 3.0

Marco Tabini

Electronic commerce systems can be very complex, particularly if you consider that they often involve much more than simply letting a user choose one or more products from a catalogue and pay for them.

For the most part, the front-end of an online store has the primary goal of helping and convincing a customer to purchase a product or a service, and therefore deals essentially with a marketing effort at first and then a purchase process once the customer has made up his or her mind. This certainly represents a significant amount of work, particularly considering that competition among online stores has been heating up considerably in the past few years, and therefore the request for “smarter” sites has grown steadily.

A well thought out e-commerce system, however, has a lot to do in the backend as well. Operations such as determining the availability of a product and shipping an order take place outside the realm of the front-end, and it takes a lot of work to make them work properly, particularly considering that you will have to deal with several partners, who will all have different capabilities and support different standards or platforms.

The goal of this presentation is to illustrate how Site Server 3.0, Commerce Edition (SSCE) can be used to design and develop a complete backend system. We’ll go through the basics of backend integration and then examine how SSCE can help us in this area.

As we’ll see, a lot of thought has gone into designing the business-to-business portion of SSCE, and this is no accident: many business analysts predict that business-to-business commerce will be the next big thing in online commercial applications, eventually outgrowing the business-to-retail market by several times.

Marco Tabini Marco Tabini is an electronic commerce specialist based in Toronto, Canada. He provides custom solution for clients who desire to bring their business online, and writes about electronic commerce and Site Server on various publications. He is co-author of "Professional Site Server 3" (Wrox Press, 1999).

Essential elements of an e-commerce system

Let’s start by taking a look at what some of the basic components of an e-commerce system. As you can see from the picture to the right of this paragraph, the storefront is really just the initial entry point into a larger and more complex entity that takes care of the entire online shopping process. The goods that the user requests during a purchase through the storefront are handled by the order processing logic, which verifies, among other things, whether they are available and, through the payment system, whether the user has enough funds to pay for them. It’s interesting to notice that there might be more than one payment system available—credit cards and checks, for example—although the order processing system will generally only use one of them at a time.

The goal of the line-of-business system is essentially that of communicating with the store’s suppliers for several reasons, such as determining whether a particular product is available, or to request a shipment for one or more products, and so on. Developing a line-of-business system is usually perceived as a difficult and expensive task, to the point where many online stores do not have a real LOB system, but perform their operations manually.

The real problem with LOB systems is that they are usually difficult to handle because it usually takes a different set of rules to communicate with each different supplier, making the development of the entire system expensive. On the other hand in the long run, for larger stores, the actual development costs might be lower than the cost of manual operation. If you are familiar with Site Server, Commerce Edition, you know that it provides a great tool for line-of-business communications through the Commerce Interchange Pipeline (CIP). The CIP will be at the center of this presentation, and we will look at it not only from the point of view of the store, but especially from the point of view of the supplier.

What is not very obvious with LOB systems—particularly if you are only familiar with the development of online stores—is that they are also often difficult and costly to implement at the supplier level. Up to this point, when the Internet couldn’t yet be considered a viable transmission medium for the kind of information that a LOB system deals with. Suppliers had to cope with having to lease private lines, encrypting their information and formatting data according to complex standards. The good news is, Site Server can be used to perform all these operations in a much simpler and more economical way, and the resulting LOB system can be compatible with the existing standards if necessary.

Designing an e-commerce infrastructure

The first step in the creation of an e-commerce infrastructure consists in determining what services it will provide. In most cases, a good LOB system will offer the following capabilities:

Ø        Catalog management

This feature enables the store to determine whether new products have become available or old ones are not part of the line offered by the supplier. Real-time access to this item is not usually necessary, as most stores will update their catalogs on a fixed schedule.

Ø        Inventory control

This operation is used to verify the inventory status of particular items, and particularly their availability or restocking time. Depending on the nature of the products sold, inventory control may or may not be necessary (for example, if the store deals in digital goods, they will not normally go out of stock!)

Ø        Goods shipment

A request for a shipment of a product to a customer or to a warehouse facility can be part of the LOB system as well. Once again, this will depend on what kind of goods are being dealt with by the store.

Different industries may warrant the addition of further operations to the LOB system—for example, in the automobile industry it might be necessary to make particular requests to determine whether a used car was reported stolen or was involved in accidents, and so on. In general, however, these are the three basic operations that a LOB system will have to perform.

Identifying your goals

When you have to work with a LOB system, it’s important that you identify your problems and objectives in four different areas from the very beginning:

Ø        Nature of the data

It’s important that you know well in advance what kind of data you will be transmitting. This is useful to determine the format and medium to be used for the transmission, because it may not always be possible or practical to send certain data using them.

Ø        Format

The format of the data is very important when dealing with LOB system, because both ends of a communication must be able to interpret the incoming data properly. In addition, you should make sure that whatever format is chosen can be sent across the medium you want to use. For example, if you chosen binary encoding, it might be impossible to send your data through a public switched network that only allows you to send seven-bit words.

Ø        Transmission medium

The transmission medium that you will use can bear a high importance on many of the choices you will make. As we have just seen, it could determine what format you may or may not use, and also force you to use encryption if it is not an inherently secure channel.

Ø        Security measures

Finally, you will have to consider what level of security you will have to embed in your transmissions. Encryption is often useful, but not always necessary: public switched networks (like normal phone lines) are usually inherently secure channels and therefore do not normally require encrypted transmissions. On the other hand, public networks such as the Internet will usually require that you encrypt your data in a fairly secure manner. As we’ll see later on, your decision to encrypt data will usually affect the entire range of information that you will be sending out, regardless of what its nature is.

Each of these areas depends at least in some measure on the others. For example, if you need to exchange large amounts of data, it will be counterproductive to use a slow VAN network like Datapac, which is slow (usually up to 9,600 bps) and expensive, since you will normally pay for both usage time and amount of data transferred. A normal phone line, where you can send up to 33,600 bits per second, might fare better in this case, as could a faster leased line or the Internet.

Similarly, you will want to gauge the advantage of implementing security measures against their cost in term of infrastructure and efficiency. Since encrypting data is usually a computationally intensive operation, on a large scale it can cause a significant performance impact on your system. Even when it is required because your transmission medium is insecure, it might be useful to separate the really sensitive data, which requires full-blown encryption, from the remainder, which might only necessitate a digital signature to ensure its integrity.

Make sure you can handle your traffic

In a dynamic environment such as electronic commerce, your system must be easily—and hopefully painlessly—scaleable to accommodate changes in the traffic it must support. Early planning in this area is absolutely essential, as it will save you a lot of headaches later on when you will inevitably need to expand your capabilities in terms of traffic or performance.

It’s a good idea to start by identifying where you possible bottlenecks and limitations are. Bottlenecks in your architecture will usually be the transmission media used, as well as the internal system used by your different computers to communicate between each other. A typical example here is trying to use a slower line to serve too many customers, or an internal communication system whose efficiency decreases exponentially—rather than linearly—in proportion to an increase in traffic, such as Ethernet.

Limitations will often depend on the technology that you use and how implement it. For example, if you database system (DBMS) cannot be clustered, you will have to first identify what the limits in terms of usage and storage capacity of your system is, and then provide for an alternative system of expanding it. Similarly, if you are using a web system for the transmission of data, you must be sure that additional servers can be added to your farm easily.

Having a well-thought out contingency plan is a key factor in ensuring your peace of mind for what concerns the scalability of your system. Knowing what you need to do should a surge in traffic happen (and chances are that it will be unexpected!) will not only reduce your response time, but also eliminate a lot of headaches when the moment comes. There are several tools available today to measure the performance of a web-based system such as SSCE, and therefore creating a baseline and determining your capacity will be a relatively simple task.

A good backup strategy is also required for a LOB system to be truly complete. The term “backup” here means both providing an alternative storage location for your data and one or more substitutes for your system should it fail for one of many reasons. The frequency with which you back up your data is critical, and should be decided based on a combination of factors, such as the sensitivity of your data and the traffic that your system will have to withstand. On average, in case of a complete “meltdown” of your storage devices, you will lose an amount of data corresponding to whatever accumulates in half of your backup cycle. So, if you backup your data daily, you will on average lose the data that usually accumulates or changes in 12 hours; the ideal backup frequency represents a good tradeoff between the cost of backups and the important and quantity of data accumulated in the semi-cycle.

Develop the basic architecture first

A business-to-business system is essentially a closed system, meaning that there are only a limited number of operations that must be done and that you will always know what your partner in the system is and what it can or cannot do. This is different, for example, from having to deal with a human being, whose behavior is inherently unpredictable, in the storefront.

Simply put, this basically means that the communication portion of an LOB system is a lot simpler to develop than the storefront or the backend system, as long as you have the right tools.

Understanding B2B communications

At the core of business-to-business communications is the concept of interchange. An interchange is a series of one or more exchanges of information between two partners. Each individual communication in the interchange is essentially an agglomerate of data that has to be formatted using a particular format that both partners understand. It’s interesting to notice that, for a particular communication, neither partner needs to understand the format completely: in fact, the sender only needs to be able to “write” it, while the recipient needs to be able to “read” it.

There are a few simple considerations that can be made about interchanges:

Ø        There are always two partners to an interchange

An interchange always involves no less and no more than two partners. This is a simple consequence of the fact that the recipient usually requests very specific information to a partner.

Ø        Both partners need to have access to each other

In order for the two partners to “get in touch” with each other, they must be able to reach each other through a common transmission system, as well as understand and support the protocols and formats used to exchange information.

Ø        Interchanges are closed loops

An interchange is not complete until both the recipient has acknowledged the delivery of the initial message to the sender using a receipt. If the sender doesn’t receive the receipt within a pre-determine timeframe, it should automatically resend the initial message and assume that it was never delivered in the first place.

Ø        Failsafe mechanisms must be in place

This also means that the system as a whole must be able to adapt well to network error conditions. For example, the recipient must be able to determine when the sender resends a message because a receipt was never delivered to it.

A few words about EDI

EDI, which stands for Electronic Data Interchange, is the worldwide standard for business-to-business interchanges. It defines a particular format in which messages should be formatted and exchanged between two business partners.

There are two major versions of EDI, one used primarily in North American and the other in the rest of the world. In addition, the EDI standard defines a large number of sub-formats that are specific to particular industries, such as automotive, mechanical, and so on. As a result, EDI is very complex and somewhat difficult to implement, requiring a significant amount of resources to set up a system that runs smoothly.

In general, if you are developing your own LOB system, you will only need to use EDI if your partners require you to. Many of the corporations that have implemented EDI will not like moving to another custom format for the simple reason that it will make it difficult for them to justify the investment that was made in supporting the standard. Keep in mind that this doesn’t necessarily indicate bad planning on their part: EDI was around long before the Internet and SSCE, and it was the only comprehensive solution available until a few years ago.

Designing and implementing interchanges

If you’re familiar with SSCE, you know that the pipeline technology is a great tool that can be used to develop a business interchange system. Its main advantage is that it is completely self contained—meaning that it includes all the elements that are needed to develop a complete LOB interchange system—and that it is not tied to a particular set of protocols or formats: even if you can’t use the built-in elements, you can always develop new pipeline components that perform whatever task you need—with the advantage that they will easily integrate with the rest of the system.

If you’re using SSCE, you don’t necessarily have to use EDI. As long as your partners are able to understand your data (and they won’t need SSCE to do so), you can choose whatever format you like; the built-in CIP components that ship with SSCE provide you with a wide range of possibility for encoding, encryption, integrity checking and transmission. Still, if you strictly need to use EDI, you will be able to either build your own components or purchase third-party tools that will enable you to read and write interchanges in that format.

Structure of the CIP

The CIP is composed of two pipelines. The transmit pipeline is used to create an interchange and send it across a communication channel, while the receive pipeline is invoked when the interchange is delivered to its destination and it must be decoded and interpreted.

The transmit pipeline As you can see from the image to the side of this paragraph, the transmit pipeline is composed of six stages. As usual, each stage is dedicated to an individual moment in the creation of an interchange transmission, and the stages are arranged so that each action is taken in the correct order. The Map stage is used to create the core of the interchange—an agglomerate of data formatted according to a specific set of rules. The components stored in this stage will usually convert the contents of an entire Dictionary object, which is part of the pipeline’s own dictionary, into a single value that contains the entire interchange. Built-in components in the Map stage allow for the encapsulation of the interchange using XML, which his undoubtedly a practical and very portable format. It’s also possible, however, to create an interchange in comma- or tab-delimited format by borrowing a few components from the Order Processing Pipeline, such as CreatePO. The Add Header stage is used to add a header to the raw interchange data. The header can be used for a variety of purposes, such as indicating what the nature of the interchange is, what data it contains, who sent it, and so on. Built-in components here will allow you to either add a custom header or MIME encapsulation to the interchange.

The header is indeed a very important part of the interchange and it should be underestimate. One of its functions will also be that of allowing you to determine, on the receiving end, whether an interchange pipeline is a duplicate of a previously sent pipeline.

The Digitally Sign stage is used to add to the interchange the necessary information to ensure the integrity of data. A digital signature, as opposed to encryption, will not protect your data from snoopers, but will let the receiving end determine whether the interchange transmission was modified since when it was generated. This is particularly important in a public network such as the Internet, in which somebody along the route your data travels could change it with potentially disastrous results. It’s important to notice that the interchange is essentially complete when it reaches this stage.

The built-in components that show affinity to the Digitally Sign stage use the Microsoft CryptoAPI to digitally sign an interchange transmission. If you use well-known standards and normal digital certificates, however, you will not necessarily need to have SSCE, or even the CryptoAPI, running at the other end in order to receive the interchange.

The Audit stage takes on the task of creating a permanent record of the interchange transmission. This is useful nor only for statistical purposes, but also to determine whether an interchange was successfully transmitted: if you use the built-in components to create an audit trail on a database, you can later on search the database for items that have not been confirmed using a receipt and resend them.

Finally, the Transport stage is used to actually send the interchange transmission using the appropriate communication channel. The built-in components here support several well-known Internet protocols, such as HTTP and DCOM, which are synchronous, and SMTP, which is essentially asynchronous (as long as the mail server is promptly available). The latter requires the mail server at the receiving end to be Microsoft Exchange.

The receive pipeline The structure of the receive pipeline, once again shown to the side of this paragraph, includes almost the same stages as the transmit pipeline, although in a different order, since the goal here is to “deconstruct” an interchange and end up with the original information. This pipeline is usually invoked as a result of the reception of an interchange transmission, for example in an ASP script that is called by an HTTP post. The Decrypt stage is obviously used to convert encrypted information in plain form. Although the built-in components support the Microsoft CryptoAPI, the same consideration that we made while talking about the transmit pipeline is also valid here: as long as the protocols used are standard, you will need neither SSCE nor the CryptoAPI at the sender’s end. However, you will need the same digital certificate used for the encryption to be installed at both ends, or the decryption process won’t work! The Verify Digital Signature can be used to determine whether the interchange was modified since the moment in which it was generated. It’s important to understand that the term “digital signature” can be used as loosely as you want: strictly speaking, it could even mean a simple CRC check. If you do decide to use one of the built-in components, however, you will once again make sure that whatever certificate you have used is also installed on the receiving end. Next, the Open Header stage will open and interpret the header with which the interchange was encapsulated. This is where you would determine whether the interchange transmission is a duplicate and should be ignored. If that is the case, you should also send another receipt back to the sender.

Speaking of the devil, the Generate Receipt stage is used exactly for that purpose. The built-in components here are able to generate a receipt automatically that will however be understood only by another implementation of SSCE.

The Map stage’s goal is actually to “un-map” the interchange and generate a dictionary that contains all the original values. If you decide to use the built-in components here to decode XML transmissions, you should be aware of the fact that you will not be able to transmit entire objects as part of the interchange.

The Audit stage performs the same function as its counterpart in the transmit pipeline, and is again used to determine whether an interchange is a duplicate or not.

Finally, the Application Integration is where the actual action is taken on the interchange. Depending on the nature of the transmission, you may be required to send another interchange back (you can do so by invoking another transmit pipeline from within your receive pipeline), invoke an external application, or just write the interchange’s contents to a storage device. Since the contents of this stage will be highly proprietary, the pipeline’s extensibility comes in really handy here.

Sequence of operations

For a final touch, let’s take a look at what is supposed to happen in a complete interchange. It’s usually important that you follow a close loop similar to the one that we are going to examine here if you want your interchanges to work properly.

As you probably have already understood, there are two partners to our interchange: the Sender and the Recipient. We have also mentioned that these are the only two partners in our exchange.

The interchange starts with the Sender preparing a transmission and sending it over to the Recipient. When the latter receives it, it decodes it and sends a receipt back to the Sender.

When the Sender receives the receipt, it updates its records and the interchange ends.

What happens if communications are interrupted at some point? If the Recipient never receives the communication, the Sender will eventually time out and resend it. It’s important to notice that the CIP does not provide any actual time out mechanism, and that you will therefore have to implement it on your own. Similarly, if a receipt sent by the Recipient is not delivered, the Sender will once again time out and resend the transmission. The Recipient will be able to recognize it as a duplicate and ignore it, but it will resend a receipt.

---