Washington, September 15-18, 1999 – London, November 21-24, 1999

Extending Site Server 3.0 Pipelines

By Marco Tabini

Pipelines are one of the most exciting technologies in Site Server, Commerce Edition 3.0 (SSCE from now on). Their main characteristic is that of providing an ordered approach at solving business problems. Generally speaking, pipelines are a good solution whenever you need to perform a task that:

1.      Can easily be divided in smaller steps

2.      Operates on the same set of data

3.      Is composed of operations that must occur in a specific order

The most obvious of the possible processes that can be performed using pipelines is order processing, which is at the core of an electronic commerce store. It should be pointed out that order processing—at least as we are using it here—is a rather broad term that encompasses everything from managing the customer's shopping cart to determining the total cost of an order, including any shipping, handling and tax fees. Other possible uses include managing business-to-business communications, such as line-of-business operations and other interchanges

Marco Tabini Marco Tabini is an electronic commerce specialist based in Toronto, Canada. He provides custom solution for clients who desire to bring their business online, and writes about electronic commerce and Site Server on various publications. He is co-author of "Professional Site Server 3" (Wrox Press, 1999).

A quick introduction to pipeline technology

Most of the presentation, as well as these notes, assume that you are quite familiar with pipeline technology. You should at least know its basics, and have used pipelines in one of your stores. The goal of the brief introduction to pipelines that you are about to read, therefore, is primarily that of creating a common foundation for the more advanced concepts that will be covered later on.

What is a pipeline?

As we have already mentioned, from a conceptual point of view a pipeline is simply a business process broken into its essential components and executed in a specific order.

From a technical point of view, a pipeline is simply a COM object configured to perform certain operations. The exact number of operations to be performed, their order and their parameters are loaded from a configuration file (whose extension is .PCF by default).

A pipeline is composed by an arbitrary number of stages. Each stage is simply a way to group operation of a similar nature in a logical fashion. Each operation inside a pipeline is carried out by a COM component that implements particular interface, generally known as a pipeline component. Site Server includes a number of built-in components whose function varies from collecting information about the products in a user's shopping cart to encrypting a set of information for a business-to-business communication.

Types of pipelines

There are two types of pipelines built-into SSCE. The Order Processing Pipeline is used for managing order processing operations in a business-to-consumer online store, while the Commerce Interchange Pipeline is used to handle business-to-business communications. Both these pipelines have their own set of stages and dedicated built-in components.

The Order Processing Pipeline

Three subtypes of pipeline templates belong to the Order Processing Pipeline (OPP). Each of them addresses a particular moment in the store workflow:

Ø        Product pipeline

This pipeline is used whenever the user requests information about a particular product in the store's catalogue. It extracts all the appropriate data from the database and applies line-item discount to deliver the most up-to-date information.

Ø        Plan pipeline

The plan pipeline is used whenever the store needs to show the user with a subtotal of his or her order. This pipeline includes the calculation of taxes, as well as shipping and handling costs.

Ø        Payment pipeline

Finally, the goal of the payment pipeline is to take care of completing a purchase by performing such tasks as processing payment information, updating the store's inventory, initiating any line-of-business procedures, and so on.

A fundamental difference between the product and plan pipelines on one side and the payment pipeline on the other is the fact that the latter requires a transactional environment in order to run. Transactions are handled using Microsoft Transaction Server (MTS) 2.0 and the transactional context is provided directly by the pipeline object that is used to execute the pipeline.

The Commerce Interchange Pipeline

As already mentioned earlier, the Commerce Interchange Pipeline (CIP) is used to perform business-to-business tasks. These usually entail the transmission of a set of information—called interchange—between two computer partners. A receipt can optionally be sent by the receiving end to acknowledge that an interchange was successfully received.

Clearly, the format in which the interchange is sent must have been previously agreed upon by the two parties, or it will be incomprehensible to the receiving end. The CIP per se is protocol agnostic, in the sense that it does neither favor nor require a particular protocol to be used when transmitting and receiving data. As a matter of fact, one of the great things about pipelines is that they can be extended in an arbitrary way by writing custom components—and therefore you can support whatever format you require to.

 There are only two subtypes of CIP pipelines:

Ø        Transmit pipeline

This pipeline is used to format the interchange according to the appropriate rules and send it to the recipient.

Ø        Receive pipeline

The receive pipeline sites at the other end of the interchange, and is invoked whenever a transmission is received by the server on which it resides.

Transmissions can take place in a variety of ways, including several Internet protocols such as HTTP, SMTP or a DCOM transmission. Pipeline extensions can provide interfaces with custom protocols, such as VANs and frame relays.

Data providers

There are two data providers within a pipeline. The dictionary contains the information that the pipeline is supposed to manipulate as a result of its execution. For the OPP, the dictionary is an Orderform object containing all the information required to process an order. For the CIP, it is a Dictionary object containing all the information required to create and transmit an exchange or, for the receive pipeline, the interchange that must decoded and interpreted.

The context represents a collection of the support data required by the pipeline to run correctly, and is also a Dictionary component that is created before the pipeline's execution and passed as one of the parameters when the pipeline object is invoked.

Stored in the context can be a variety of commonly used MSCS objects, such as intrinsic ASP objects (e.g.: Response, Request, and so on), the MessageManager, the QueryMap and others. The context can contain an arbitrary number of objects and other data values of any type that can be simply added as a name/value pair to the Dictionary object.

Editing pipelines

SSCE offers two ways for editing pipeline configuration files. A Win32 editor can be used if the PCF file is directly accessible from the machine where the editor is running through an NT filesystem or UNC pathnames. The system also includes an HTTP editor that can be used through a normal Internet connection (using the appropriate login information).

Generally speaking, the Win32 editor is more powerful than its HTTP counterpart, because it takes full advantage of the Windows GUI and makes it possible to create customized pipelines (which we'll briefly introduce below). However, the HTTP editor is extremely convenient, since it can be used from anywhere, as long as your server is reachable from the Internet.

The Win32 editor

The Win32 editor is a C++ application that can is installed together with SSCE. It provides a simple and intuitive way to edit pipelines through the graphical interface that you can see above.

The range of functionality provided by the Win32 editor is very varied. Besides adding and removing components from the pipeline, it is also possible to modify their property and, if the components implement the proper COM interfaces, view the name/value pairs that they access in both the context and the dictionary.

In "Expert" mode, the Win32 editor makes it possible to create custom pipeline that do not necessarily follow the same sequence of operations as their pre-defined counterparts. This allows the developer to create pipeline that address problems beyond order processing and business-to-business interchanges: stages can be moved around, added or deleted altogether, and components can be hidden so that they become "built-in" when the editor runs in normal mode.

The HTTP editor

On the other hand, the HTTP-based editor provides a smaller set of functions. First of all, because the access to COM functionality through an HTTP connection is limited, a special set of ASP pages must be created to let the user edit the properties of a component. Naturally, this means that, if you want to develop a component, and you want it to be fully compatible with all of Site Server's functionality (quite an important requirement, particularly if you want to sell it as an add-on to SSCE), you will have to work out property pages that can work both in their Win32 and HTML form.

In addition, the HTTP editor doesn't have any "expert" mode. This means that it can only be used to edit existing pipelines, and only those pipelines that respond to SSCE's standard specification (that is, traditional OPP and CIP pipelines only).

Pipeline interfaces

When writing a pipeline component, you must implement one or more COM interfaces in order for the pipeline objects and editors to become aware of the fact that your component exists. Even though there are several interfaces that you can implement, only one is absolutely required, and in general you will not necessarily need to use more than two. A full-fledged component, however, will need all of them!

IPipelineComponent

The only required interface for your object will be IPipelineComponent, which defines the entry points used by the pipeline objects to execute the component's code.

IPipelineComponent only has two member functions, Execute and EnableDesign. Although they must both be implemented, only the first one will usually be of any interest; in fact, you can leave EnableDesign empty altogether.

Execute takes a number of parameters that represent the pipeline's entire set of data:

IPipelineComponent:Execute (

  Dictionary,

  Context,

  Flags,

  ErrorLevel)

The Dictionary and Context parameters contain the pipeline's dictionary and its context respectively. As mentioned earlier, these will be two Dictionary objects—or two objects that implement the IDictionary interface (such as Orderform). Flags is a reserved parameter and should be left empty, while ErrorLevel must be set before exiting the component to a value indicating its result. The possible values are:

1.      Success

2.      Warning

3.      Error

It should be noted that these values have been arbitrarily defined to work with the built-in MSCS pipelines. If you edit the pipeline using the Win32 editor in expert mode, you will be able to set the maximum error level that it can withstand. When the value of ErrorLevel exceeds the maximum value set for a given pipeline, the execution stops and the pipeline returns an error.

The reason why there is a distinction between warning and error conditions is that in the plan pipeline, not all the components may be able to run properly even though the pipeline is being executed with all the appropriate data required for a given step in the purchase process.

For example, suppose the store is at the basket level with a new customer; in that case, no shipping or billing information, which will most likely cause several components in the Shipping, Tax and Handling stages to fail. However, that information is not needed in the basket page, and therefore the pipeline should not return an error condition. As a result, the plan pipeline can withstand a maximum error level of two (Warning). The Purchase pipeline, on the other hand, is run only at a critical point in the store, and therefore can only withstand the Success condition.

IPipelineComponentAdmin

When exposing properties for your pipelines, it's always a good idea to implement them as COM properties of your component first. This will allow you to implement any validation that you need for them in one place and one place only, as opposed to having to handle it separately if you implement property bags or property pages. It's also a good idea to make these properties handle Variant values directly, so that they will be easily accessible from ASP scripts. This will mean a little more work if you're using Visual C++, but it will be worth the effort if you are planning to use Micropipes in your projects or (indirectly) if you want it to be compatible with the HTTP editor.

IPipelineComponentAdmin provides a simple mechanism for setting and retrieving a component's properties through a Dictionary object, which can once again used when you're planning to work with Micropipes or the HTTP editor. The GetConfigData function is called by the object that wants to set your component when it needs to read the configuration parameters that your component will accept:

Function IPipelineComponentAdmin.GetConfigData as Object

This function should return a Dictionary object that contains a name/value pair for each property that the component supports. The Dictionary should contain a complete collection of all the properties, even if they are Null or empty, because the caller might use GetConfigData to establish how many and which properties the component supports.

SetConfigData is a method of IPipelineComponentAdmin that is called whenever the object's properties must be changed:

Function IPipelineComponentAdmin.SetConfigData

  (

    dict as Object

  )

It's important to understand that dict doesn't necessarily have to contain a name/value pair for each and every property that your component exposes. In fact, it could even be empty! This means that your implementation of SetConfigData should enumerate all the pairs in dict and set only the properties that are defined in it. If it doesn't do so and, for example, it simply attempts to read all possible properties and set their value to the value returned by the Dictionary object, it would be unable to determine whether a particular value is Null because the pair doesn't exist in the object or because the caller set it.

IPipelineComponentDescription

The last interface defined by SSCE for writing pipeline components is used to report the values that are read and written by the component to the pipeline's dictionary and context. As you can imagine, there is no strict need to implement this interface, since the information that you return is…well, just information, and it won't be used actively during the pipeline's execution.

However, you should consider that it might be useful for other people to know what the component does when it is run. This is particularly true if you are considering to sell your component as an add-on to SSCE and therefore its users will not be able to contact you directly for more information.

IPipelineComponentDescription provides three methods, all identical except for the nature of the information that they report:

Function IPipelineComponentDescription.ContextValuesRead as Variant

Function IPipelineComponentDescription.ValuesRead as Variant

Function IPipelineComponentDescription.ValuesWritten as Variant

ContextValuesRead is used to report the values that the component requires from the pipeline's context (no values should ever be written in the context); similarly, ValuesRead reports the values that the component reads from the pipeline's dictionary, while ValuesWritten reports the values that are written by the component to the dictionary.

All three methods return a Variant value that points to an array of Variants. Each element of the array, in turn, contains a string that corresponds to one of the values that are being reported. If you are working in Visual C++, the array that you return encapsulated in a Variant must of type SAFEARRAY.

Getting ready to write pipeline components

The first step in writing pipeline components consists of creating the appropriate environment, so that you will worry about writing optimal algorithms and code rather than trying to figure out why you can't compile your component.

You should start by making sure that the machine on which you are going to work knows what MSCS objects and interfaces are. You won't be able to write a component in Visual Basic, for example, unless all the MSCS objects and interfaces you want to use are registered on that machine.

If your development computer runs on Windows NT Server, you could install a complete copy of Site Server on it—that would ensure that all the required components are registered. If that isn't the case, or if you don't want to install additional copies of SSCE, you will need to copy the DLL files from the folder where an existing copy of SSCE is running.

Next, you'll have to install the Site Server SDK, which is part of the second CD of Site Server. This will be particularly helpful if you're planning to work in Visual C++, since it contains an extension for the ATL wizard that will let you create a new component in just a few simple steps.

Choosing the right language

Because pipeline components are essentially a particular class of COM objects, you will be able to write them using any development tool able to generate apartment- or free-threaded COM objects. This includes Visual Basic and Visual C++, with which most of you will be already familiar, but also other tools, such as Visual J++ and Borland C++ builder. For this presentation, we'll focus on VB and VC++.

Choosing the right language for writing your components is essential to your success. There are at least to aspects of each language that you will want to consider: the overall performance and the convenience. While you will certainly want to lean more on the former for your production environment, convenience will help you concentrate on the validity of your algorithms, rather than on the nuances of a particular language, when you are trying to build a prototype.

If you follow this line of reasoning, VB will certainly be a good choice when you're developing your components and therefore have to first of all make sure that your conceptual design works flawlessly. VB is easy to program, fully compatible with COM and provides a great debugging environment.

On the other hand, you it's probably fair to say that VB does not provide the best possible performance, not the highest flexibility in what can be done with it. Naturally, you can extend its possibilities by writing DLLs in another language to which your VB code will interface, but if you have to adopt another language to get things done, why not go all the way and use a development tool that produce faster and more compact code?

If that's what you want to do, Visual C++ will probably be your best choice. While it's certainly more difficult to write a component using this language, the Site Server SDK contains an extension to the project creation wizard that will automatically create the entire skeleton for a complete component with just a couple of mouse clicks.

Writing pipeline components in Visual Basic

The development of a pipeline component in Visual Basic starts with the creation of a new COM DLL project. You can also add a new custom component to an existing COM DLL project. Keep in mind that the project will have to use the Apartment threading model, or you will not be able to run the pipeline component.

Implementing the basic functionality

Next, you can start by implementing IPipelineComponent. It's a good idea to start by writing an empty implementation for EnableDesign, so that you will be able to forget about it and concentrate on more important matters, such as writing your version of Execute, which, as we have seen earlier on, is required to make your component work properly—or even compile.

The important thing to remember when writing the Execute function is to remember to return a value—if you don't, the pipeline's execution will terminate with an error condition even if your component worked perfectly. Also to keep in mind is the fact that you should never write any value in the context dictionary, because that is discarded when the pipeline has completed its execution.

Finally, be careful about the values that you set in the pipeline's dictionary. As you probably know, pipelines rely on a "trust" system for controlling the flow of operations, rather than on conditional operations. As a result, you should set a value in the dictionary if, and only if, it is Null, which indicates that no other components has already set it.

Handling parameters

A good way to handle configurable parameters, as we mentioned earlier, is to start by implementing them as properties of your component. This will make it possible to centralize all the conversions and validations that you need to perform on each parameter in one place only. It's a good idea to leave the parameters as Variants, which will make it possible to easily use them from ASP scripts.

Once you've implemented your properties, the next logical step will be to implement property pages and property bags; this simple mechanism will make your component easily usable from other COM objects and applications, including the pipeline objects and the Win32 editor. In other words, once you've implemented property bags, your component is ready to go.

Finally, you can implement IPipelineComponentAdmin. This is useful only if you need to make your component compatible with the HTTP editor, since the latter provides a number of built-in functions that make it easier to save and retrieve parameters.

When implementing SetConfigData, keep in mind that the simple fact that a value you extract from the Dictionary object is null alone does not mean that you should set the corresponding property to null as well. In fact, you will receive a null value even if the pair wasn't set at all! Therefore, the best way to retrieve all the correct values is to enumerate the contents of the Dictionary one at a time and set the properties that correspond to what you receive.

Additional steps

To add more user-friendliness to your component, you may want to implement IPipelineComponent. As we have already mentioned, there is no particular requirement to do so, except for the fact that it will make it easier for developers to use your component. There is no need for complex code to write the various methods of IPipelineComponent—all you need to do is return a Variant that contains an array whose dimension corresponds to the number of context or dictionary name/value pairs that you want to report. Each element of the array will, in turn, be a Variant containing a string that describes the name/value pair that you intend to report.

Making the component visible

Your next step should consist of making the component visible to the Win32 and HTTP editors. This is done by adding several GUIDs to the Implemented Categories subkey of the Registry key where your component's information is registered. This not only advertises the component as a pipeline component, but also indicates what stages it has affinity with.

You can start by opening the Registry Editor and look for the component; assuming that you called it MyLibrary.MyComponent, you'll find it under the following key:

HKEY_CLASSES_ROOT/MyLibrary.MyComponent

Within that key, you will find a sub-key called CLSID, whose default value contains the Class ID for the component. If you copy that into the clipboard and then use it to make a search starting from the HKEY_CLASSES_ROOT/CLSID key, you will find the component's main registry entries.

One of the sub-keys in the component's main entry is called Implemented Categories, and contains several other sub-keys that reference a number of Class IDs. This sub-key is used to express what kind of component is described in the Registry, and that's where the editor looks to see whether a COM object is indeed a pipeline component.

In order to be marked as such, a pipeline component needs to implement the following category:

{CF7536D0-43C5-11D0-B85D-00C04FD7A0FA}

Specifying affinities

You will also need to specify the stages to which your component can be added. MSCS provides a different category for each possible stage, plus a special category for those components that work in any stage. They are all listed in the Include\Microsoft Site Server\Commerce\Pipe_Stages.h C++ include file. Even if you are not familiar with VC++, you will not have any problem finding and copying the GUIDs.

Adding support for the HTTP editor

Even after you've added all the necessary entries in the registry, you will not be able to edit your component through the HTTP editor. This happens, obviously, because the editor has no means to show the property pages for the component.

The solution of this problem consists of creating a set of ASP pages that take care of editing the component's properties. You will need two pages: the first will allow the users to edit the properties, and the second will work as the target of an HTML form and write the new values into the pipeline configuration file. The editor will automatically recognize the two files, as long as their names are generated using the following convention:

Ø        The main property page must be called with the same name as the component, with the exception of any dots, which should be turned into underscores. For example, the property page for MyLibrary.MyComponent should be called MyLibrary_MyComponent.asp.

Ø        The second page should be called using the same convention as for the main page, with the _Post suffix appended to it (i.e.: MyLibrary_MyComponent_Post.asp).

Writing pipeline components in Visual C++

The essential concepts that we have seen so far also apply to Visual C++ component development. The major difference will be in the increased complexity of the language, which will, however, also result in increased flexibility and performance.

Most of the hurdles of creating a new component and implementing the right interfaces are taken care by the ATL wizard extension that comes with the Site Server SDK. In fact, once you've created a new pipeline component using it, you will end up with a COM object that already implements all the pipeline interfaces, persistence and property pages, as well as providing a plethora of support functions that can be used to perform tasks as varied as accessing the Registry or manipulating MSCS objects.

The helper functions

There are a large number of functions that the template created by the ATL wizard extension provides. They can, however, be grouped in these categories:

Ø        Manipulating MSCS Objects

These functions make it possible to access MSCS objects, which are usually passed as pointers to IDispatch interfaces. In particular, the functions provided by the template focus on Dictionary and SimpleList components, which will be the most likely candidates for manipulation.

Ø        Generating unique IDs

The GenUniqueID function generates a unique identifier similar to the ones that an MSCS store uses for order numbers and shopper IDs.

Ø        Accessing error lists and the MessageManager object

If you need to generate an error and make it available to the pipeline, you can access the MessageManager and the error lists in the Orderform by using these specialized functions.

Ø        Manipulating VARIANT values

Since all the values that you extract from a Dictionary object are Variants, it's good to have a few functions that let you convert a VARIANT value into either a string, an integer or an IDispatch pointer, which are easier to handle for a VC++ application.

Ø        Accessing the Registry

Finally, if you need to access the Registry while your component is being registered, the template includes several functions that make the process of declaring supported threading models or stages.

Implementing the component's functionality

As before, you will want to start by implementing the Execute method, which is where the heart of your component is. The same considerations that we mentioned above are essentially valid, and the only real difference that you will encounter is in the way the language works when you compare it to Visual Basic. This is the main reason why Visual C++ should be your choice for production but not for prototypes: in addition to the difficulty of having to write a sometimes complex algorithm, you also have to deal with the fact that C++ is a much more complex language (although not necessarily more difficult to use) than VB.

Next, you will want to work your way through the process of implementing properties. Once again, you may find it easier to start by writing the properties as methods of your component, and then adding the property pages and the methods of IPipelineComponentAdmin at a second stage. Keep in mind that working with VARIANT values is not as easy here as it is in VB, although you can use the functions provided by the template to simply convert the Variants into manageable C++ types.

The final step will be that of adding the implementations of the methods that belong to IPipelineComponentDescription. In this case, you will have to create SAFEARRAY objects and return those encapsulate in VARIANT objects. Even though this might be a slightly more difficult task in VC++, the template already contains boilerplate implementations of the methods, so you shouldn't have any problem.