Washington, September 15-18, 1999 – London, November 21-24, 1999

Lecture Notes for Using XML in ASP Applications

Stephen Mohr

How is XML Used in ASP?

Why do we want to use XML as a tool in an ASP-based application?  HTML, after all, has served us well so far and it, too, is text. As I see it, XML does three important things for the ASP community. First, it divorces data from its presentation. This has long been the biggest strike against HTML, in which data is submerged beneath presentation. Second, like HTML, it allows us to transmit data between dissimilar platforms without concern. In this, XML preserves the best of HTML and enhances it. It allows us to express complex structure and content in text. Because it is purely data, there is no concern for distortion between platforms. Finally, XML lets us build Web-based services that may be consumed by nontraditional clients. The requestor of an ASP may be an automated agent or some other type of program. Using XML and leaving visual presentation to the last minute on human-focused browsers lets us offer our content to a wide variety of clients. XML becomes a tool for implementing applications. Instead of its original focus on marking up static documents, XML is a dynamic technique for expressing data as it flows between applications.

Stephen Mohr Stephen is a system architect for an Internet and e-commerce consultancy. His work originally involved the programming of object oriented applications in C++ and COM. More recently, he has evolved to the specification and design of software systems using COM and Web development tools and languages. He is involved in the selection of XML for ongoing projects involving multimedia content over intranets. Stephen is currently working on his book, Designing Distributed Applications - with XML, LDAP, IE5 and MSMQ.

Browser Compatibility

Speaking of browsers, however, we must recognize that for the immediate future browsers will be our primary consumer. Additionally, the browser will be part of your debugging arsenal as you build ASP applications using XML. Internet Explorer 5.0 is the first mass-market browser to offer native XML support. It includes a default style sheet that displays XML content as a collapsible tree view. It includes the MSXML parser, which is going to become very familiar to you over the course of this presentation.

Mozilla hasn’t ignored XML, however. The NGLayout engine, code-named Gecko, will include XML visual presentation similar to that found in IE. Specifically, XML documents with linked CSS will be rendered appropriately. It relies on James Clark’s C-language expat parser for its processing. There is also an effort to embrace the MathML vocabulary so that Gecko will not only render appropriately marked-up equations, but serve as an equation editor as well.

Configuring IIS for XML

There are two things you should do to ensure XML content is passed correctly. First, you will want to configure the MIME type xml/text. For IIS, go into the MMC and right click on the Web server. Under Properties, select the HTTP Headers tab and click the File Types button in the MIME map section. Add text/xml as a type associated with the extension .xml.

Next, whenever you are sending pure XML documents as the result of an ASP (we’ll delve into this a bit more fully later), set the ContentType property of the Response object to text/xml. This ensures the proper headers get set and enables the default support for XML in IE.

Parsers

XML is just a simple text construction. You could write your own parser, but you don’t need to do this. There are plenty of excellent parsers out there for you to use. Best of all, two standard models for processing XML have been devised and are supported in a number of parsers. These are the Document Object Model from the W3C (whose Level 1 specification is a Recommendation), and the Simple API for XML (SAX), which was an ad hoc effort contributed by the XML community. A number of these parsers are available in component form, either as COM components or as Java classes.

W3C DOM

The Document Object Model offered by the W3C consists of a tree representation of an XML document. Individual tags and attributes become nodes on the tree, so working with an XML document with the DOM becomes an exercise in walking through a tree and performing some processing based on the nodes encountered. The DOM also has methods for the creation of XML documents and tags, so we can dynamically create and modify XML documents in our applications.

The DOM includes several key objects:

Ø        Document – represents the entire document and lets you create new tags

Ø        Node – used to represent an individual element or attribute

Ø        Element –refines Node to focus on elements

Ø        Several collections – for handling child elements and attributes

While each object can be represented by an interface, the DOM also provides for a simplified set of interfaces that streamline things for scripting languages. In the simplified view, for example, you deal with Nodes, not elements and attributes. While this adds some ambiguity, it simplifies things in practice, as we typically know from the context what type of object we’ll be dealing with. Since everything is a node, we don’t have to spend time obtaining the particular interface we need.

The DOM consists of two levels. Level I is the core model and has attained Recommendation status. It defines the interfaces and objects that we will use to manipulate XML documents.  Level II is still in draft form and consists of extensions to the core. It is intended to add support for an event model, access to a CSS object model, and a query language.

SAX

SAX takes a different approach to working with an XML document. It stays true to the parser concept by not building a data structure to represent the document. Instead, an application using the SAX API responds to events – such as the start and finish of an element – generated as the parser processes the incoming stream.

SAX was developed when a group of interested programmers decided not to wait for the standards bodies to develop an event-driven XML API. Although it does not have the backing of a formal body, SAX is implemented in several parsers, including those from IBM. Since SAX-implementing parsers don’t attempt to build a structure to represent the complete document, they can be highly efficient in their use of system resources. This makes them useful for working with very large documents or when the consumer is interested in only a small subset of the larger document. The burden of maintaining the larger picture, however, rests with the application.

MSXML

MSXML is a COM component that ships with IE5 and implements the W3C DOM as well as some extensions. It also supports a demonstration of the XSL draft and Microsoft’s XML Schema proposal for metadata. A redistributable version is available, although it relies on the presence of some components from IE4 or later. Originally developed to support channels in IE4, the success of XML led Microsoft to build a more robust version of the component and implement the DOM. This is the component we shall be using in our sample application.

Because the original version did not support the DOM, and IE5 was then well off in the future, Microsoft partnered with DataChannel to develop a DOM-compliant parser in Java. DataChannel is currently maintaining this parser under the name XJParser.

Other Parsers

Dozens of XML parsers are available, so many that it seems you can simply choose your language, API, and licensing model. A long listing is available on http://xmlsoftware.com/parsers/. I’d like to mention several that keep turning up in applications, however.

IBM has made a substantial commitment to XML through its alphaWorks organization. Two products of this are XML4J and XML4C, parsers for Java and C++, respectively. Especially noteworthy is the fact that these products support both the W3C DOM and SAX.

Sun has jumped on the XML bandwagon with the Java Project X package of Java classes for XML. A recent effort, this package supports the DOM and SAX.

DataChannel offers XJParser, a Java implementation of the DOM. XJParser grew out of DataChannel’s partnership with Microsoft before MSXML became DOM-compliant.

The Agenda Application

At this point I’d like to introduce you to a simple application I’m going to use to illustrate the various tasks involved in using XML in ASP-based applications. This is an agenda search application for this conference. A user specifies a venue, day, and track, and receives a list of sessions. The client consists of an HTML form page, FindAgenda.html, that constructs an XML document containing a query. This document uses a vocabulary whose root is the AGENDA_QUERY element. Conformant documents contain one element for each form element: VENUE, TRACK, and DAY. An ASP, Agenda.asp, parses the document, finds the data, and sends a reply back to the client. This can happen in one of two ways because we need to look at several methods of communication between the client and the server. If we click on the button labeled New Page, an XML document linked to a CSS is returned, resulting in a richly formatted page. If the button labeled Same Page is clicked, the document is returned to the HTML page. It is parsed in script that creates HTML tables for presentation within the page. All results documents use a vocabulary whose root is the AGENDA element. AGENDA contains one or more SESSION elements, each of which contains the elements TOPIC, SPEAKER, ABSTRACT, ROOM, and TIME.

While this simulates a search engine, I’ve greatly simplified the search function. This is because this is a presentation on XML and ASP, not databases and ASP. In the new page case, I’ve constructed three static documents, one for each day. In the same page case, I’m dynamically building the results document, so any search returns the same data. What is important here is how documents are constructed, transmitted, and parsed, not the somewhat tongue-in-cheek results.

Document Building: Strings

Since XML is just text, the easiest technique is to construct documents using the string manipulation techniques native to the scripting language. This is best for small documents with simple structure. We could have used this approach in FindAgenda.html. In the example in the slide, we build a string, sXML, from hardcoded fragments and the values of the HTML form elements. This is easy, but you get no help from MSXML.

Document Building: Data Islands

The structure of our form and the query it represents dictates a fixed structure for all our queries. It would be nice to have a template for the fixed portions of the document. This is a pattern that will turn up again and again in XML applications. Happily, MSXML implements a proprietary technique termed data islands. The name comes from the fact that they allow for data binding between XML documents and HTML elements in the browser. For our purposes, however, you can think of data islands as dynamic XML documents embedded in Web pages that expose the DOM interface to script in the body of the page. We use it to provide the shell of an AGENDA_QUERY document. When the user clicks one of the buttons, we insert the values from the HTML form, then retrieve the entire text of the document for transmission to the server. This is particularly useful in the same page case; we can reuse the same document over and over again, changing only the particular PCDATA values each time. This happens in three lines of code:

root.childNodes(0).text = Venue.children(Venue.selectedIndex).text;

root.childNodes(1).text = Track.children(Track.selectedIndex).text;

root.childNodes(2).text = Day.children(Day.selectedIndex).text;

The childNodes Collection

Clearly, the childNodes collection is something important to understand. Each node, from the root on down, has this collection as one of its properties. It represents the immediate (i.e., one level) child elements of the node in question. Attributes are handled by another collection. Our nodes VENUE, TRACK, and DAY are all children of the root, AGENDA_QUERY. The collection is organized by an ordinal index that is zero based. Thus, the first child, VENUE, has the index 0, TRACK has 1, and DAY has 2. ChildNodes(x) returns an instance of the child Node, so appending .text allows us to set the text property – the PCDATA content – of the child node. While we know the number and order of the child elements in this case, we can always find out how many children a node has by referring to the length property of the childNodes collection.

Sending the Document to the Server

At this point, we have a query document we want to send to the server. HTTP gives us two ways to obtain any resource: GET and POST. GET is simple – we append one or more parameters to the URL and make one round-trip to the server. MSXML has a built-in method, load, that performs a GET and loads the target directly into the parser. GET, however, imposes a restriction (about 2000 characters in IIS and IE) on the length of a URL and its parameters. POST breaks up the request into a URL and one or more transmissions of application data. It’s a little more complicated, but MSXML exposes an interface, IXMLHTTP, to help us with this. Because the request is broken up, there are no length restrictions.

GET: MSXML

There are actually two ways to perform the GET. Since I want to demonstrate several methods of rendering an XML document visually, I decided to display the results as a new page. I use the browser’s document model and perform the GET through the window object’s navigate method. This is the same as entering the URL into the Address bar and hitting ENTER. The URL to the ASP is fixed. I’m sending the entire contents of the XML document as the value of a parameter I named QueryDoc. I get the text string representing the document through the xml property of the XML Document object. Query is the name of the data island. Each data island exposes its Document object through the XMLDocument property, so I get the document text with this line:

Query.XMLDocument.xml

If I had wanted to keep the results in the FindAgenda page, as I do when I move on to the POST method, I could have used the load method of MSXML. This method takes a single parameter, the URL of the document to obtain. This method performs the GET within the component (i.e., without involving IE) and tries to parse the target of the URL as an XML document. If it is successful, a DOM tree is constructed.

POST: IXMLHTTP

Because POST is a little more complicated than GET, MSXML exposes an entire interface, IXMLHTTP, to support the POST operation for XML documents. We use it in three steps: obtain the interface, establish a connection, and perform the POST. We get the interface by creating an instance of the progid Microsoft.XMLHTTP. The connection is established through the Open method of the object obtained in the previous step. Open takes three parameters: the name of the operation (POST) as a string, the URL of the target, and a boolean value telling whether the POST should be executed asynchronously. For simplicity, we’ll keep things synchronous. Once we have a connection, we call the Send method to transmit the application data, in this case our XML document. Here is how it is done in the sample:

var PostObject = new ActiveXObject("Microsoft.XMLHTTP");

     

PostObject.Open("POST",

   "http://localhost/XMLASP/AgendaPost.asp", false);

PostObject.send(Query.XMLDocument);

Note that we call send with the XML Document object, Query.XMLDocument, not the string representation. MSXML performs the appropriate COM marshalling for us.

Getting the XML: String

In the GET case, we sent a string to the server. We can use the QueryString collection of ASP’s Request object to retrieve that string. We give the collection the name of the parameter we sent, QueryDoc, and retrieve a string that we can submit to the loadXML method of the server’s instance of MSXML:

var parser = new ActiveXObject("Microsoft.XMLDOM");

parser.async = false;

parser.loadXML(Request.QueryString("QueryDoc"));

This technique works best in cases when you are not absolutely sure the request consists of XML. It is simple and somewhat forgiving. This code is found in AgendaGet.asp, the target of the GET operation in FindAgenda.html.

Getting the XML: DOM

When we used POST, however, we passed an XML Document object. Just as MSXML was smart enough to do the proper marshaling for us, IIS and ASP are smart enough to do the complementary marshalling on the server. In this case, the Request object uses the IStream interface MSXML marshaled over. We can thus pass this object directly to the parser’s load method to perform the parse:

var parser = new ActiveXObject("Microsoft.XMLDOM");

parser.async = false;

parser.load(Request);

This is found in AgendaPost.asp.

Walking the Tree

Whichever version of the target we use, AgendaGet.asp or AgendaPost.asp, we end up with a DOM tree to process. In general, we need to walk through the entire DOM tree. This may be familiar to those of you who have worked with trees before. We can either select a child of the root and dive down to its bottom-most level before moving on to the next child (depth first), or process each child at any given level before dropping to the next level (breadth first). The slide shows depth first traversal using recursion. The function RecurseTree calls itself recursively on a child node until the bottom is reached. At that point, the call stack is unwound and the function proceeds to the next child. The example simply walks the tree; to do any useful work we will need to do some processing based on the nature and content of the nodes we encounter.

Node Features

What we do depends on the application and the nature of the node (XML element) itself. The Node object gives us the information we need through its properties and collections. nodeName gives us the tag name of the element. nodeType is an enumerated value that tells us whether we are looking at an element, an attribute, or some other specialized form of Node (e.g., processing instruction). For ASP applications, we will generally be dealing with elements and attributes. nodeValue gives us a variety of values depending on the type of the Node. For an element, it is null. For an attribute, it is the value of the attribute. text is the PCDATA content of the element and all its children. xml is the XML content of the element and all its children. In our query document, we can use the text property safely to get the PCDATA because we know each child of the root has only PCDATA content.

Nodes also have two collections: childNodes, which we have discussed previously, and attributes, which is the collection of all attributes of a given element.

Node objects also provide some methods allowing us to manipulate the contents of the node. appendChild adds an element to the childNodes collection, while insertBefore, removeChild, and replaceChild perform similar functions with more precise positioning. Each takes a node object as a parameter. cloneNode makes a copy of the node in question, which we can use independently of the original. We can change its value, or use it even after the original has been destroyed. We will make good use of this shortly.

Document Building: Objects

Once we have a document instance (by creating a new instance of Microsoft.MSXML), we can create all the contents of an XML document. createElement and createAttributes are methods that create new elements and attributes. The resulting nodes still need to be attached to the DOM tree in their proper places. We start a new document by calling createElement to create a new root element, then assign it to the documentElement property of the document. We will see an example of this in the sample.

Once we have a Node (XMLDOMNode instance), we need to set the PCDATA and element content as well as any attributes of the element. We have seen how text can be used for the PCDATA content. setAttribute sets the value of an attribute on an element, while appendChild and setAttribute add new elements and attributes, respectively.

Returning the Results

How does this relate to the sample application?  We have a query document. Assume we’ve walked the parse tree and performed a search. Now we need to build a reply document and send it back to the requesting client. We perform three steps: set the content type of the reply, use text manipulation or DOM methods to create a reply XML document, and call Response.Write with the resulting document.

Building a Reply using the DOM

In our example, we expect to have the same element – SESSION – repeated over and over with the same structure of child TOPIC, SPEAKER, ABSTRACT, ROOM, and TIME element. We can use cloneNode on a SESSION element to speed things along. We use createElement and appendChild to create our first SESSION element and set the PCDATA content appropriately. We clone the SESSION node and its children (termed a deep copy) and append the copy to the root AGENDA element. We continue this way, setting PCDATA and cloning, for each SESSION element. When we are done appending copies, we have a completed reply document:

function BuildReply()

{

   var replyDoc = new ActiveXObject("Microsoft.XMLDOM");

   var sessionSubTree, workingNode;

  

   replyDoc.async = false;

  

   /* Build a document using the DOM. We'll hard code one for demonstration purposes, but you might use ADO and a database in a production system. */

     

   // Create the root node

   replyDoc.documentElement = replyDoc.createElement("AGENDA");

  

   // Create a SESSION subtree

   sessionSubTree = replyDoc.createElement("SESSION");

   sessionSubTree.appendChild(replyDoc.createElement("TOPIC"));

   sessionSubTree.appendChild(replyDoc.createElement("SPEAKER"));

   sessionSubTree.appendChild(replyDoc.createElement("ABSTRACT"));

   sessionSubTree.appendChild(replyDoc.createElement("ROOM"));

   sessionSubTree.appendChild(replyDoc.createElement("TIME"));

  

   // Set the PCDATA values

   sessionSubTree.childNodes(0).text =

      "What Forest?  I Only See (DOM) Trees.";

   sessionSubTree.childNodes(1).text = "W. 3. Cee";

   sessionSubTree.childNodes(2).text =

      "Build XML documents using DOM methods.";

   sessionSubTree.childNodes(3).text = "Sherwood Room";

   sessionSubTree.childNodes(4).text = "08:00";

  

   // Append a clone to the document

     replyDoc.documentElement.appendChild(sessionSubTree.cloneNode(true));

  

   // Set new values and repeat -- this is where you would loop through a result set

   sessionSubTree.childNodes(0).text = "ASP: A Dissenting Perspective";

   sessionSubTree.childNodes(1).text = "Cleopatra";

   sessionSubTree.childNodes(2).text = "The queen presents

      another view of ASP";

   sessionSubTree.childNodes(3).text = "Nile Room";

   sessionSubTree.childNodes(4).text = "09:00"; 

   replyDoc.documentElement.appendChild(sessionSubTree.cloneNode(true));

   sessionSubTree.childNodes(0).text = "Law Enforcement

      and Web Coding";

   sessionSubTree.childNodes(1).text = "J. E. Hoover";

   sessionSubTree.childNodes(2).text = "We're pretty sure he's going to

      quote Shakespeare about tangled Webs and deception.";

   sessionSubTree.childNodes(3).text = "Dillinger Room";

   sessionSubTree.childNodes(4).text = "10:00"; 

   replyDoc.documentElement.appendChild(sessionSubTree.cloneNode(true));

  

   return replyDoc.xml;

}

Visual Presentation

I stressed the importance of separating data from presentation at the outset of this session. Sooner or later, though, we are going to want to show XML to a human user. This means we need to make some intelligent decisions about how to present XML to the user. From an application point of view, we want to decide when the data will be presented to a human. Moving from data to presentation too early eliminates the possibility of using the ASP with a non-human consumer. That would suggest we leave presentation to the client browser. Unfortunately, that can cause problems with browser-specific object models. If we are sure we are giving the data to a human user for viewing, we might like to perform the presentation steps on the server.

There are three general techniques we can use to format XML for presentation: server-side creation of HTML, cascading style sheets linked to XML documents, and XSL style sheets used to transform an XML document.

HTML is safe and simple. CSS is almost as simple, but not every browser implements CSS the same way. XSL is extremely powerful, but it is still in the draft stage.

Server-generated HTML

To use this technique, we use DOM methods to walk the XML tree on the server and create whatever HTML is needed. We can also do this on the client using DHTML, as we do in the same-page scenario in the Agenda application, the heart of which appears here:

for (ni = 0; ni < xmlAgenda.documentElement.childNodes.length; ni++)

{

   // Topic and speaker

   session = xmlAgenda.documentElement.childNodes(ni);

   sessionTable = document.createElement("TABLE")

   ReplyArea.appendChild(sessionTable);

   sessionTable.border = 1;

   row = sessionTable.insertRow();

   cell = row.insertCell();

   cell.innerText = session.childNodes(0).text;

   cell = row.insertCell();

   cell.innerText = session.childNodes(1).text;

        

   // Abstract

   row = sessionTable.insertRow();

   . . .

        

   // Room and time

   row = sessionTable.insertRow();

   . . .

        

   // Spacing

   ReplyArea.appendChild(document.createElement("P"));

}

CSS

Using cascading style sheets linked to XML documents is very easy and results in rich formatting with little effort. We create a style sheet file with styling rules for each element in the XML document. Next, we link the style sheet to the document through a processing instruction in the XML document:

<?xml-stylesheet type="text/css"

      href="http://localhost/XMLASP/agenda.css" ?>

When IE loads the XML document and detects the processing instruction, it gets the style sheet and applies it to the XML. This gives us control over the presentation, but the order and content are still specified by the XML document.

XSL

The Extensible Style Language (XSL) is a very powerful language that allows us to specify rules that transform an XML document tree into a formatted tree we can display in the browser. It is still a draft, so MSXML’s implementation is subject to change. Because it transforms the XML document before presentation, we can write rules that change the order of content and even vary the transformation of a particular element depending on the context in which it appears. One benefit of this is that XSL can also be used as a query language to locate a particular element or attribute within an XML document. It is powerful, but harder to learn than the other techniques. It allows us to perform styling tasks like CSS, but unlike CSS the rules are expressed in XML.

The sample given in the slide says that for each SESSION element, we place the text value of the contained TOPIC element into a DIV HTML element with 14 point Verdana as the font.

Summary

ASP and XML are a powerful combination. You can use XML to create a platform-neutral wrapper around platform-dependent content like databases and COM objects. MSXML is a powerful implementation of the DOM API, which, because it is a COM component, fits the needs of ASP development very well. Once you are familiar with the DOM interfaces and IXMLHTTP, you can quickly write code to perform any XML-related task you require for an ASP application.

Separating data – XML – from presentation – richly formatted HTML – adds a new dimension to ASP applications. We can offer the same content in a variety of presentations based on what most effectively meets our user’s needs. In addition, we lay the groundwork for a future involving automated agents for search, shopping, and business to business e-commerce by first offering our data as data – XML – then formatting it for presentation in a separate step.

XML is a powerful yet simple technique in the toolkit of ASP developers. Quick to learn, it offers almost limitless flexibility to express your innovative solutions using ASP.

---