Lecture Notes for Encapsulating XML:  Applying the Lessons of COM to Web Design

By Michael Corning,
Microsoft Corp.

Introduction

Context

The major premise of this presentation is that since we can now see documents as objects, we should see documents as objects. Further, we should take a few lessons from COM beginning with seeing these document objects as "conceptual components." These conceptual components, then, can "discover" other conceptual components, and this ability of conceptual components to discover things about each other permits other object-oriented advantages to accrue to everyday documents such as polymorphism.

This presentation, for example, can be viewed either as a web site or as a PowerPoint-like stack. No changes are necessary to the document in order for this transformation to take place.

I am not suggesting there is a direct implementation of the COM constructs in these document objects, but I suggesting that the ideas that motivated the development of those constructs can and should now be applied to documents at large.

My Goal

My three main goals, therefore, are to show you how to

• write documents that can be treated like objects
• write an "interface" that enables document objects to be polymorphic
• implement these interfaces and transform any interface-compliant document.

My secondary goal is to introduce some related ideas that will help you to better see documents in this new object-like light.

My Strategy

Predicting trends in technology is like predicting earthquakes. Such prediction is difficult, and if you miss one, the consequences can be painful. But that has never stopped me before and it won't stop me now. This presentation is part tuition part prediction. The tuition is straightforward since is presents working code; the prediction is an intuitive hunch partly based on my assessment of the trends in technology (both in the art of Design and in implemented systems) and partly based on personal experience (building XML-empowered document management systems) and flights of fancy.

So here's my strategy: To illustrate those trends I will briefly trace the development of web-based application design, and I will make a brief sortie into the part of Design most of us either take for granted or are not aware of. The reason for this diversion is that I believe a more conscious understanding of what attracts us to our work will only help us do better work.

I will then introduce a fascinatingly simple "meta-pattern" to help you visualize what's going on as XML and XSL help me implement my document objects.

Finally, I will introduce the Design Framework I think is so appropriate to the task as we all move into the next era of Web-based application design; an era where our documents talk to each other and even to other programs we use in our daily work.

Design in an Invisible World

I want to set the stage first with a brief discussion of Design, Holarchies, and Memetics. These remarks will serve as a segue to a detailed discussion of the application that enables the document you're reading to switch between being a story and a lecture.

The Origins of Design

Design, as a specific skillset, began with physical products. And even with physical products, design was thought to be ephemeral, not something you "add" to a product as you would a different grade of steel or a different color of paint. Still, I believe it was John Kenneth Galbraith who first noted that the Europeans have generally been far ahead of Americans in recognizing the actual economic value of design -- so much for design being a by-product of fabrication.

Design in Software

The main point, at any rate, is that design began in the physical world . Soon, design became important in the computer world, too. A watermark of recognition of this ascendancy of design is Mitch Kapor's "Software Design Manifesto" first published in 1990. In his document, Kapor advocates a specialized profession that deals with issues of usability of computer software, a profession whose issues cannot be resolved with a traditional computer science tuition. Kapor's main point: design is important and is as separate (yet related) from programming as architecture is from/to physical engineering.

Another major movement driven, at least partly, by Design, is the development of object-oriented programming techniques . Among those techniques I include the development of the Component Object Model (COM). This object-oriented environment fostered the development of the canonical "Design Pattern Catalog" by Gamma, et al .

I will return to this catalog in a moment, but for now, I need to change gears a bit and relate all this design business to the business at hand: Web Development.

Design in a Web World

In the Web world Design has manifested itself in three related but fundamentally different ways. Indeed, the Web itself has created new dimensions of Design.

Well-designed Web sites can morph themselves to the needs of individual users; never before in history have so many had it "their way." This means the way people -- no individuals -- think and feel have become more important than ever to Design. The second way Design is made manifest on the Web is visually and dynamically, and the third way is in how Web applications are implemented. Let's take a closer look at each.

Look and Feeling in Web Design

This phase of web design passed through two stages, two generations.

I consider second-generation Web sites as those who factor in deep "psychological design.". Pioneers in this design modality are Hoffman and Novak and their Project2000. My favorite version of their work is actually the oldest. To my mind their http://ecommerce.vanderbilt.edu/cme.conceptual.foundations.html is a classic and a must read for all Web developers.

Third generation Web sites are ones that broke out of the straight jacket of HTML and incorporated dazzling visual and aesthetic effects. The notorious champion of Third Generation Web Design has to be that irascible iconoclast, David Siegel (I just noticed that Donna and Tom have applied some of David's lessons in the latest look they' given Project2000<g/>).

The Advent of Component-Based Web Design

Fourth-generation web site designers are the ones who took the first steps into the invisible world of design, for the fourth generation of design shifts from appearance to implementation.

Specifically, fourth-generation web site design is:

• object-centric or component-based design
• extends the reach of objects in design with JScript (to easily program objects) and
• DHTML (to give HTML an object model to program)

Active Server Pages is an important platform for implementing component-based fourth-generation web design.

Fifth-generation sites take the next logical step in object-centric Web design. Where DHTML gives the presentation of data an object model, the Extensible Markup Language (XML) gives an object model to the data itself. The ubiquitous presence of so many objects and components and so many technologies to exploit them means that web sites of unprecedented power will soon be emerging from fifth generation Web designers' studios.

So what's next?

We on the Application Server Test Team think XML will enable a virtual "living document" to emerge. To us a living document is a "conceptual component" that

• can respond to stimuli in a predictable and ordered way
• can discover information about other conceptual components
• can modify their appearance to one most appropriate to their environment
• are adaptive
• teach by example

To learn to create living documents, then, you take one apart and examine it. A virtual vivisection. A task we return to as soon as we take a closer look at what makes Good Design, and we need to examine the meta-pattern that lies beyond hierarchies.

The Qualities of Good Design

Defining Design is like defining Quality . If you're not careful, you risk your mental health. But I am prepared to make the following remarks about Design before I get into the real purpose of my talk today.

Flow

Hoffman and Novak built their work on the work of Mihalyi Csikszentmihalyi, in particular, Csikszentmihalyi's Flow construct. My synonym for flow is "deep connect" (a phrase taken from Verner Venge's "Marooned in Real Time"), and I'm sure each of you here today has also felt time, hunger and fatigue stop. These are all sure signs you're in a state of flow. Behaviorally, and essentially, flow may occur when your skills and challenges match, and both exceed a certain threshold. In a word, flow (or deep connect) is transcendent.

Design's Relation to Flow

I contend that a good Design promotes flow, and I hold this for several reasons (each of which I will discuss in turn):

• A good Design provides a context for discovery.
• A good Design is relatively easy to "relate to" meaning over time you become familiar with it; good Design, therefore, has an environmental dimension and is inherently stable.
• A good Design, like flow itself, is " autotelic " meaning it's intrinsically valuable (suggesting, at a practical level, that a good design enables us to do something we couldn't do before which is my definition of an asset).
• A good Design is bigger than any of us; meaning that even though with each repeated visit we keep learning from it, it always seems to have something more to teach us.
• We believe a Design is good because of how good we feel when using it.

Programmer Culture Conducive to Flow

Programmers are an especially inquisitive lot. They are true entrepreneurs -- never willing to leave well enough alone. They are on a never-ending journey of discovery. They are also often generous to a fault, anxious to share what knowledge they do acquire. This creates a particularly fertile environment for the production of Designs. Good Designs, then, are the product of the best minds in the business and are the fertilizer for the creativity of those who come behind the Design's original collaborators.

Designs Constantly Unfold

A Good Design is like a Great Book: each reading shows you something new, and virtually everything they (either Design or Book) show you is meaningful. Because good Designs are so rich and well conceived, good Designs tend to be stable, this means good Designs tolerate intellectual thrashing better than poor ones, and this means that good Designs are easier to learn than poor ones (or no design at all).

Good Designs tend to be bigger than any single student of them, and yet they quickly get under your skin. At this point you've crossed another line. You begin to think about them, even when you're not "working." You begin to use them often, perhaps too often, and it's not uncommon to begin to feel guilty about it; you become afraid you're a "solution in search of a problem." But you can't help yourself, so you go on using the Design getting more and more comfortable with it all the time.

Design, Memetics, and Flow

Soon, you cross another threshold, and you realize that when you're using a good Design you're working with it on its terms, not yours. Your project becomes the vehicle, the instrument with which you engage the Design. At this moment finishing the project is a secondary priority; applying the needs of your project to the Design to let the Design teach you more becomes primary.

At this moment you may realize you don't have good ideas. Good ideas have you. At this moment you have become a "Memetic Engineer."

In the final analysis, (Freud was correct) it boils down to how you feel about your work. It's almost impossible to feel bad in the presence of Good Design.

Beyond Hierarchies Holarchies Defined

Everyone who uses XML understands and appreciates the value of hierarchies. Not many, I fear, understand and appreciate the value of holarchies. Hierarchies are structures with fixed ancestry. Parents have children in hierarchies, and parent can be children themselves. But parents and children can't switch places in hierarchy. Holarchies consist of holons and clonons instead of parents and children. Holons and clonons can be combined four ways. But the key attribure of holons and clonons is that they can take each others place.

XML as Holarchies

XML and XSL provide an excellent example of holarchies. A simple XML file is a holon consisting of clonons; nodes, text, and attributes all examples of clonons. A Processing Instruction is a clonon that points to another holon: an XSL stylesheet. The XSL stylesheet (is an XML file, and thereby a holon) -- to the XML file -- is a clonon because the XML file uses the XSL file. But this same XSL is a holon while it processes the XML file, and the XML file becomes the clonon.

But wait, there's more. Once rendered, the XML file can change the underlying XSL file As a result, the next time the XSL file processes the XML the result is different -- perhaps a sort or filter has been activated. But in all cases, the same meta-pattern was used to model this interdependent processing.

Holarchies and Innovation

When you expand you thinking in this way, you invite innovation. You attract memes. Holarchies encourage you to think more dynamically, and in the process, things like holarchies enable you to come up with new ways of seeing things. For example, how can you edit an htm file in IE alone and not rely on something like the DHTML Edit component?

By thinking of your document as a holarchy you can see that while IE displays the HTML off the file system, you can also render the same HTML file as an XML file in an XMLDOM object. You can then edit the HTML using custom DHTML dialog boxes or property sheets, change the underlying XML, and send this changed XML back to the server to be saved back to the file system. Here the document is a holon when you authored it, and a clonon when you loaded it in memory as XML. And it was both holon and clonon at the same time.

So, enough with the preliminaries. Let's get on to something concrete.

The Model-View-Controller Design Framework

My hands down favorite Design, especially for user-interface projects (but I'm exploring some pure server-side designs, as well) is the Model-View-Controller Design framework. MVC is a collection of Design Patterns (mainly the Observer, Composite, and Strategy patterns).

The Model-View-Controller Pedigree

MVC has a long and distinguished pedigree. MVC was first created in Smalltalk-80 to program the Smalltalk user interface (Krasner and Pope, Journal of Object-Oriented Programming, Aug/Sep 1988). One of the most thoroughly documented treatments of MVC can be found in Gamma, et al.'s "Design Patterns," Addison-Wesley, 1995 . Another excellent treatment of the framework can be found in Clemens Szyperski's "Component Software: Beyond Object-Oriented Programming," Addison-Wesley Longman, Inc., 1997. And a very useful tutorial is in Chapter 3 of Rod Stephens' "Advanced Visual Basic Techniques," John Wiley & Sons, 1997. One of the MVC constituent design patterns, the Observer, was recently incorporated into Java with the Observer interface and the Observable class ( Observer and Observable).

When it comes to programming the user interface, I have turned to MVC time and again. Several times I have approached a programming project the old fashioned way, more procedural or linear than anything else, and every time I've run into problems that don't exist in an MVC application. As you can see from the diversity of MVC programming environments, it is a remarkably versatile framework. Ironically, I have found the most approachable environment in which to use MVC is scripting. You'll see what I mean shortly.

The main goal of my talk today, then describes the nature of the MVC design framework. Incidentally, I have mixed feelings about referring to MVC as a framework instead of a pattern, but I find this little idiosyncrasy of MVC all the more endearing. I generally opt for framework because MVC is a collection of design patterns. MVC kind of reminds me of Bertrand Russell's famous paradox : is a catalog of books a book? If you prefer to call MVC a design pattern, I won't argue with you (though you should probably read Szyperski pp138 first). And to the extent you develop an implementation of MVC and especially if the implementation is reusable, you are almost certainly (even according to Gamma and company) talking about a framework.

One reason the MVC framework is so popular (still) is that it is so intuitive. MVC "componentizes" an application into three roles. Each role is related to one orthogonal aspect of the application. The MVC Model encapsulates all the data of the application. The View represents that data on the screen, and the Controller mediates the user's (and occasionally other views') interaction with the screen. Actually it's more accurate to say that Views mediate Model data and the screen and Controllers mediate the screen and the user's interactions. Figure 1 shows the essential relationships between these roles, the data, and the user. In most applications, however, the View and Controller are more tightly coupled than I show here.

The MVC Rules of Engagement

This division of labor immediately creates "rules of engagement" with the user and among the roles inside the framework. As you begin to work in the world of MVC you will probably have lots of competing feelings about what you're doing. Part of you will probably want to be an MVC purist and follow MVC rules scrupulously. Then later you will probably feel like rebelling, haunted by vague memories of Emerson's grave warning, "A foolish consistency is the hobgoblin of small minds."

The first rule: only the Model can manipulate data. This is especially important in user interface designs with complex interacting views.

The second rule: each view looks to the Model for the data to display. I have found XML to particularly useful as a data manager because XML can use XSL to morph any given data into a virtually unlimited presentation. Still, it's up to the View to actually change the screen. The combination of Jscript and XML enables me to create elegant object models for each of my MVC applications.

The third rule is that MVC applications are inherently and necessarily event-driven. This is the most unusual, interesting, and unique part of the MVC framework. Some of us cut our event-driven teeth on the Macintosh (I used a Mac to create a hospital patient accounting system back when FoxBase/Mac was in beta), and most of us still remember the difference an event model made on our designs and on the way we thought about our systems. You will see that most of your creative cycles will be invested in thinking about the event model in your MVC apps, and it's the event model that will get you out of most of the jams less sophisticated design patterns leave you in. The event model is crucial to MVC designs because it is through the propagation of event masks throughout the application that all three independent roles (Model, View, and Controller) become interdependent .

I note in passing here, that MVC also presages a broader movement in the computer community toward a declarative style of programming. COM+, for example is pregnant with technologies that rely on the propagation of events. A word to wise (and otherwise).

Views register (and if they will be destroyed, unregister) their interest in events with the Model. The Model maintains a collection of View/Event pairs so that when one of the events in the event model fires, the Model can notify the View(s), and the View(s), in turn, respond by looking to the object model for the current state of data they present to the screen. By the way, the Events in those recorded View/Event pairs are bit masks. This means that interest in more than one event can be stored for each View. See my article, Bit Masks and Bitwise Operators at asptoday.com for the details of how bit masks work.

Before we go much farther, permit me to give a very simple demonstration of how an MVC framework could improve the Search screen as http://www.asptoday.com/search.asp. In that window you will notice that all the different search properties are mutually exclusive, and that the default property is category.

I typed my name in the author field, and if I hit the Enter key I get ADSI/CDO articles?and that's a topic on which I have never read, never mind written.

Clearly, the problem is that there is a disconnect between the search page's object model (if there is one...and there always is one, it's just that too often the object model is unconscious or implicit or accidental, depending on the consequences) and its event model. I interacted with the application but there was no way for the "Search by author" radio button to know that the "author" text box it is related to now has data. Further, the page is relying on a radio button to inform the application of the value of the search parameter. This is odd, since the value of the radio button alone is insufficient information with which to conduct a search. But more important, in an MVC application, the data is managed by the Model, not the Views. In an MVC application the search function would look to the Model, not the form, to get its search parameter. And in an MVC application as soon as I start entering data in the author field the Model knows about the edit and sends an event notification that the search type is now author. The "Search by author" radio button is clearly interested in this edit event so when the "Search by author" radio button is notified it clicks itself .

I noted above that the Model can expose an arbitrarily rich object model, and while that's true it obscures the fact that each view has an object model as well. Let's turn to a more detailed examination of those object (and event) models now.

Reflexive Documentation The Implementation is the Message

Yes, I'm shamelessly corrupting Marshal McLuan's oft quoted insight, but I know of no better way to express an important point about this document and the application is documents. When you separate content from conduit, you can do remarkable things. The document you read, and the application you use are like the two faces of the Roman god, Janus. It's rather like looking at your own throat with an endoscope: you're looking at your own internals, and that usually takes some getting used to.

For example, here's a screen shot of the application i will now document. The content you see here in this shot is the content you are reading now.

Here's a tip: always explicitly cite WIDTH and HEIGHT attributes on IMG tags. If you don't you will generate invalid XML.
And here's a related tip: open your XHMTL files with XML Notepad. It will test for well-formedness, validate XML, and if all goes well, will clearly present the structure of your schema (very helpful when you're developing XSL patterns).

The Roles

So, from the screen shot you can see three main Views, topics, points, and discussion. I use DIV tags instead of SELECT controls because I need more flexibility than SELECT and OPTION tags provide.

The Model encapsulates all data used by an MVC app. In the MVC app documented by this document, that data comes from two sources:

1. the current xhtml file
2. stylesheets embedded in presentMgr.htm

The Model also exposes some data of its own such as the current selection in any of the three Views.

Designing Controllers can be one of the trickiest things about MVC designs. In the case of the MVC presentation manager, I encapsulated all Controller functionality in one Controller interface/object.

The Object/Event Model

Writing about how an object model works is the least effective of three ways to convey how an object model works. Far better to study the object/event model diagram.  The third option is to run through a debug session of the application, which I encourage you to do.

Code Review

The main documentation of this MVC application is not in paragraphs, it is in the form of an IDEF0 diagram. Readers of "Working with Active Server Pages" will probably recognize my favorite technique for modeling hierarchical, process-oriented systems.

For those unfamiliar with the formalism, input arrows enter from the left of activity boxes. Activity constraints enter from the top. Outputs exit the right side of the boxes, and upward arrows model mechanisms that enable the activity to proceed. All boxes must have at least one constraint, verbs label boxes, and nouns label arrows. Boxes can be decomposed fractally to expose refinements in activities. A strict conservation of arrows ensures no detail is lost during the drill down process.

My diagram was drawn using Visio 5 using their IDEF0 stencil. This feature was added after "Working with Active Server Pages" was published, but I'm sure that's just a coincidence. By the way, this diagram is hierarchical, but you cannot reach those lower levels when you access a Visio file from IE (unless i set it up for such a feat), so I encourage you to view the file in Visio, i you own a copy of that terrific graphics program.

• Keep your promises: In COM, promises are interfaces. Interfaces are contracts that permit clients and servers to continue to work together even as both change. COM is like marriage in that way.
• My MVC app makes and keeps it promises with its underlying XML schema. Any client document that wants to use the services of my MVC app merely needs to present its data in that schema.
• Since my MVC app is an implementation of key design patterns (that's the main reason MVC is a framework, not a Design Pattern), it can be reused and extended easily.
• Write once, use in many ways: In COM, this is polymorphism. At the machine level polymorphism works through the magic of abstact base classes and pure virtual functions. But in my MVC app, polymorphism isn't that exotic.

• I set out to write an essay about Design and saved the document as (eventually well-formed) XHTML.
• The document went on to show you how to use XML to extend the functionality of the document itself.
• The essay included a discussion of MVC applications that can use the XML that extends the traditional document.
• The essay documented an MVC application that displayed the original document in a new way, relying only on the XML added to the original document.
• In other words, I gave you a document that showed you how to use XML and MVC to create documents, and the document I created used the contents of the document I created to...create itself.
• So, to create your own MVC application, study this app and the document that describes it; they're all the same thing...

Future Research

This application of MVC was primarily a client-side demonstration. The other MVC app I showed you briefly was a bit more server-dependent, but still far from the limits of applicability MVC and XML can have on the server.

• Componentize this MVC framework using Windows Scripting Components.
• An MVC framework using VB on the server for down-level browser support.
• Using XML in Application scope as the Model in a client-based MVC application.
• And most interesting to me, an MVC framework that permits web clients to be Views on a server-side MVC application. This requires a UDP-based messaging system for server/client event notification.