Washington, September 15-18, 1999 – London, November 21-24, 1999

Three Approaches to Web Data Retrieval

Dino Esposito

Data retrieval implies access to some kind of data source, be it remote or local. In traditional Client/Server programming, you usually have a server module that connects to a RDBMS, submits its query or runs the needed stored-procedure. When it receives the data, it returns it to the client. This module, in the meantime, is keeping its user interface unchanged, waiting for the data to arrive. This same simple schema doesn’t apply to Web-based applications just because the Web is based on the HTTP protocol, which is stateless. To put it another way, the request made by the client and the response from the server are two independent connections that nothing know about each other. The most typical way to get the result of a query today is requesting an ASP page to collect the data and return HTML code back to the browser. With a bit of coding, you can get the page before and the page after the data retrieval to look nearly identical (with, of course, the exception of the data displayed!) This scenario is destined to change over time due to three technologies that let you do data retrieval across the Web applying the old Client/Server interaction: the client asks and waits, the server works and returns.

Gone are the days when you necessarily needed to leave the current page in order to retrieve data and display that to users. Today three somewhat related technologies let you connect to remote data sources, invoke methods, and obtain recordsets, XML strings and so forth, directly on the client as a local call. What remains to be done is sorting the data and updating the page using Dynamic HTML. The three options are: Remote Scripting, DHTML data binding, and Remote Data Service (RDS). They are different both for syntax and architecture. I'll compare them and feature their differences.

Dino Esposito Dino (desposito@artis.net) is a trainer and consultant based in Rome, Italy. He specializes in Windows and scripting and authored Visual C++ Windows Shell Programming and Windows Scripting Host Programmer's Reference both from Wrox Press. Dino is also a contributing editor to Microsoft Internet Developer, MSJ and MSDN News. He's the cofounder of www.vb2themax.com, a VB-oriented Web site and works for Artis srl, a Rome-based consulting company active in the system integration area and Web-based architectures.

Remote Scripting

Remote Scripting (RS) is a small library available in source code that lets you export functions from within ASP pages. RS comes with Visual InterDev 6.0 and is available as a separate add-on from the Microsoft's scripting site. Using its functions, you can issue both synchronous and asynchronous calls to any ASP page that exports its functions publicly. These calls could be made from a client-side HTML page as well as a server-side ASP page. If you are familiar with Visual InterDev 6.0, then bear in mind that RS is just the underpinning behind the PageObject object.

The archictecture is based on three components: a client proxy, a server stub and a connection module that acts as the transportation layer between the communicating pages. To enable RS in your applications you need to import in sources the client proxy (which is a rs.htm file) in the page that needs to start a remote connection. The server page must import the stub portion of RS (a file called rs.asp). Both client and server pages must change their structure slightly to accommodate RS - in particular, the client must call the RS initialization function (RSEnableRemoteScripting()), whereas the server needs to put a call to the RS dispatcher function at the top of it. This function works as a switcher, analyzing the input parameters and invoking the requested ASP function. A Java applet ensures communication, without causing the calling page to be replaced. You don’t have to worry about it as it is completely transparent to your RS-based applications. It just spookily slips in your code when you initialize the RS engine. The proxy and the stub modules take care of dialoging with it, shielding your higher-level code from these details.

Invoking a remote method is as easy as calling a RSExecute() function specifying the URL to the ASP page, the name of the function, and any parameter it may require. What you get returned is a composed structure that includes fields for the actual return value as well as error messages. If you want the call to be asynchronous, then specify the client side function that the RS proxy will call when it gets back from the stub.

The typical startup of a RS-enabled page:

<BODY>

<script language="JScript"

        src="http://expoware/_ScriptLibrary/rs.htm">

</script>

<script language="JScript">

        RSEnableRemoteScripting("http://expoware/_ScriptLibrary");

</script>

The typical call to a remote page method:

function execSynch() {

   var co = RSExecute(serverURL + pageURL, "GetEmpInfoAsArray",

                      frm.empName1.value);

   refreshPage(co);

}

The remote ASP page must expose its functions through an object with a precise name: public_description. Here's a typical table of functions:

function Description() {

    this.GetEmpInfoAsArray = DoGetData;

}

public_description = new Description();

Notice that you can use a JScript object as well as a VBScript 5.0 class. What matters is that the name of the object must be public_description. Any function which is not defined as a method of this object won't be callable through RS.

Other significant features of RS are:

Ø        The possibility of issuing asynchronous calls

Ø        The possibility of getting a local copy of the server object

The former approach ends up being really useful if you use it for validation purposes. Instead of sending the whole form to the server for validation, you can check each field asynchronously while your users are filling the rest of the form. The call to RS returns quickly despite the speed of the connection and the validation takes place in the meantime.

An asynchronous call differs for an additional parameters being the function to call back.

function execAsynch() {

   RSExecute(serverURL + pageURL, "GetEmpInfoAsArray",

             frm.empName1.value, RefreshPage);

}

function RefreshPage(co)

{

…

}

Mirroring Remote Objects

RS is a function-based technology, whereas a more object-based syntax is welcome and somewhat more natural in the land of scripting. To work around this, you can ask the RS engine to return a local copy of the public_description object. Any call you make will still traverse the network. Instead of repeatedly calling a function like RSExecute, you call RSGetAspObject() and then use the object reference returned from the function to issue calls following a more natural syntax. The call is forwarded the same way as RSExecute, so from this point of view there are no particular differences in performance. Using local mirrors, you pay the price of an additional call to the server (RSGetAspObject) but gain a simpler syntax. Furthermore, with RSGetAspObject you can't issue asynchronous calls.

DHTML Data Binding

Supported by IE 4.0 onwards, data binding (DB) is the Web counterpart of Visual Basic's data bound controls. Data binding lets you bind a data source with one or more HTML tags on a page. DB has nothing to do with ASP. It works on the client by design and is strictly tied to Dynamic HTML. Basically, while you're designing the body of an HTML page you could decide to establish a direct link between one tag and a field of remote database. In this way, at display time it's up to the browser (IE) to download the value and fill properly the innerHTML or innerText field of the tag. (The particular property that gets filled depends upon the presence of an additional attribute, DATAFORMATAS, which can take two values: text and html.)

With DB you don’t have to take care of establishing the connection yourself or retrieving and marshaling a recordset. Everything takes place automatically: you have just to make sure that you comply with the required syntax. The loading of the data is asynchronous, and while this is occurring the rest of the page continues to download.

To enable DB in your pages, first of all include the following object:

<OBJECT style=display:none id=rds

        classid="clsid:BD96C556-65A3-11D0-983A-00C04FC29E33">

</OBJECT>

It corresponds to the RDS.DataControl object, which is part of the Remote Data Services 2.0. However, you need to include it as an ActiveX control since you need to use its DHTML ID. Feel free to give it any name other than rds.

Once you've done this, just decide which elements in your page you want to link to a datasource. You have to specify either a table or a field. For example:

<SPAN DATASRC=#rds DATAFLD="employeeID"></SPAN>

At the level of the HTML tag, the data source is just the ID of the DataControl object. (Remember, though, to prefix it with a #.) The DataControl should point to a recordset that is identified by a HTTP server name, a connection string and a command. Notice that you should use the same identical strings and commands you would use with ADO from, say, an ASP page or a Visual Basic program. Interestingly, this means that you could use any OLE DB provider through data-binding!

To specify the OLE DB provider to ask for the data, use the properties of the DataControl object:

function getData() {

   sql = "select * from Employees where lastname='";

   rds.Server = "http://expoware";  

   rds.Connect = "DSN=NW";  

   rds.Sql = sql + frm.empName.value + "'";

   rds.Refresh();

}

An even more interesting feature has been implemented for the <TABLE> tag: you can’t specify a DATAFLD for the tag, but just a DATASRC. In return, though, you specify only the DATAFLD for its <TD> and have the DB engine to take care of filling automatically the table adding as many rows as needed! You can use DB without managing the recordset yourself. However, by hooking the ondatasetcomplete event of the DataControl object, once you get your data you can format the table dynamically adding as many columns as needed. (By default, in fact, the number of the columns is fixed at design time and the engine doesn’t provide for them.) The recordset is cached and could be even sorted or filtered having the table to redraw automatically.

Remote Data Services

RDS (which includes the DataControl object seen above) also includes a couple of other really useful controls that provides the third way to get data through the Web without leaving the current page. In my opinion, RDS is probably the cleanest solution as well as the most elegant of all. The major characteristic is that it's based on COM/DCOM. Despite the complexity of the underlying machinery, the idea behind is really simple: RDS lets you connect via HTTP to a COM object and work with it by calling methods and properties with the same (apparent) programming comfort as it was a local one.

With RDS you can call objects either via HTTP and DCOM with the unique prerequisite that the object is properly registered on the server if HTTP, and is enabled for DCOM communication in case you want to go through DCOM over a network.

To call remote objects you just need to create locally to your calling HTML (or ASP) page an instance of the DataSpace object:

rds = new ActiveXObject("RDS.DataSpace");

This object provides the necessary infrastructure to create and manage remote objects. Your own remote objects can be created this way:

dataf = rds.CreateObject("RDSServer.DataFactory", "http://expoware");

rst = dataf.Query("NW", sql);

You specify the HTTP server and the progID of the object. When an object is created through the DataSpace.CreateObject() method a proxy layer on the client packages the COM calls into an HTTP request and passes it to the corresponding stub on the server (IIS or PWS). The stub receives the request and takes care of actually instantiating the COM object. It executes the call and passes back the results to the proxy through an HTTP response.

In the code snippet above I'm using a Microsoft-provided component that comes with MDAC 2.0, and that is a simple wrapper for a query-maker component. It exposes simple methods to get a recordset from a query or to create a custom recordset. Executing the two lines from within a client HTML page causes your script to receive a recordset as the result of the execution of the specified query on the server. The recordset is cached locally and any other operation you perform on it after this will not require further access to the server. It definitely reduces the traffic over the network. You can call any automation COM object through RDS and, of course, you can write your own object to be used with it. This makes things very flexible since you could preprocess data on the server and return a recordset that is ready for display. For example, you might want to incorporate a few fields in a more friendly displayable format (say, concatenating first and last name.)

Since RDS works over HTTP, which is a stateless protocol, a new instance of the object is created for any call you make to that object. You never have a local instance of a remote object, but just a pretty standard proxy/stub layer that sits in the middle and hides the location of the object. This has a concrete impact on how you should design object to be used with RDS. Your object must be stateless as well, since any internal state is lost between two successive calls.

An Overall View

We have three ways to access data through the Web without navigating to a new ASP page. All have pros and cons. I concentrated exclusively on the data retrieval aspect and neglected the data update. In the following sections, I summarize what's good and bad with any approach with the common scenario in which you might want to take advantage of it.

Remote Scripting

Low-level approach, requires to adapt existing code, based on Java and JScript. Allows asynchronous calls. It came out before RDS and is going to be superceded by it. You can use it to share code between ASP pages. Doesn’t require any COM knowledge, but consider also WSC to get reusability through script. It is the only approach that may work outside IE 4.0 and higher. (So far, though, I've been unable to get it to work with Netscape Communicator.)

Data Binding

Web counterpart of Visual Basic's data-aware controls lets you plug data directly in the page tags with no effort and no contact with data management structures such as recordsets and connections. It inherits the good and the bad of data-bound elements: easy to use but not always what you need in real-world scenarios. You can reach a good compromise between flexibility and ease-of-use if you encapsulate data-binding in a DHTML scriptlet that formats the view adapting it to the recordset. It also provides a unique feature: the capability of automatically handling raw text as well as HTML data. Should this capability extend to other MIME types, we would have a great way to import images directly from databases.

Remote Data Services

Based on COM it has no impact on existing code (differently from RS) and gives you the possibility to isolate in a separate component all the logic that you need to apply to the raw recordset returned by a query. In this sense, it is better than DB. The same ultimate power, but far easier to code from the HTML point of view. It looks like RS since it lets you issue direct calls to a remote object, but now the object is a COM object (also a WSC object is fine). It works with any browser that supports script with COM capability.

In general, try to use RDS wherever is possible. RDS is the newest of these technologies and probably the only one with a certain future. RDS also encompasses the other two as, after all, DB is part of RDS. Consider using RS but through the VID PageObject object (that hides much of the difficulty) and from within VID projects. Also consider it to get remote connection to ASP objects also from IE 3.x and Netscape. (Microsoft claims this is possible, but I can't reproduce it, not even with using their RS kit examples.)