A Resusable Web Architecture

There are as many reusable web architectures championed by their inventors and proponents as there are technologies available to an Enterprise development team. This series of articles proposes, demonstrates and validates one such architecture based on the Microsoft Windows DNA ^TM 3-tier framework and the Rational Unified Process modeling methodology. We begin with the use of UML in modeling use cases and distilling the object and data models. We then outline some Active Server page proto-patterns or templates, followed by an extensive and detailed look at some COM design patterns. We then lead into a discussion of database modeling techniques. At several key waypoints in this series, we stress and highlight the twin virtues of scalability and performance as being the ultimate test of the success and validity of any web architecture. To prove the re-usability of these patterns and methodologies, most of the source code has been stripped of almost all business context in order to better highlight their framework .

It is assumed that the reader has some familiarity with Active Server Pages, Javascript, COM, Visual Basic and SQL. A working knowledge of IIS, MTS and any SQL database platform such as MS SQLServer, Oracle or DB2 will be useful while implementing some of the solutions presented. Familiarity with RUP and UML is also assumed. All the scripting examples are presented in Javascript and all COM components have been written using Microsoft Visual Basic 6.0^TM

Table of Contents

· Modeling the Enterprise

o Planning the process through RUP

§ Describing the business charter

§ Assessing technological boundaries - the bleeding edge effect!

o Life-cycle tools through UML

§ Describing the Use Case

§ Static Model

§ Sequence Diagrams 

· The User Interface

o An Active Server Page proto-pattern.

o Reusable server-side scripts.

o Reusable client-side scripts.

o The XML/XSL Transform pattern. 

· Middle-Tier Strategies

o The Middle Tier & COM.

o The stateless design pattern.

o COM Design Pattern - I (implementation of the stateless pattern).

o Data Access Review.

§ Overview of OLEDB/ADO/RDS.

§ ADO and ASP

§ RDS and ASP

o COM Design Pattern - II (extending COM with ADO).

§ Basic ADO and the data tier

§ Enhancing the COM Template for data retrieval/update

o COM Design Pattern - III (The C.R.U.D. Pattern).

o COM Design Pattern - IV (C.R.U.D. on hierarchical data)

§ Using Shaped Recordsets for retrieval and updates

§ Custom Recordsets

· Reusable Security Models

o COM & LDAP

· Effective Data Tiers.

o Spreadsheet Origami (A reusable data-cube pattern)

· Scalability and Performance Review.

COM Design Pattern - I

Using a Visual Basic class module we demonstrate a physical implementation of the stateless design pattern we expounded upon in the previous section. An understanding of the Visual Basic programming language is required for this section. For a 30 second Intro to Visual Basic click here.

Anatomy of a Stateless Design

Using your Visual Basic IDE, use the following steps to create a ActiveX DLL project

· Click on the File..New project menu and click on the ActiveX DLL option. See accompanying image

· Your IDE will create a class module file which is going to be the home for your stateless object definition. See accompanying image

· Since we are going to expand this template later to include database retrieval/updates ina transactional environment, we will also include references to ADO and MTS. We will cover both these topics in detail later on in the show. You can access these properties by going to the Project..References menu in the IDE See accompanying image

The sample code below describes the structure of any Stateless object design

 
Option Explicit
 
'
'The Public keyword identifies this as an object 
'interface that can be used by an application
'
'-----------------------------------------------------
Public Function BuyCar(ByVal IsAdult As Boolean, _
 ByVal Name As String, _
 ByVal Address As String, _
 ByVal CarMake As String, _
 ByVal CarModel As String, _
 ByVal CarYear As String, _
  ByVal CreditCardNumber As Long, _
 ByVal BankId As Long, _
 ByVal DriversLicenseId As String) As Boolean
 
 
On Error GoTo Err_Handler
 
 '
 'conduct business logic here..
 '------------------------------------
 
 BuyCar = True
 Exit Function 'if everything went well
 
Err_Handler:
 '
 'conduct error-handling here
 '---------------------------------
 
 BuyCar = False
 
End Function

Stateless Specifics

What this implementation demonstrates is the framework of any stateless COM component independant of the business context. The framework has certain key components

· The class module comprises of one or more method calls and no public attributes (properties). This is really what stateless means in real terms. Stateless components perform actions only.

· Each method call is implemented as a function in order to provide a return value. In Visual Basic^TM, possible return values range from basic data types (such as strings and numbers) to objects (other COM components) and arrays.

· Each method call must have built-in error handling and garbage collection.

· Garbage collection is necessary after both successful and failed method invocations.

Programming Commandments

· Always use ByVal to declare the method parameters if you can. This will send the parameters in by value, not by reference.

o ByRef is the default in VB6, and is a hold-over from the days when applications were always desktop applications, and passing "Pointers" to the data was more efficient. Unfortunately, ByRef requires a network trip from client to server and one more back again to the client to get the pointer to the variable.

o Using ByVal, we make a copy of the variable, and eliminate an unnecessary network trip. In a distributed environment, this boosts scalability by an enormous amount 

· Make sure your threading model is "Apartment Threaded". This is the model required for a COM component in VB6 to run concurrently under MTS.

· We also need to set the following properties of the Project as well in order to run these components smoothly in MTS (Microsoft Transaction Server) See accompanying image You can get to this menu by clicking on Project..[Project Name] properties through the IDE

o The two options 'Unattended Execution' and 'Retained in Memory' must be checked on for an ActiveX DLL project to run successfully in MTS.

o Unattended Execution will treat any MessageBoxes in you DLL as NT events which you can view through the Event Log Viewer. This is an invaluable debugging tool.

o Retained in memory will prevent your DLL from being released unexpectedly. (This is a little known Microsoft feature/bug that can cause headaches is a component is used frequently by an application)

· Do not use Global variables in your classes. Global variables are stored on a per-thread basis, and when you use a middleware tool like MTS to host your components, there is no guarantee that your component will get the expected value (unless they accidently happen to share the same thread).

There will be more programming commandments as we get into database access using COM objects, which is the next object of our attention

Where can I get this ?

The sample code for this COM object along with the other class files used in later articles as well can be found by downloading this Zip file SampleCOM01.vbp After unzipping into a direcotry of choice open up the project by clicking on the Sample.vbp file. You must have Visual Basic 6.0 installed on your machine for this to work.

COM Design Pattern - Part II

We now do continue our autopsy of the Visual Basic implementation of our stateless design pattern.
A recurring design problem in most Web applications is data retrieval and updates from one or more data stores. This stateless object model is ideally suited to do this.

Anatomy of a Stateless Design

The sample code below describes the structure of our enhanced Design pattern implementation.

 
Public Function BuyCar(ByVal IsAdult As Boolean, _
 ByVal Name As String, _
  ByVal Address As String, _
 ByVal CarMake As String, _
 ByVal CarModel As String, _
 ByVal CarYear As String, _
 ByVal CreditCardNumber As Long, _
 ByVal BankId As Long, _
 ByVal DriversLicenseId As String) As Boolean
 
 
On Error GoTo Err_Handler
 '
 'set up the database connectivity here
 Dim sDSN As String
 Dim oConn As ADODB.Connection
 
 '
 'a typical connection string to a database
 'the DSN is the ODBC data ssource name on the
 'server;
 'the UID and PWD are the user-id and password
 'strings.
 '
 'this connection string is often provider specific
 '------------------------------------------------------
 sDSN = "DSN=eSRP_Auto; UID=test;pwd=test;"
 '
 'open the connection
 Set oConn = New ADODB.Connection
 oConn.Open sDSN
 '
 'conduct business logic here..
 '------------------------------------
 oConn.Execute _
 "INSERT INTO CAR_TABLE " & _
 "(Make, Model, NewOwner) " & _
 "VALUES('" & _
 CarMake & "','" & _
 CarModel & "', '" & _
 Name & "')"
 
 oConn.Close
 Set oConn = Nothing
 'complete the transaction
 '-----------------------------
 GetObjectContext.SetComplete
 
 'return a success flag
 '---------------------------
 BuyCar = True
 Exit Function 'if everything went well
 
Err_Handler:
 
 '
 'Error handling code here
  '---------------------------
 
 '
 'always close connections
 If oConn.State = adStateOpen Then oConn.Close
 
 'abort the transaction
 GetObjectContext.SetAbort
 
 'return a false code to the application
 BuyCar = False
End Function
 

Programming Commandments

· Use Early Binding wherever possible.

o Early binding is declaring the object type explicitly. Dim oConn as ADODB.Connection

o Late binding is declaring the object type as in Dim oConn as Object. Using late-binding results in two method calls, one to get th ID of the method, and the second to actually make the call.

· Aquire resources late, release them early. This is a great performance saving tip. note how the database connection is opened right before the SQL query, and is terminated immediately after, and all expensive database resources are releases shortly thereafter.

· In VB, you can often save a line of code by writing Dim oConn as New ADODB.Connection.

o This will cause every reference to the object to be slower. This is because we are telling VB to create an object instance for us whenever we need one, and it does this check every time an object variable is used. 

o Also prevents you from using the GetObjectContext.CreateInstance() method to create objects within MTS.

Got a Clue about Transactions

In the previous examples, you saw code that involved the GetObjectContext object, and are probably wondering what Transactions are, what they mean to us, do we need them, how do we use them, and a myriad of other perfectly reasonable questions. Let's try and answer them as simply as possible

What are Transactions ? 

A transaction is one unit of work, defined as an All or Nothing approach to the work unit. A transaction consists of three pieces

· Creating a Transaction (the transaction context)

· Committing the work within a transaction

· Aborting and rolling back all the work within a transaction on error.

Why use transactions ?

For precisely the reasons outlined above. It lets us treat a unit of work (which may in reality be comprised of multiple retrievals and updates from and to the database) as an indivisible unit. This provides for incredible safetly in modeling business processes.

· For example, if the BuyCar() method needed to withdraw money from the Person's checking account and in the next step credit it to the Dealers bank account, these two steps need to be treated as one.

· If you successfully withdraw the money from one account but fail to deposit it in the other, the entire unit of work should be treated as a failure. In this example, the withdrawl from the Person's account would be nullified, and no money would change hands..!

How to use Transactions ?

· Creating a Transaction: Easy, all you have to do is to go to the properties of a class, and set the MTS Transaction mode to either Supports Transactions or Requires Transactions . See supporting image here.

o Supports Transactions will allow methods in the class to participate in transactions created by other classes

o Requires Transactions will create a transaction if a method in the class is invoked without any transactions as well as let it participate in already created transactions.

· Committing the work: Even easier, just write GetObjectContext.SetComplete just before exiting the unit of work, and MTS will commit the transaction for you. it will manage any Database provider transactions it may have created in the background for you.

· Roll-back: Typically, rollback occurs on an error. So always include error handling and make the last line of code in your method GetObjectContext.SetAbort

Now we enhance our understanding of a COM/MTS design by looking at a new design pattern. This is the C.R.U.D. design pattern

COM Design Pattern - Part III

In part III of our COM design pattern marathon, we now investigate a kind of design pattern unique to the Stateless object world. We call it C.R.U.D., which stands for the very boring Create Read, Update, Delete pattern. The reasoning behind this pattern is that these four operations are a recurring design problem. Most Web applications will at some point in time, Create database rows, Retrieve them, Update them and finally Delete them as well.

Anatomy of a Stateless Design

The sample code below describes the structure of our enhanced Design pattern implementation. First we talk about the Read method since it is the most interesting, followed by the Update method. The Create and Read methods are available in the Sample application

The Read method

 
Option Explicit
 
Public Function getCarDetail( _
 ByVal VinNumber As String) _
As ADODB.Recordset
 
On Error GoTo Err_Handler
 '
 'set up the database connectivity here
 Dim sDSN As String
 Dim oRst As ADODB.Recordset
 
 sDSN = "DSN=eSRP_Auto; UID=test;pwd=test;"
 
 Set oRst = New ADODB.Recordset
 '
 'create a client-side recordset
 oRst.CursorLocation = adUseClient
 
 '
 'conduct business logic here..
 '
 'using the DSN string with the recordset object
 'you can have ADO open up a connection in the
 'background
 '------------------------------------
 oRst.Open _
 "SELECT * FROM CAR_TABLE WHERE VIN_NUM = '" & _
 VinNumber & "'", _
 sDSN, _
 adOpenStatic, _
 adLockOptimistic, _
 adCmdText
 
 'disconnect the recordset
 Set oRst.ActiveConnection = Nothing
 
 'set the return vlue to the recordset
 '--------------------------------------
 Set getCarDetail = oRst
 
 '
 'garbage collection
 '-----------------------
 oRst.Close
 Set oRst = Nothing
 
 'complete the transaction
 '-----------------------------
 GetObjectContext.SetComplete
 
 Exit Function 'if everything went well
 
Err_Handler:
 
 '
 'Error handling code here
 '---------------------------
 
 '
 'always close connections
 If oRst.State = adStateOpen Then oRst.Close
 
 ' Create a custom recordset with the error 
 ' message as the only field. Must create a 
 'Static cursor to be able to add fields and rows
 'to a client-side recordset
 '-----------------------------------------------
 oRst.CursorLocation = adUseClient
 oRst.Fields.Append "ERROR_MSG", adVarChar, 255
 oRst.Open
 oRst.AddNew
 oRst.Fields("ERROR_MSG").Value = _
 "An error occurred: " & Err.Description
 oRst.Update
 
 Set getCarDetail = oRst
 
 'abort the transaction
 GetObjectContext.SetAbort
 
 'return a false code to the application
 
End Function
 

Programming Commandments

· Think about when you need transactions, and when you simply need to support them.

o In the above example, we created a new class called getCar, which has an MTS transaction mode of "Supports Transactions". We do this because a retrieval from a database does not require a transaction on it's own. However, the class may be required to participate in a transaction, so we leave that design door open for future developers to exploit.

· In this design pattern, data is returned as an ADO recordset

o Advantages:

§ A rowset allows for the most flexible treatment of data, data can be sorted, filtered easily after being sent to the client.

§ A rowset contains meta-data, such as field type, size, etc.. this is often very useful information for the UI if it needs to validate user updates to this recordset

o Disadvantages:

§ Recordset size can be often be an issue when costing out bandwidth requirements. it is usually offset by the fact that recordsets can be used to send back more data in one network call.

· The recordset being returned is a disconnected recordset. Disconnected recordsets provide a huge scalability and performance advantage to the Web application.

o We do that by two lines of code

§ loRs.CursorLocation = adUseClient. This tells the COM component that the cursor associated with the rowset is the responsibility of the client and not the database server.

§ Set loRs.ActiveConnection = nothing. This line of code severs the connection of the recordset from the database, and makes it a disconnected recordset.

o We specify the cursorType = adOpenStatic, because we would like to be able to update this recordset when it comes back to the COM component through the application interface.

o We specify the lock type = adLockOptimistic. This allows the database to lock the rows record by record, only doing so when the Update method is called.

· Note another advantage of telling the component that we want a disconnected recordset. If there is an error during the retireval of data, we use a custom recordset  to send back error messages to the client.

o We do this by first closing the recordset if it ever got opened

o Then we append a field to this empty recordset, call it ERROR_MSG.

o This field is set to a datatype of VARCHAR(255)

o A new row is added to this empty custom rcordset and the value of the field is the complete error message that was returned from the database

o This error message is richer and more complete, and helps the client figure out what to do if there was an error.

o

The Update method

In the Update method, we see how to use a design pattern where a disconnected recordset is passed into the Save method and returns itself back to the client application

 
Option Explicit
 
Public Function saveCarDetail( _
 ByRef ThisRs As ADODB.Recordset) _
As ADODB.Recordset
 
On Error GoTo Err_Handler
 '
 'set up the database connectivity here
 Dim sDSN As String
 Dim oRst As ADODB.Recordset, oConn As ADODB.Connection
 
 sDSN = "DSN=eSRP_Auto; UID=test;pwd=test;"
 '
 'open the connection
 oConn.CursorLocation = adUseClient
 
 Set oRst = New ADODB.Recordset
 oConn.Open sDSN
 '
 'conduct business logic here..
 '------------------------------------
 oRst.Open ThisRs, oConn
 oRst.UpdateBatch
 
 'set the return vlue to the recordset
 '--------------------------------------
 Set saveCarDetail = oRst
 
 '
 'garbage collection
 '-----------------------
 oRst.Close
 Set oRst = Nothing
 
 'complete the transaction
 '-----------------------------
 GetObjectContext.SetComplete
 
 Exit Function 'if everything went well
 
Err_Handler:
 
 '
 'Error handling code here
 '---------------------------
 
 '
 'always close connections
 If oRst.State = adStateOpen Then oRst.Close
 
 ' Create a custom recordset with the error 
 ' message as the only field
 '--------------------------------------------
 oRst.CursorLocation = adUseClient
 oRst.Fields.Append "ERROR_MSG", adVarChar, 255
 oRst.Open
 oRst.AddNew
 oRst.Fields("ERROR_MSG").Value = _
 "An error occurred: " & Err.Description
 oRst.Update
 
 Set saveCarDetail = oRst
 
 'abort the transaction
 GetObjectContext.SetAbort
 
 'return a failure code to the application
 
End Function

Programming Commandments

·  In the above example, we created a new class called updCar, which has an MTS transaction mode of "Requires Transactions". We do this because an update to a database requires a transaction on it's own.

· Note that in this example, the recordset is actually sent back ByRef. This is because we want to work with the actual recordset and not a copy of the recordset. The increase in size associated with the addition of this meta-data information is compensated for by the fact that the returned information is compressed into one single recordset object.

Now we move on to PartIV of our discussion of COM design patterns, where we will talk about the final enhancement to the CRUD design pattern as applied to hierarchical data sets.

COM Design Pattern - Part IV

In part IV of our COM design pattern marathon, we now investigate an enhancement to the CRUD design pattern we talked about in the earlier section. Much of enterprise data is hierarchical, in that they obey parent-child relationship hierarchies. An example shown below extends the simple example of relationships between people, their cars and computers under a typical organizational umbrella.

This model describes an organizational hierarchy. The top-level parent is the Company which is comprised of divisions. Each division is simplistically modeled as a collection of Employees. Employees have a sematic relationship with cars as they are required to operate one or more company vehicles. Often, it is necessary to traverse this hierarchical tree in several ways

The Traditional approach

In the traditional approach, if we wish to traverse from Company to Cars, we would have to use several (inefficient) SQL Outer Joins. Joins have one inherent flaw, they return too much data. A Join has no other representation that one rowset. To traverse the structure described above, it would be necessary for the query to provide a vey large denormalized rowset at the level of individual cars, with duplicate parent information at all of the four levels this hierarchy models.

The Shaped approach

The Shaped approach utilizes a provider independant feature of ADO called Data Shaping. Data Shaping allows the creation of nested hierarchical data sets that retuen data in a Parent-Child format. The shaped syntax is quite simple and elegant.

Using the above structure as an example, we can write a shaped query to retrieve all divisions under one or more companies. We assume that the data structures have a one-to-one mapping to this data model. We must also assume that each object in the database is related to it's parent with a foreign key that is part of the primary key of the parent. In the absence of this definining relationship, the Shaped Provider may behave unpredictably.

 
SHAPE { SELECT * FROM Company }
 APPEND ( { SELECT * FROM Division } 
 RELATE CompanyId TO CompanyId) 
 AS chDiv

We can nest a grandchild inside the child recordset and find out which employees are in each Division.

  
SHAPE {SELECT * FROM Company } 
 APPEND( ( SHAPE {SELECT * FROM Division }
 APPEND ( {SELECT * FROM Employee } 
 RELATE DivisionId TO DivisionId ) AS chEmp ) 
 RELATE CompanyId TO CompanyId ) AS chDiv 

Anatomy of a Shaped Design

The sample code below describes the structure of our enhanced Shaped Design pattern implementation. First we talk about the Read method followed by the Update method. The Create and Read methods are available in the Sample application

 
Option Explicit
 
Public Function getShapedCarDetail() _
As ADODB.Recordset
 
On Error GoTo Err_Handler
 '
 'set up the database connectivity here
 Dim sDSN As String
 Dim oRst As ADODB.Recordset, oConn As ADODB.Connection
 
 sDSN = "DSN=eSRP_Auto; UID=test;pwd=test;"
 '
 'open the connection
 set oConn = new ADODB.Connection
 oConn.Open sDSN
 
 '
 'set some connection properties: client side recordset cursors
 'and set the data shape provider
 '--------------------------------------------------------
 oConn.CursorLocation = adUseClient
 oConn.Provider = "MSDataShape"
 
 Set oRst = New ADODB.Recordset
 '
 'conduct business logic here..
 '------------------------------------
 oRst.Open _
 "SHAPE{SELECT * FROM Employee } " & _
  "APPEND ( {SELECT * FROM Car} " & _
 "RELATE EmployeeId to EmployeeId ) AS OPERATOR_OF " & _
 VinNumber & "'", _
 oConn, _
 adOpenStatic, _
 adLockOptimistic, _
 adCmdText
 
 'disconnect the recordset
 Set oRst.ActiveConnection = Nothing
 
 'set the return vlue to the recordset
 '--------------------------------------
 Set getShapedCarDetail = oRst
 
 '
 'garbage collection
 '-----------------------
 oConn.Close
 
 'complete the transaction
 '-----------------------------
 GetObjectContext.SetComplete
 
 Exit Function 'if everything went well
 
Err_Handler:
 
 '
 'Error handling code here
 '---------------------------
 
 '
 'always close connections
 If oRst.State = adStateOpen Then oRst.Close
 
 ' Create a custom recordset with the error message as the only field
 '--------------------------------------------------------------------
  oRst.CursorLocation = adUseClient
 oRst.Fields.Append "ERROR_MSG", adVarChar, 255
 oRst.Open
 oRst.AddNew
 oRst.Fields("ERROR_MSG").Value = _
 "An error occurred: " & Err.Description
 oRst.Update
 
 Set getShapedCarDetail = oRst
 
 'abort the transaction
 GetObjectContext.SetAbort
 
 'return a false code to the application
 
End Function

Data Services WorkFlow - I

· Notice that the connections Data Provider has been set to MSDataShape

· Everything else about this method is the same as the earlier design pattern implementations.

Next, we discuss how to update a shaped recordset through the COM object

 
Public Function updateShapedCarDetail( _
 shapeRs As ADODB.Recordset) _
As ADODB.Recordset
 
On Error GoTo Err_Handler
 '
 'set up the database connectivity here
 Dim sDSN As String
 Dim oRst As ADODB.Recordset
 Dim oRstChild As ADODB.Recordset
 Dim oField As ADODB.Field
 
 Dim oConn As ADODB.Connection
 
 sDSN = "DSN=eSRP_Auto; UID=test;pwd=test;"
 '
 'open the connection
 oConn.CursorLocation = adUseClient
 oConn.Provider = "MSDataShape"
 
 Set oRst = New ADODB.Recordset
 Set oRstChild = New ADODB.Recordset
 set oConn = new ADODB.Connection
  
 oConn.Open sDSN
 '
 'first update every child recordset
 '------------------------------------
 For Each oField In shapeRs.Fields
 If oField.Type = adChapter Then
 Set oRstChild = oField.Value
 
 oRst.Open oRstChild, oConn
 oRst.UpdateBatch
 oRstChild.Close
 End If
 
 Next
 
 '
 'finally update the parent
 
 oRst.Open shapeRs, oConn
 oRst.UpdateBatch
 
 'disconnect the recordset
 Set oRst.ActiveConnection = Nothing
 
 'set the return vlue to the recordset
 '--------------------------------------
 Set updateShapedCarDetail = shapeRs
 
 '
 'garbage collection
 '-----------------------
 oRst.Close
 Set oRst = Nothing
 Set oRstChild = Nothing
 oConn.Close
 Set oConn = Nothing
 'complete the transaction
 '-----------------------------
 GetObjectContext.SetComplete
 
 Exit Function 'if everything went well
 
Err_Handler:
 
 '
 'Error handling code here
 '---------------------------
 
 '
 'always close connections
 If oRst.State = adStateOpen Then oRst.Close
 If oRstChild.State = adStateOpen Then Set oRstChild = Nothing
 If oConn.State = adStateOpen Then oConn.Close
 Set oConn = Nothing
 ' Create a custom recordset with the error message as the only field
 '--------------------------------------------------------------------
 oRst.CursorLocation = adUseClient
 oRst.Fields.Append "ERROR_MSG", adVarChar, 255
 oRst.Open
 oRst.AddNew
 oRst.Fields("ERROR_MSG").Value = _
 "An error occurred: " & Err.Description
 oRst.Update
 
 Set updateShapedCarDetail = oRst
 
 'abort the transaction
 GetObjectContext.SetAbort
 
 'return a error code to the application
 
End Function

Data Services Workflow - II