My colleague wrote an article for XML-J two years ago about an opportunity we had to solve our data management challenges with XML ("Managing Data Sources with XSLT," www.sys-con.com/story/?storyid=40411). The result of our work was our XML Data Services (XDS), an XML data access language and processing engine, which allowed us to quickly and easily manage the bi-directional transform between data sources and XML. A couple years have passed, and new challenges have arisen and been solved by leveraging our XML data access engine and Web services.

My primary focus over the last couple of years has been enabling legacy data sources and applications to provide visibility into the enterprise and enable business process orchestration. These systems include everything from large-scale Department of Defense databases to automotive engineering applications. In this article I present situations in which applying XML and Web service technologies in a data-oriented manner provided system interoperability and enhanced the ability of systems to more quickly adapt to changing requirements. I also explore the use of Web services as they apply to overcoming barriers within internal development and deployment environments. Finally, I present self-services, a concept to allow clients to get exactly the right information in exactly the right format on-demand. These concepts expose isolated enclaves of information as visible and accessible commodities for the benefit of the enterprise.

Web Service Basics
There are numerous excellent articles and online resources that define service-oriented architectures and Web services in great detail, so I will only briefly review them here to provide context. Web services are adopted for their use of platform-independent open standards enabling the interoperability of distributed disparate systems. Web services can then be deployed either independently or in a service-oriented architecture. Service-oriented architectures are composed similar to traditional distributed software architectures but with much more loosely coupled layers. Figure 1 depicts some typical layers we deploy in a service-oriented system.

Data-Oriented Service Basics
The examples presented in this article focus on the foundation layer in Figure 1, the data access service layer. A recent pattern in service development involves configuring an object-relational mediation framework to instantiate objects encapsulating data contained within a database. The instantiated objects are then marshaled to XML using an object-XML binding API for publication through the service interface. The object model has been relegated to little more than getters and setters for a single database table with no behavior linked to the objects. Our goal is to simplify the building and deployment of data-oriented services by focusing on the fundamentals, namely the source data, the XML Schema defined for publication, and the data access rules.

My first example examines the processes used to Web-enable a legacy Department of Defense database application. The specification called for the application to run in a Web portal environment. To achieve the separation of content from presentation, we published our data as XML using our XDS engine and used XSL transforms to generate our portal-compliant displays. An example use case is finding all personnel located in an area defined by a geographic bounding box expressed as parameters upperLeftLat, upperLeftLon, lowerRightLat, and lowerRightLon. Listing 1 is a snippet of an XDS mapping document implementing this use case (code listings are available at www.sys-con.com/xml/sourcec.cfm). It shows the binding of the results of an SQL query, encapsulated in the xdl: statement, and the XML output identified by the xdl:elements. The details of using the XDS grammar can be viewed in the XML-Journal article previously referenced.

This particular application was developed under a joint program for the Department of Defense so it experienced a high degree of visibility. The project was soon approached by developers of another program investigating the feasibility of our portal data being made available in a distributed services environment. The architecture already supported the generation of XML content, so the only thing we needed to provide was the service interface. We were asked to provide an RPC-style service exposing the parameters for the geographic bounding box and return the content as defined by our initial XML Schema. SOAP was the natural selection because of its lightweight XML-based messaging framework able to exchange RPC calls over a variety of underlying protocols. The abridged WSDL can be viewed in Listing 2.

We implemented the SOAP Web service using the Apache Axis toolkit, an implementation of the SOAP submission to the W3C [Axis]. The method shown in Listing 3 shows how we develop Web services using Axis to proxy requests to the XDS engine.

There has been extensive discussion regarding the definition of Web services and what constitutes a valid service. These discussions have given rise to a new model for Web services called Representational State Transfer (REST) Web services. REST Web services focus on the basic underpinnings of the Web in general, namely URIs, HTTP, and XML. In order to be REST compliant we need to define a simple URI to identify our XML resource, which can be accessed via basic HTTP requests. In our case we define a JavaServer Page (JSP) endpoint with a set of query string parameters, e.g., http://host/personnel.jsp?upperLeft Lat=Y1&upperLeftLon=X1&lowerRightLat=Y2&lowerRightLon=X2.

The JSP implementation, shown in Listing 4, consists of using the XDS Tag Libraries to reference the mapping document and pass parameters to that mapping document. These simple tag libraries activate the XDS engine, execute the mapping and return the XML results through the JSP response output stream.

Figure 2 shows that by simply providing an Axis wrapper or a JSP endpoint we are able to quickly provide our XML- enabled data in both SOAP- and REST- compliant Web services. The isolation of XML generation and ease of service implementation demonstrates the ability of data-oriented services to adapt to changing requirements. These same principles also apply for new content creation and addition of capability. This is an example of how we successfully deployed our service, but bear in mind that there are a number of data-enabling technologies and data-oriented services do not only apply to databases.

Beyond the Basics of Data-Oriented Services
My second example involves a system that requires a trusted client to access data stored in a database that resides on the opposite side of a firewall between untrusting networks. The current solution to the firewalled database problem requires a client to remotely log into a machine within the trusted network, create a query file identifying the stored procedure to execute, and save this file to their home directory. The database maintainers have a process that runs periodically that will pick up the query file from the client's home directory, execute the specified query, and create a result file. The client will then copy the result file back to their working environment in order to process the results.

There are several problems with this solution, including stale database results, human in the loop, and additional work to handle the proprietary input and output formats. This situation is exactly what web services were designed for. Web services provide on-demand information in a standardized format allowing for machine-to-machine communications.

Based on my first example, it is straightforward to prepare XDS maps for the stored procedures and publish the results as XML in either SOAP or REST services. The clients and their applications are now able to access the stored procedures on-demand and have the results tunneled through the firewall in a standardized format.

The example database I am speaking of is not atypical in that it is quite large, often requiring very complex SQL, and full of special conditions. The demands on the database maintainers are becoming unmanageable, so much so that they are investigating ways to allow the clients ad-hoc query access. The proposed solution to empowering clients in a distributed environment is self-services.

Self-Services
Self-service is a concept in which the service provider exposes a sufficiently flexible interface that allows a knowledgeable and trusted client to control functional aspects of the service. This concept brings Web services from external system interfaces to internal heterogeneous application environments. Candidates for self-service providers include open source content providers with a large, diverse client base or global corporations with distributed IT environments.

While the self-service concept does not apply only to data-oriented services, I'll apply this concept using the technologies and constraints from my previous two examples. Instead of the service provider predefining a set of strict services, a generic service, as shown in Figure 3, is provided where clients are able to send XDS mappings and receive the requested content in the desired XML format. As with all engineering trade-offs, self-services expose more complexity to client in order to achieve the services flexibility.

XDS mappings are defined using an XML grammar, which allows for a seamless fit as the payload of a SOAP envelope. Listing 5 shows how I partition the parameters and mapping.

Once on the service side, the incoming SOAP envelope is parsed to create the parameter HashMap and extract the XDS mapping document. The XML results generated by the mapping are packaged in a SOAP response envelope and sent back to the client. Listing 6 shows the generic envelope parsing and XDS execution.

An HTTP POST of the XDS mapping document to our JSP endpoint implements our REST-style self-service, simple XML in and XML out. Listing 7 demonstrates the use of the XDS Tag Libraries, this time using the REQUEST identifier to specify the location of the mapping document as the POST body of the HTTP request.

While other self-services may not require the intimate detail a database provider may expose, the key is the client can specify precisely what data they want. The clients do not have to settle for a pre-configured service requiring them to selectively process the result elements. The client is also able to define the response format avoiding costly translation layers.

As with all sensitive services, security standards should be leveraged to authenticate the client and protect the integrity and confidentiality of the messages. In addition to protecting the message itself, we want to protect the resource we are service enabling. This is accomplished by allowing different clients to have different capabilities based on the service provider's level of trust in the client. The higher the level of trust, the more freedom the provider may allow the client to exercise. The levels of trust include not only the right of the client to see specific data, but also the trust in the client that they will not do harm to the provider either intentionally or through ignorance.

Staying with the database example we define three levels of trust. The first-level clients are restricted to using stored procedures with defined parameters. The provider defines a set of stored procedures, the procedure's input parameters and output fields. The second-level clients are permitted to execute the stored procedures as well as execute dynamic queries against a set of database views created by the provider. The views restrict read/write permissions as well as hide the details of the data model. The third-level clients are allowed full access to the database and are able to construct ad-hoc queries. All clients are able to construct an XDS mapping allowing them to set the values of input parameters and define XML mappings for output data fields.

Conclusion
Web services have given developers a new tool enhancing system interoperability and data sharing. Engineering is about tradeoffs of complexity, maintainability, flexibility, and performance among others. The opportunity to work firsthand with many legacy systems has proven the value of data-oriented services and the possibilities self-services hold. I'd be happy to answer questions or hear your thoughts on these topics.

Resources