Essential Tools
The "tools" that will help you develop or work with the above list are all essentially graphical user interfaces around the files and fragments that come out of the end of the development effort, and include:

Essential Infrastructure
The infrastructure that needs to exist before any of these can be worked with includes:

Essential Standards
The development environment must enable applications to work with accepted standards, such as DOM3 and SAX.

'Nice-to-Haves'
The "nice-to-haves" in this picture might include:

We're rapidly arriving at an extremely complex picture of the requirements for an XML development tool, yet the picture is murky. Is it actually possible to combine all of the above into one tool? And if so, would anybody buy it or use it? What are we going to develop with these tools?

IDE
Let's look at the tools currently on the market. The term IDE (integrated development environment) is used by many manufacturers in the XML community these days. Products such as XML Spy from Altova and TurboXML from TIBCO/Extensibility spring to mind. The term environment represents a collection of tools in this context.

We've been conditioned by such products to interpret XML development as a means of creating DTDs, documents, stylesheets, and transformations. Yet when I create a DTD, then write a document that validates against that DTD, then store that document in a content management system, have I developed XML? Arguably, yes. But to what end? The XML does nothing without an infrastructure to use, publish, process, and communicate it.

Furthermore, as soon as I want to use an element in various places and for various purposes, I have a problem. Powerful software development environments provide the functionality to support true object reuse, inheritance, versioning, backups, and a permission system. The XML IDEs on the market at the moment don't.

If I create an XML-driven system that relies on a Transaction ID element (as described earlier), an XML IDE will help me put it into dozens of schema fragments, XSLT files, stylesheets, and so on. But the IDE won't manage the fact that the Transaction ID is actually one object that has been used in all these different places.

This is very disturbing from a software developer's perspective, especially if you're used to an object-oriented development environment and accustomed to the notions of, for example, single source and extension of object classes.

XML as a 'Language' Is Weird
The adoption of XML in IT is now unilateral, from finance to pharmaceutical, aerospace to defense. XML is mainstream. Nobody can afford downtime, and systems must support the business process flawlessly. In the pharmaceutical sector the legal repercussions of a mistake due to a problem in the publishing process could mean lawsuits totaling many millions of dollars. In the stock markets two minutes of downtime could represent millions of dollars in lost opportunity for an investment bank.

No Source Code
If we're going to work with XML in these environments, we should treat it with as much respect as all the other parts of the system. We should store the source code in a source control system, and apply versioning, branching, permissions, automated testing functions, and so on. But (I hear you say) XML doesn't lend itself to such management. There's no source code!

Compared with third-generation programming languages (3GL) such as C, COBOL, or Fortran, XML is very strange. Of course, comparing XML with 3GL isn't entirely fair. XML isn't a language in the traditional sense of programming languages. XML lets you use words that describe your business and apply constraints to those words. XML has no source code in the way that C++ or Java can be said to have source code, so we can't store and manage source code for XML. There are no XML compilers and development studios that provide the kind of functionality that, say, a Java developer would expect from an Enterprise version of Borland's JBuilder product.

However, it's this very inability to treat XML as we treat traditional programming languages that gives us so many management challenges in mission-critical environments. XML needs source, it needs powerful development environments and compiler-type technology, and it needs infrastructure to manage the development environment.

XML Is Impossible to Model
In complex IT environments, models are important. Models allow us to conceptualize the components in a system and generate the pieces that we need to develop software systems. For example, from a CASE (Computer Aided Software Engineering) tool or UML (Unified Modeling Language) modeling tool, I can generate Create Table scripts and stored procedures that build my database environment for me. Models simplify the work of application integrators and allow people who care about business processes to relate the functioning of the organization to the IT infrastructure supporting the organization.

XML Is a Runtime Expression of Something Else
As seen in Figure 2, when two processes exchange information in XML:

1. The XML represents a small part of a process model.
2. It's probably a subset of a database schema.
3. And it probably contains temporary values generated for the purpose of the current process but not likely to be stored.

Serious Problems
At least, that's the idea. That is also the reason so many organizations are beginning to run into serious problems with XML development. The following problems arise for the developer:

A Model of the Document Is Not a Solution
To solve the foregoing problems, we need to model the document in a schema or DTD. The "model" of an XML document is its schema (see Figure 3).

However, having a schema simply isn't sufficient. Think back to the Transaction ID object mentioned earlier. I care about transaction IDs because I'm building a software system for a commercial application - an insurance claims system, say, or a foreign exchange business line application for an investment bank. The Transaction ID isn't something that starts life in XML - it's probably there because of gigabytes of legacy data, existing software applications, and a UML model on somebody's desktop. It turns into XML because I'm using XML to glue all the different pieces of my system together with XML. And my transaction ID travels around and touches every system, every program, every database....

My company acquires another company and we start to integrate our IT infrastructures. My transaction ID is an integer. The other company's transaction ID is an integer plus a timestamp. I've built my system using XML; my transaction ID occurs in thousands of different places and I have no way of knowing where. I don't have a model of my XML. The upside of XML when I built the system was the phenomenally short time-to-market for a highly sophisticated system. The downside is that the result is difficult to maintain and there are inherent risks.

A Document-centric Example
Imagine the publishing system at a pharmaceutical company. My project scope is the packet of tablets that a pharmacist sells to the public. I have four documents: a label on the drugs, a label on the box, a sheet of information inside the box for the customer, and a sheet of information for the doctor who prescribes the drugs. I have 300 products. I sell in 130 countries. I need to publish in at least 30 languages. I have to deal with the local legislation and cultural requirements of each country. XML offers me the chance to build the perfect solution (and nothing else can).

One of my attributes is a piece of text describing a certain warning in a certain set of circumstances for pregnant women at a certain stage in their pregnancy. By the time I've rolled out my product in 130 countries and 30 languages, a complex authoring and publishing environment will have been built to process the attribute, and the attribute itself will have been used or referred to in potentially millions of places. There's no way of knowing where, or what the impact will be if a related property needs to change. Not only is this dangerous (if an error in my publishing system causes a life to be lost, is this XML-induced death?), but you can't forecast budgets for unpredictable outgoings.

How can I manage the complexity of such a system? This is where a 3GL-like XML development environment could significantly reduce the risks and costs involved.

Creating Source Code for XML Environments
Let's examine what it would take to create a 3GL-like development environment for XML. First, we need source code. If a schema is neither the source code, nor the model, for an XML document, what else can play this role? The answer is that we need a way of modeling the model. In other words, we should create an object model that can store enough information about a structure to be able to generate the pieces needed in the XML-based application. It should be possible to generate schemas, stylesheets, and the like from an object model.

Enabling Object Reuse and Inheritance
An extra dimension is required for true object reuse and inheritance (see Figure 4). For example, to treat a transaction ID as one object that has been used in multiple places, we need a model of the model of the model of the document. That is, from an object model we need to be able to assemble structures (models of schemas) from which we can generate schemas and other property files, which describe the validation rules and process requirements of a document.

The structures we assemble are like contexts in which an object is used. In context one, A contains B followed by C, which contains D followed by E. In context two G contains A, which contains H followed by B. If I know that A in any particular context is actually a reused object from a model where it occurs as one and only one object, I can apply true object reuse and inheritance.

The next hurdle is versioning of objects. This is where the underlying technology becomes very important. In any sophisticated source control environment, developers can check out an object, work on it, and check it back in again. This activity is controlled by a versioning mechanism that allows updates to happen on a branch of the parent code line. Full permissions and transaction support needs to be applied. Multiple variants of a code line are possible. Published versions are identified by a time line that essentially picks up the relevant pieces and freezes them into a state that cannot be violated.

The only technology that can support this level of control over source code is repository technology. In other words, everything about the model of the model of the model of the document needs to be captured and managed in a database, which requires a very sophisticated approach to metadata. Get this piece of the puzzle right and XML development will take a quantum leap forward! To my (admittedly biased) mind, only one product anywhere in the world is capable of providing such support - CorteXML from Barbadosoft, which was designed to enable high-speed change in complex XML-based environments.

Conclusion
Currently, XML development environments do not provide the level of developer support customary in a 3GL programming environment. The question we need to answer is, Do we need 3GL-like XML development environments? For simple systems, probably not. But for complex systems I would argue that we cannot do without them.