eclarative programming can take a little getting used to, especially if your standard mode of operation is working with languages like Java or C#. In essence, such programming requires that you think not of objects, properties and methods but rather of rules, filters and pipelines.

Indeed, one reason that the future is looking increasingly declarative is that the web, as a network, does not lend itself well to being described as a collection of objects with methods and properties. That resistance is at least part of the reason why SOA (service oriented architecture) essentially requires that you build an entire infrastructure on top of the web just to make it work properly.

I've found, over the years, that to me a far better analogy for developing distributed web applications is to envision one of those ball and pipe machines that you see occasionally at science museums: the complex clockwork that involves moving balls around in a circuit. The balls, in general, are in one of a few different states—they are either waiting in a queue, they are hitting a filter or node, or they are moving between nodes and queues.

However, to make the analogy a little more interesting, suppose that the balls can be of different colors, each of which corresponds to a given "object," and the color of each ball can be detected and acted upon every time the ball encounters a node. What's worth noting here is that, as far as the bigger picture goes, the machine does not really care about the underlying properties of the balls; the individual filters will care, certainly, but from the standpoint of this model, the specific state of the balls beyond their color is immaterial.

In this sense, the filters themselves can be thought of in a purely functional sense as black boxes, but again at a somewhat higher level of abstraction than is normally typical for functions. In essence, at the top of each black box are zero or more queues, each of which can hold zero or more balls of a given color. Each box has a set of rules that indicates that when a set number of balls in each queue exist, then the black box will load in those balls and generate zero or more balls of a potentially different color, sending it on its way down another pipe.

Each colored ball is (typically) an XML document that satisfies a given schema. For instance, blue balls may correspond to an XHTML document, red balls would be Atom messages, yellow balls might be a data format. The cascade of such balls through the pipeline system in turn then illustrates the fact that while XML moves like messages through the system between nodes, realistically, the balls put in at the beginning of the pipeline will likely bear no relationship to the balls coming out the other side.

Processing XProc
As much as this might seem an open invitation to building Rube Goldberg-like solutions to architecture, it turns out that this particular model is really quite powerful. For starters, there is functionally no distinction between a large number of filters, each of which perform simple operations, and a single filter which performs very complex operations (i.e., it's decomposable). All the state is carried in the balls and not the filters. Indeed, even "application environment" variables can be encoded as a bundle of state values and passed around in the same manner.

Finally, because of these two properties, a declarative pipeline architecture can be described as an XML document itself, one that defines types of actions (such as transformations or queries), with the individual files implementing those actions being other balls (or bundles) of state. An early, but classic, example of this is Java Ant (or the .NET equivalent, Nant) which is a make file written in XML that is beginning to replace the dominance of the C++ make command.

However, there's a new XML process language that's making some significant progress within the W3C. The XProc, or XML Pipeline Language, is designed as a way of describing a set of declarative processes, along with the inputs, outputs and throughputs of those processes, used within an XML pipeline. For instance, consider a simple pipeline for an XHTML document with enclosed XInclude statements for loading in other resources into the XHTML document, after which the resulting document needs to be validated (see Figure 1).

Figure 1. XInclude/Validate Pipeline: You need to validate the resulting document.

The XProc specification for this particular document could be written as shown in Listing 1.

In this particular example, the sequence as given includes two distinct parts. The header section defines two key input "ports," source and schemas, and one output "port," result. A port can be thought of as a named entrance or exit to the XProc file, and is typically established by the implementation itself. For instance, if the variable _source and the variable _schemas contained an XML DOM within XInclude elements and an XML Schema document respectively, then an implementation of this XProc might look something like this:

var proc = new XProc();
var _output = new Object;
proc.load("schema-proc.xml");
proc.setPort("source",_source);
proc.setPort("schema",_schema);
proc.setPort("result",_output);
proc.exec();
print(output);

Note, this code is just a theoretical interface, the specific implementation is up to the application provider.

When the Javascript procedure runs, the XProc is run in sequence (unless inclusions occur, in which case processing is a little more complex). Thus, the first step, included, will take the content contained in _source and will render any XIncluded links as their included content in the document. This is the first xinclude step:

<p:xinclude name="included">
<p:input port="source">
<p:pipe step="xinclude-and-validate" port="source"/>
</p:input>
</p:xinclude>

The above code indicates that the source of the input is in fact the same as the source for the "xinclude-and-validate" input. Not surprisingly, much of XProc is involved with establishing the sources and sinks of pipes in the pipeline. One conceptual way of thinking about how such pipes work is that to this point there are two distinct sources: xinclude-and-validate.source and included.source that happen in this case to be the same thing. However, in the next block,

<p:validate-with-xml-schema name="validated">
<p:input port="source">
<p:pipe step="included" port="result"/>
</p:input>
<p:input port="schema">
<p:pipe step="xinclude-and-validate" port="schemas"/>
</p:input>
</p:validate-with-xml-schema>

The source input is defined as the result of the "included" step. If an output isn't defined for an XProc pipe (i.e., for xinclude or validate-with-xml-schema in this particular instance), then the port is assumed to be named "result," which is made explicit in the "validated" block. The second input, the schema, is pulled from the explicitly named "schemas" port that was declared for the whole block. With these two inputs, the validate-with-xml-schema can be run to validate the content. If the document is in fact valid, then the post-schema-validated-infoset (PSVI) is passed to the "result" port.

Note that the default behavior for validation with XML Schema (XSDL) varies somewhat from validation for RelaxNG, in that an unvalidated copy is passed on rather than error messages, because the result of an XSD validation is in fact a distinct (and different) object, rather than a simple Boolean flag. However, an option can be added to this block to specify that the assertion that this is valid must be true. If the validation fails, a dynamic error is called, and the XProc processor will either stop at that point or, if it's defined will perform the <catch> action in a previously defined try/catch block. Put more simply, XProc supports exception handling.

The output of this operation will then be the original XHTML document with XIncludes added in, and then run through an XSD validator to return PSVI document. Notice that the process given is also basically generic. Both the source input and the schema are parameters. If you change either of these (or, as a more sensible operation, replace the schema validation with a transformation, and use two different transformation files) the results will be different, but the XProc is the same.

An Inventory of Steps
A number of "keyword" elements act as steps within XProc, including the following:

• <pipeline>: Defines a named set of pipes, wrapped up into a single referenceable object.
• <for-each>: This is a compound statement, wherein a given set of documents (or nodes within a document) are passed, one at a time, to a given sub-pipeline and processed.
• <viewport>: A viewport is similar to a for-each block, in that for a given input document, it retrieves a set of nodes from a given XPath, applies a subpipeline to those nodes and replaces the original nodes with the results of the subpipeline.
• <choose>: Chooses a particular pipeline among several based upon constraints in the source document, then executes the subpipeline on that input. If nothing matches, a default operation is available.
• <groupara>: A group is something of a convenience step for subpipelines, making it easier to identify subgroups and apply just that subgroup of pipes.
• <try><catch>: Provides an exception handling mechanism when a process throws an exception.

These commands make it possible to handle fairly complex logic, including conditional processing, enumerations, exception handling and the like, just as analogous keywords make it possible to create complex procedural logic.

A second set of pipes handle specific operations. The XProc specification lists a number of them:

• <add-attribute>: Adds an attribute name and value to specific elements.
• <add-xml-base>: Includes a reference to the absolute URL of the process, rather than a relative one.
• <compare>: Compares two documents for equality, passing the document if true and otherwise raising an exception.
• <count>: Counts the number of documents in the input source.
• <delete>: Deletes any item from a source document that satisfies a given XPath pattern.
• <directory-list>: Retrieves an XML structure containing the files and directories of a given directory in a file or related system.
• <error>: Actually generates a custom dynamic error on the given input document.
• <escape-markupara>: This serializes the selected sub-elements in a given XML document.
• <http-request>: Performs an XMLHttpRequest to either send or retrieve content from the web.
• <identity>: Makes a verbatim copy of the input as an output.
• <insert>: Inserts a block of XML into a source document at the matching position(s) specified.
• <label-elements>: Creates a label for a matched element and stores it in the specified attribute.
• <load>: Loads a document the URL of which is specified in a parameter.
• <make-absolute-uris>: Converts an element or attribute's value in the source document into a uniform resource identifier (URI) in the target document.
• <namespace-rename>: Changes the namespace in a given document to a new URI, useful when working with proxy namespaces.
• <pack>: Combines two documents into a single document with a new wrapper element containing them.
• <parameters>: Passes a set of parameters with associated values to a given step.
• <rename>: Renames a given element or attribute.
• <replace>: Replaces an element and its children with a different element and its children.
• <set-attributes>: Sets the attributes on matched elements.
• <sink>: Accepts a set of documents and discards them. This primarily occurs when the role of the preceding pipeline was to perform out-of band operations.
• <split-sequence>: Takes a sequence of documents and splits that into two sequences.
• <store>: Serializes the content of the input document and persists the output to the given URI.
• <string-replace>: Replaces all instances of a given string in the source with replacement text in the result. Note that if XPath 2.0 is used, this will likely support regular expression replacements.
• <unescape-markupara>: Converts serialized text and parses (converts) the text in XML.
• <unwrapara>: Replaces matched elements with their children.
• <wrapara>: Replaces any elements that match the given expression with a wrapper element. If two or more matched elements are adjacent, then will wrap all of the adjacent nodes under a single wrapper.
• <wrap-sequence>: Wraps a sequence of elements with a wrapper element.
• <xinclude>: Includes the content specifed in an element into the source document to replace that element.
• <xslt>: Performs an XSLT transformation on the given source document.

These are all considered required for XProc operations. In addition to these, the specification also defines a set of 10 additional optional steps that don't have to be implemented, but if they are implemented should take a given form:

• <exec>: Executes an external command using the arguments specified.
• <hash>: Creates a digital hash of the corresponding source.
• <uuid>: Generates a unique global identifier.
• <validate-with-relax-ng>: Validates the document using the Relax NG Schema Language.
• <validate-with-schematron>: Validates the document using the Schematron Schema Language.
• <validate-with-xml-schema>: Validates the document using the XML Schema Definition Language (XSDL).
• <www-form-urldecode>: Decodes a URL-encoded string into parameters.
• <www-form-uelencode>: Encodes a set of parameters as a form-url-encoded string.
• <xquery>: Calls an XQuery script on the source and retrieves the result.
• <xsl-formatter>: Takes content rendered in XSL-FO and generates output in PDF or similar formats.

Obviously, given the range of commands given here, its possible to create very complex pipeline operations just with the standard set of commands, especially given the meta-nature of pipelines. It's also worth noting the use of the optional <exec> command that is designed to make system calls to the underlying operating system, using the source document as the first input parameter and sending the output to the result port. Obviously, this particular command may not be available in a non-secured environment.

Beyond this core set of pipeline "steps," the XProc specification also provides extension mechanisms both for the underlying XPath used to select nodes and for defining additional steps within the XPRoc processor. For instance, one such step might be to provide a command to send the associated content as an email message to a given email list, while another might perform authentication against an LDAP server to ensure that the rest of the pipeline can proceed.

XProc In the Pipeline
XProc heralds a significant shift in the building of XML pipelines and web applications. The specification itself will likely be out either late in 2008 or early in 2009, and already a few XML database creators are exploring the deployment of XProc within their own systems, either as something that can be invoked from within other processes (such as an XQuery call) or as scriptable entities in their own right. Because of it's declarative nature, it’s also not hard to foresee a point in the near future where XProc will be used to marshal actions across multiple server environments, though this first specification only hints at that vision—in short, XProc has the potential to become a vehicle for larger scale multi-system orchestration.

In the more immediate term, you can get a first glimpse of XProc via prototype implementations listed at http://del.icio.us/ndw/xprocimpl. Some of the steps listed here may not yet be available (the most recent XProc "drop" was May 1, 2008), but the overall form of the specification isn't likely to have changed dramatically. More information about the XProc working draft itself can be found at http://www.w3.org/TR/xproc/.

If other standards like XSLT2 and XQuery are any indication, adoption of XProc is likely to be slow at first, given the presence of commercial workflow systems, but like those two standards, adoption should pick up pretty quickly with one or two solid implementations, as XProc neatly solves a number of problems that tend to transcend working with any single XML operational language. Developers are moving to ever larger levels of abstraction as programming moves beyond single processor environments or even standard client/server architectures and it is likely that XProc will be one of the languages leading the charge to that next level.