colleague of mine, a database programmer who had spent some time working with an early XQuery implementation, once referred to the language as the smiley language. When I asked whether that had to do with the simplicity of the language, he replied in Hungarian-accented English, “Oh, no, the language itself is a bear, but they made the comment delimiters into smileys (: 🙂 to make you smile even as your database gets thrashed.”

While not exactly a ringing indictment of the language, his comments served to highlight the fact that while XQuery has a structure similar to most languages have, the differences can trip you up dramatically. XQuery is not a hard language to learn, but it can be a language that makes you try to understand why it’s not working.

Xquery’s control structures have been given the rather quaint acronym of FLOWR, shorthand for the most critical (but not the only) XQuery structures used by the language. FLOWR itself stands for five operations:

• For
• Let
• Order by
• Where
• Return

Four of which have analogs in SQL:

• SELECT
• SET
• ORDER BY
• WHERE

These terms are used to either assign or retrieve items from a set.

XQuery is a set manipulation language. It’s whole purpose is to work with sets of information, not just single scalar values. Additionally, as a set manipulation language, it is meant to augment—rather than replace—the XPath 2.0 language that’s built into the specification. Indeed, in essence, most of XQuery is just a means to wrap a control language around XPath, in a manner somewhat similar to the way that XSLT combines a templating language with XPath.

For this reason, when you are working with XQuery, the most effective use of the language is to do as much as possible within XPath 2 first, then go to the XQuery command structures when you’ve reached a point where XPath 2 can’t quite get you all the way.

For example, suppose that you had an XML data source that consisted of a collection of employment records. For now, assume they are in the file employees.xml (see Listing 1).

The for statement provides a way of setting the context to each employee in turn. The following code shows a simple XQuery script that outputs each employee in the order encountered in a list:

for $employee in doc("employees.xml")/employees/employeereturn $employee

In this case, a sequence of employee elements with their corresponding bodies is returned. Of course, the exact rendering of this sequence depends on the particular XQuery implementation used. For instance, if you use Saxon 9’s XQuery engine, the output would look something like Listing 2.

The eXist XQuery engine, on the other hand, would return a sequence of element nodes with no containers or closures. The primary reason, is that while Saxon assumes that the output will be an XML object (and hence needs to have some kind of containment), XQuery does not make that assumption. Note that you can bypass this wrapper problem by placing the whole query into an XML container.

{for $employee in doc("employees.xml")/employees/employeereturn $employee}

For Saxon, the output is similar, but not identical (see Listing 3).

The expression for $item in $seq can be a little misleading. In essence, as the for statement iterates through the sequence, what is passed into the $item variable is in fact an internal pointer to each item in that sequence in turn, rather than a copy of that $item. In other words, the $item context variable is live in that it is referring to a structure within an underlying XML (or related) data model, and that the result is in turn a sequence based upon this context.

For instance, in the expression:

for $employee in doc("employees.xml")/employees/employee 	order by $employee/lastname ascending	return $employee

what gets returned is a list of employees ordered by the employee’s last name—in essence, the order by statement creates a virtual sequence of the list reordered by the given criterion:

for $employee in doc("employees.xml")/employees/employee order by $employee/lastname ascending return $employee

where the bold expression represents this virtual sequence.

Obviously, for a complex expression, typing this repeatedly could get tedious. Fortunately, you can use the let command to establish a temporary variable holding this sequence:

let $sorted-seq := for $employee in doc("employees.xml")/employees/employee 	order by $employee/lastname ascending 	return $employee

This particular statement may seem somewhat counterintuitive, especially if you assume that let can hold only a single scalar value (as is the case with most programming languages). However, if you understand that the let statement is intended to hold sequences (and more sophisticated data structures) as well as scalars, then the expression makes more sense. Moreover, because the sequence created still points to specific elements within the XML structure, this re-ordered sequence is essentially just a sequence of pointers, not of whole (possibly huge) XML structures.

One upshot is that you can create a staggered filtering mechanism at surprisingly low cost, processing-wise. For instance, suppose that you wanted to create an expression that would sort the employees by last name and then provide you with records 11 through 20 inclusive of this sorted list. The following code illustrates one approach:

let $employees :=  doc("employees.xml")/employees/employeelet $sorted-employees := for $employee in $employees 	order by $employee/lastname ascending 	return $employeelet $paged-employees := subsequence($sorted-employees,10,10)return $paged-employees

In this case, each let assignment is, in fact, working with a sequence of nodes—the initial set of nodes from the employees.xml document, a sorted sequence of the same content, and the results of retrieving a subsequence of this ordered list of employees. Yet in all cases, what’s being extracted here are just pointers to elements. This paradigm is very useful in creating efficient queries, because rather than moving around large blocks of XML data (or rearranging an XML database) what is instead being manipulated is simply a list of such pointers, making the operations orders of magnitude faster.

One major caveat with this approach, however, is that this pointer manipulation holds true only so long as what is returned is a naked result (for example, $employee). Anything that changes the resulting content ends up creating new nodes of information, and the XQuery engine then has to effectively de-reference the nodes, even if the changes are fairly insignificant.

The code listed above, returns exactly the same result (with the bracketed results indicating that the XQuery expressions be evaluated and the results replaced in the stream) but because the first return statement creates new content, this is a considerably more expensive—and hence far slower—operation.

In general, you should try to avoid creating new content until you have a data set filtered down to as small a sequence as possible. Indeed, there’s a surprisingly common cadence to filtered searches and views that seem to recur whenever you’re dealing with querying potentially large data sets:

1. Retrieve the initial collection of resources and store that into a working variable.
2. Filter the items of this collection to retrieve only the relevant ones.
3. Sort the filtered data set.
4. Page through the sorted and filtered data set to retrieve a workable subset.
5. Transform through the paged content for the appropriate output.
6. Output the transformed content.

Retrieval occurs several ways. The doc() function takes either an absolute or relative URL and attempts to parse the contents of the URL as an XML document. The collection() function, on the other hand, retrieves a collection of nodes from an external source, without the need for that collection to have a containing element. This distinction may seem fairly meaningless in retrieving an XML file, but as most XML databases are set up around the notion of collections (with URIs corresponding to the collection rather than a single element), the collection() element is actually quite useful when invoked from within an XML database.

In an eXist database, you could create a collection of individual employees (though the exact mechanism for doing so won’t be covered here) and assign the collection to a given path, perhaps something like db/employees. You could then reference all of the items from this collection by writing:

let $employees :=  collection("/db/employees")

or, if using the XQuery for Java (xqj) notation:

let $employees :=  collection("xmldb:exist:///db/employees")

Where xmldb:exist:/// indicates that it is using the xmldb: protocol and that eXist is the server being referenced. The triple slash notation is also shorthand for the full protocol path, which most likely has the form:

let $employees :=  collection("xmldb:exist://localhost:8080/db/employees")

Internally, retrieved collections are treated as sequences, at least with respect to queries (updates, which are out of the scope of this article, do make a distinction). This means that regardless of whether you are querying the results of a collection, or of an XPath sequence pulled from a document, you work with the content the same way.

Filtering involves reducing the initial collection side to handle particular records that are important to you. For instance, suppose you want to retrieve only those records where people are in a given division (for example, “Materials”). You can of course, do this as part of the retrieval process:

let $employees :=  collection("/db/employees[division = 'Materials']")

On the other hand, splitting this into two distinct steps provide advantages from both design and performance standpoints:

let $employees :=  collection("/db/employees")let $filtered-employees := for $employee in $employees[division = 'Materials']” return $employee

You can also use the XQuery WHERE command, which lets you separate the evaluation predicate:

let $employees :=  collection("/db/employees")let $filtered-employees := for $employee in $employees	where $employee[division = 'Materials'] 	return $employee

Which is better? When the predicate (the expression in brackets) is relatively small and self-contained, using XPath on the sequence is usually faster. However, when the expression is complex, when it involves more than one variable or when this process is accompanied by a SORT BY, then using the WHERE clause is usually both more efficient and easier to read.In theory, you could get by with the primary FLOWR operators, but there are a fair number of situations where building such expressions could prove awkward. For this, you can use the IF … THEN … ELSE construction:

if ($condExpr) then $resultExprTrue else $resultExprFalse

Both the then and the else statement have an implicit return associated with them, meaning that you can use an IF statement to create a fairly complex script. For instance, consider a situation where you want a table showing the list of employees in a given section, but you want a status message if there are no employees in a given section. The IF…THEN…ELSE statement works remarkably well for this (see Listing 4):There are times when you have a conditional statement but you only want output when the condition is true or false exclusively. In this case, you can use an empty sequence (denoted by “()”) as the output:

if ($cond) then $output else ()

If the condition proves to be false, the empty sequence is returned, which in turn becomes blank output.

You can nestIF…THEN…ELSE statements within the result blocks, though expressions can get fairly complex when you have multiple nested statements. Suppose you want to create different header styles based upon the value of a variable $h (which can hold values from 1 to 6). You can use an embedded IF statement to create such a switch:

let $title := "This is a test."let $result := if ($h = 1) then 
{$title}
else if ($h = 2) then 
{$title}
else if ($h = 3) then 
{$title}
else if ($h = 4) then 
{$title}
else if ($h = 5) then 
{$title}
else 
{$title}
return $result

On the other hand, you can also make use of the element statement to create an element directly:

let $title := "This is a test."let $result := element {concat("h",$h)}{$title}

The element statement treats the first expression it encounters as the name of the newly created element, and the second contained expression as its content. You can see now there are a couple of different ways to accomplish the same task in XQuery.

Because XQuery can get complex fairly quickly, one construct that could be useful is a switch statement that allows you to create certain output based upon a given case value, for example:

switch($expr){    case $expr1:  $result1    case $expr2: $result2    default: $fallthruResult     }

XQuery doesn’t have a simple switch. What it does have is a type switch, which lets you perform actions based upon the data type of a given variable. The idea behind the type switch initially was to provide a means of identifying an element in an XML document and performing some processing based upon that element:

typeswitch($context)    case $a as type1 return $expr1    case $a as type2 return $expr2    case $a as type3 return $expr3    default $a return $fallthruResult

This approach may seem a little counterintuitive, though an example may make it a little clearer. Suppose you want to apply specialized formatting to each node in an employee record. You could do so with the typeswitch control as follows:

{for $employee in doc("employees.xml")/employees/employee return   for $node in $employee/*  return typeswitch($node)    case $a as element (firstname) return {string($a)}     case $a as element (lastname) return {string($a)}    case $a as element (title) return 
{string($a)}
    case $a as element (division) return 
{string($a)}
    default $a return 
{string($a)}
}

The problem with typeswitch is that it solves an edge case in the language, but doesn’t do terribly well with the more common problem of conditionally choosing different paths based upon a given string token. Fortunately, with a little bit of work, you can make typeswitch more like a traditional switch. The trick is to turn a token into a temporary element by using an XQuery expression. You can generate different outputs depending upon which particular division a given person is in (see Listing 5).

The key to this routine is in the line:

    let $division := element {string($employee/division)}{}

The expression element {string($employee/division)}{} looks rather cryptic, but can be decomposed fairly readily—the first braced expression evaluates to the name of a division such as Materials or AcctsPayable, while the second empty braced expression indicates that this is an empty element. Together, this creates an element in the form , , etc. This element can then be used to trigger the various cases, such as:

case $a as element (Materials) return  
    
Materials Section
    
{concat($employee/firstname,' ',$employee/lastname)}
    
{string($employee/title)}
    
{      let $manager := $employees[@id = string($employee/supervisor)]      return concat($manager/firstname,' ',$manager/lastname)      }
  

The output in turn is the appropriate format for that type of material. In the default case where there is no match, a warning message is thrown up indicating that there’s a problem with that particular employee record. The $a variable in this case holds a pointer to the division element, but in the above example it is unused and is simply a placeholder for the case routine.

The XQuery 1.1 working draft hints at other control structures, including the GROUP BY operator that makes it possible to aggregate results by group selectors and WINDOW clauses, which provide an ability to easily do set operations on subsequences of a given sequence. Additionally, the XQuery Scripting Extensions (or XqueryScript) provide other control structures that are designed to make XQuery easier to use within a more formal scripting role. However, both these drafts are still very much in development, and currently no commercial or open source implementations of XQuery support these capabilities.

Control structures are not necessarily glamorous; indeed, they are about as exciting as rebar scaffolding, but like such scaffolding they are a critical part of building any XQuery application. Understanding how to work with these structures can make the difference between a useful, flexible application, and a one-off piece of code that will have to be written over and over again.