f you’ve ever worked with sophisticated database systems or large applications you’ve probably accumulated sizable libraries of stored procedure code. Such libraries typically hold a few hundred stored procedures, ranging from perhaps a few hundred lines of code to thousands of lines?but only minimal documentation. That’s because existing database documentation generators operate at the database-object level: that is, they describe the database schema, list the table structures, and show object relationships. They do describe stored procedures, but only with a skeleton?typically just a list of parameters and their data types, and then typically include the entire source code. There is no injection of user-supplied documentation from the source file. Clearly, that is not an adequate API.

I have been described as a documentation fiend; I document extensively, from design through development, wherever I go. So I could not, of course, let this situation stand. If these SQL files were C# or Java libraries instead, it would be inconceivable not to have thoroughly documented the code using common documentation generator conventions, so that anyone could generate a complete API describing how to use the code base. For example, look at any C# API in MSDN, and you will find a description of each method and each parameter within each method, as well as sample code usage rather than source code. In C#, NDoc is commonly used for this; Java has JavaDoc, and so on.

To address the problem, this article leverages a Java application I developed some time ago called “XmlTransform,” whose purpose is to convert one XML variant to another using XSLT mapping. Originally, the program was intended to generate web pages, but as the ideas behind it are general purpose, XmlTransform serves equally well to generate SQL documentation. To tailor the transformation you must define a few custom XML tags in an XSLT file and instrument your SQL source files with those tags. The tags delineate the various types of content in the final documentation, just like XML documentation used in other languages. Using those tags, XmlTransform then generates a finished HTML documentation set for you.

Author’s Note: See the related article, “XmlTransform?A General-Purpose XSLT Pre-Processor,” for more in-depth information about using XmlTransform.

The sample project that accompanies this article contains all the necessary code in these directories to demonstrate generating documentation:

• src?SQL source.
• XMLsrc?XML documentation source files extracted from src.
• api?HTML documentation generated from XMLsrc.
• code?programs to generate the SQL documentation set from src into XMLsrc and api (described below).
• sandbox?Component samples you can experiment with.

The code directory contains:

• sqldoc2xml.pl?Perl source code to extract XML documentation comments from the SQL source files in src, putting the results in XMLsrc.
• XmlTransform.java?Java source code for the transformation engine that maps the source XML in XMLsrc into the target XML in api. The source XML is a simple, custom XML dialect that’s similar to other documentation languages. The target XML is XHTML.
• translate.xsl?an XSLT file that directs XmlTransform how to map the XML source to the XHTML target.
• cleancode.css?style sheet for displaying the resultant XHTML.
• XmlTransform_params.dat?parameter file for running XmlTransform
• customJar?Java libraries
• lib-external?Perl external libraries
• lib?CleanCode Perl library

You’ll find two downloadable zip files in the downloadable code that accompanies this article. The small code archive (175 KB) includes only the collection of sample project files discussed here, along with the output of each technique covered. The larger zip (4.7 MB) is a superset of the first; it also includes all the external Perl and Java library files you need.

Step 1: Instrumenting SQL Files with Doc-Comments
Your goal in this step is to embed documentation into your SQL source files. This step probably takes the most time because you have to do the work (in subsequent steps the computer does the work). As an example, suppose you have a stored procedure unimaginatively named MyProc, defined in a file by itself, named MyProc.sql, which takes two parameters.

	?
Figure 1. Instrumenting a SQL File: Insert XML documentation elements as SQL comments. (The body of the procedure has been omitted for brevity.)

Figure 1 shows the initial base file on the left (base.sql) and the first stage of instrumenting it on the right (xhtml.sql). The highlighted region on the right contains the inserted XML documentation. The inserted text is pure XHTML that describes the procedure?in other words, this is just an XHTML document embedded within a SQL document. Each XHTML line is preceded with a SQL comment marker. SQL Server will interpret the file as pure SQL because it ignores comments, but the XML extractor application does just the opposite: it ignores the SQL and any plain comments, and interprets only those comments containing the documentation. Comment position is irrelevant; you may put all the documentation comments into a single section, as in Figure 1, or interleave them with the code, creating multiple comment sections.

The XML extractor application (sqldoc2xml) reads each line of text. It recognizes the start of an embedded XML documentation section by the presence of a SQL comment marker (?) followed by an opening bracket for an XML tag (<). Any whitespace between the two tokens is ignored. After finding an XML comment, sqldoc2xml considers all subsequent, contiguous comment lines to be part of the embedded documentation section as well; however, the documentation section ends when sqldoc2xml encounters a non-comment line. At that point, it again starts ignoring all text until it finds another comment marker and XML opening bracket on a single line. In other words, sqldoc2xml collects all the documentation comment sections, strips off the SQL comment markers, and thus distills a pure XML file while ignoring everything else. It is a simple process, but quite effective.

The next stage is to use custom XML instead of pure XHTML to enhance the maintainability of your code base and leverage the power of the XmlTransform transformation tool. Your task is to extend XHTML with new elements and then provide XSLT rules to map these new elements into standard XHTML. Think of the new elements as XHTML “macros.” You might have a macro to specify what an example looks like, what a usage section looks like, what copyright you want to use, and so on?all things that you want to specify in one and only one place?so if you need to change something later, you can simply change the macro definition. After defining your macros, you create XSLT rules that convert them into pure XHTML, which may be rendered in any browser. Finally, with your macros and XSLT rules in hand, you then instrument your SQL files with documentation comments written in your custom XHTML dialect, rather than pure XHTML. This extra initial effort makes your documentation comments both completely customizable and highly maintainable.

The translate.xsl file included in the sample project accompanying this article provides a number of macros and translation rules to get you started, as will be described in the sections that follow.

Define a Return Value for a Function
The element specifies the type and a description of the return value of a function.

Author’s Note: By design, the element and other documentation elements mirror similarly named C# documentation elements, although they may add or omit attributes.

Table 1 describes the details of this element; the same format is used for each subsequent element. The table first identifies the XML element used in documentation comments (prefaced by SQL comment characters in your code). The element is shown as a template, with the parts that you must fill in indicated in blue.

The XML schema representation presents a more formal and precise representation of the same information; it shows a portion of the formal schema for the custom extension of XHTML discussed in this article. The cc namespace of is implicit in the diagram, hence it appears as returns in the blue box. Conversely, the XHTML elements that do not use namespace prefixes in the files in the sample project are shown with an xhtml prefix in the diagram. In words, this diagram states that the <cc:returns> element has a type attribute, and its body consists of any number of standard XHTML block or inline elements.

Author’s Note: XmlTransform can use the schema file behind this diagram to validate your input files; that is, XmlTransform can tell you whether your files use the correct XML grammar. Although schema validation is beyond the scope of this article, see the XmlTransform API for information on how to invoke the validation.

The next row of the table shows the XSLT rule that operates on the XML element. When your file is being converted from XML to XHTML and the given XML element is encountered, the rule specifies how to map that element to XHTML.

The remaining rows show a sample, starting with how the element would appear in a SQL file, the intermediate output from sqldoc2xml, and the final output from XmlTransform. In most cases, moving from the SQL input to the intermediate XML is trivial; the process just removes the leading SQL comment markers. But the next element, , shows one instance where additional work is performed.

Table 1: XHTML Macros: The element functions as a documentation macro tag that outputs the return type of a function.

XML element	function-return-type“> ?????parameter-description
XML schema representation
XSLT rule	?? ???? Returns: ?????? ???????? ????? ????? ??
SQL input sample	? ?This table-valued function processes blah, blah, blah… ?
Intermediate XML sample	This table-valued function processes blah, blah, blah…
XHTML output sample (see Figure 2, frame 1)	?? Returns: ???? ???????table[data as varchar(256)] ?? ?? ????This table-valued function processesetc…. ??

	?
Figure 2. Page Components Revealed: Each customized element maps to a standard XHTML representation.

The final row in the table for each custom XML element includes a reference to a frame in Figure 2, and each frame shows rendered XHTML for a particular macro tag in context.

In Figure 2, the numbered sections correspond to these macro tags:

1. cc:returns
2. cc:param
3. cc:usage
4. cc:see
5. cc:seealso
6. cc:apiSection
7. cc:product-summary

Define Procedure or Function Parameters
Each parameter is immediately followed by a custom element, in the SQL file in the left-hand image (see Figure 3). The positioning is important for two reasons. First, if you later decide to change a parameter name or meaning, the corresponding documentation comment is right there, which makes it much more likely that you will also remember to update that as well.

	?
Figure 3. Automatic Parameter Documentation: You describe the parameter in your source SQL file, and then sqldoc2xml interprets your source code to insert the parameter details while XmlTransform converts sqldoc2xml’s XML output into finished XHTML.

Second, when the XML elements are properly interleaved with the parameters, the documentation tool saves you some initial work and some ongoing maintenance. This is the one case where the sqldoc2xml tool looks not just at the comments but at the code as well; it parses the parameter names, types, and default values, and attaches those attributes to the XML element, as shown in the middle image in Figure 3. Notice that the final HTML in the third image in Figure 3 is a combination of the documentation comments and the actual SQL code.

I use for my parameters, but you may use any name?sqldoc2xml attaches attributes by looking at the position of the XML element following the SQL parameter definition, not by its name. It is the second-stage tool?XmlTransform, which actually converts the XML to XHTML?that requires you to identify the XML element semantics and how those map to XHTML.

The custom XML element gets mapped back to XHTML with an XSLT transformation, which renders it as a simple definition list (

…

). The rule that creates the XHTML rendering shown in Figure 3 includes a variety of formatting elements (bold, italic, colors, and indents) and even fills in the phrase “-none-” if the SQL code doesn’t define a default value. You may, of course, choose to use something simpler or something more complex. Unlike all the other XML elements discussed here you do not need to fill in any attributes for the element; sqldoc2xml fills those in automatically, as shown in Figure 3 and in Table 2.

Table 2: The Element: Use this macro tag to provide a description of each parameter to a procedure or function.

XML element	parameter-description
XML schema representation
XSLT rule	?? ???? ??????? ???????[ ?????????Type: ?????????????????????? ??????????????????? ??????????Default: ????????????????????????? ?????????????????????????????????????????????????????????????”string-length(@default)”> ????????????????????????????????? ????????????????????????????? ?????????????????????????????-none- ????????????????????????????? ????????????????????????? ??????????????????????? ????????] ???????? ???? ???? ??
SQL input sample	@Description nvarchar(2048) = null, ????? ?????This parameter specifies blah, blah, blah, and blah. ?????
Intermediate XML sample	???name=”@Description” ???type=”nvarchar(2048)” ???default=”null”> ?????This parameter specifies blah, blah, blah, and blah.
XHTML output sample (see Figure 2, frame 2)	?? @Description ???? ????????[ Type: nvarchar(2048) ???????????Default: null ] ????? ?? ?? ????This parameter specifies blah, blah, blah, and blah. ??

Display a Usage Block
This element provides a compact description of usage, a synopsis, specifying the syntax for a given procedure or function, a convention used for years on Unix/Linux manual pages. The XSLT rule for this element uses the HTML

 element, which renders output in a fixed-width font, so it looks like a code sample. Additionally, the rule specifies a CSS style name to apply.

Author's Note: There is significant overlap between what you can do with XSLT transformations and what you can do with CSS style applications. Often a combination of both is most efficient for producing a particular visual effect. See the  discussion for a more complex example of XSLT and CSS together.

Table 3: The  Element: This macro provides a place to specify syntax or a synopsis for a procedure or function.
XML element

????usage-text
XML schema representation
XSLT rule

????
????????
????
SQL input sample
??
??MyProc?@Description,?@Verbose
??where:
?????????@Description?::=?[string]
?????????@Verbose?????::=?[smallint]?in?the?range?0?to?3
??
Intermediate XML sample

MyProc?@Description,?@Verbose
where:
???????@Description?::=?[string]
???????@Verbose?????::=?[smallint]?in?the?range?0?to?3
XHTML output sample
(see Figure 2, frame 3)

MyProc?@Description,?@Verbose
where:
???????@Description?::=?[string]
???????@Verbose?????::=?[smallint]?in?the?range?0?to?3
Display an Inline Reference as a Hyperlink
The XSLT rule for this element shows some of the flexibility and power available from XSLT. Table 4 shows two variations on the element, but there are actually four because you can omit the reference-text in either variation. All variations get transformed into the HTML  element you can see in the last row of Table 4. If you do specify the reference-text, the text is used as the displayed portion of the hyperlink; if not, then whatever you supplied for the cref or href attribute serves double duty as both the link and the display text. The cref (short for code reference) attribute lets you specify other procedures or functions within your own documentation tree. As the example shows, that gets converted to the appropriate XHTML in the  element link. An href attribute, on the other hand, is used verbatim, and lets you specify any URL.
Note that the XSLT rule for this element contains two templates because the <_see> template is shared by the  element, which is discussed next.
Table 4: The  Element: Use this element to insert reference hyperlinks to either internal (other function/procedure) or external (any link) documentation.
XML element
reference-function-or-procedure-name“>
????reference-text


-?-?OR?-?-
reference-URL“>
????reference-text
XML schema representation
XSLT rule

??

??
????
??????
????????
??????????.html
????????
??????????
????????
??????
????
????
??????
????????
??????
??????
????????
??????
??????
????????
??????
????
??
?
SQL input sample
??Adapted?from?
??The?Meaning?Of?Life
??and?.
Intermediate XML sample
Adapted?from?
The?Meaning?Of?Life
and?.
XHTML output sample
(see Figure 2, frame 4)
Adapted?from?The?Meaning?Of?Life
and?MyOtherProc.
Display a Reference Hyperlink
The XSLT rule for  uses the same base code, the same parameters, and the same variations as . The only difference is in the final XHTML rendering:  is for an inline reference, while  renders as a separate paragraph, typically used at the end of a documentation page.
Table 5: The  Element: Use this element to render a “See also” hyperlink in a separate paragraph.
XML element
eference-function-or-procedure-name“>
????reference-text


-?-?OR?-?-
reference-URL“>
????reference-text
XML schema representation
XSLT rule
See?also:
 

Omitted?body?.?.?.?same?as?for??element
?
SQL input sample
??
??The?Meaning?Of?Life
??
Intermediate XML sample

The?Meaning?Of?Life
XHTML output sample
(see Figure 2, frame 5)
?See?also:
????The?Meaning?Of?Life
?See?also:
????MyOtherProc
Define a Main Section
The  element is a top-level container in which you would typically put most of the rest of your documentation. For example, the downloadable sample files generally contain a preamble or introduction, and then you’ll find several sections delineated with the  element: Details, Usage and Parameters, and Examples. Table 6 shows the details.
Table 6: The  Element: This top-level container lets you render clearly delineated sections in the output.
XML element
section-name“>
????section-text
XML schema representation
XSLT rule

 
 
 
SQL input sample
??
??
??This?function?provides?a?configurable,?flexible?interface?to?blah,?blah,?blah.
??
??
Blah
??
Blah
??
Intermediate XML sample
This?function?provides?a?configurable,?flexible?interface?to?blah,?blah,?blah.
Blah
Blah
XHTML output sample
(see Figure 2, frame 6)
Details
????
????This?function?provides?a?configurable,
????flexible?interface?to?blah,?blah,?blah.
????
????
Blah
????
Blah
Nutshell or Product Summary
This nutshell summary usually appears at the beginning of each documentation page to allow readers to grasp the salient points of the page in a few seconds. The top container is the  element, which has no attributes?just contents. The contents consists of a series of  or  children. For SQL pages, the examples use Module, Description, Environment, Author, and Since, as shown in Table 7. The XSLT elements provide a way to distill the information for the nutshell summary, but all they give you is an XHTML table. It is the CSS “nutshell” class wrapping the table that renders the sleek, professional nutshell box on the final web page.
Table 7: The  Element: This top-level element holds a series of child elements containing common header-area information such as Author, Module, and Description.
??????N?U?T?S?H?E?L?L
????
??
XML element

??
??item-name“>
????item-text
??
XML schema representation

See full-size image
XSLT rule

????????
??

??

????

????

??

??

????

????

??

??
??
SQL input sample??
??
??????SP_map
??
?????Maps?a?list?or?a?matrix?to?an?arbitrary
?????DML?statement
??
?????SQL?Server?2000/2005
??
?????Michael?Sorens
??
?????CleanCode?0.9.17
??Intermediate XML sample

??SP_map

???Maps?a?list?or?a?matrix?to?an?arbitrary
???DML?statement

???SQL?Server?2000/2005

???Michael?Sorens

???CleanCode?0.9.17
XHTML output sample

(see Figure 2, frame 7)??????

??????N?U?T?S?H?E?L?L

????

ModuleSP_map

????

Description

??????

Maps?a?list?or?a?matrix?to
?????????an?arbitrary?DML?statement

????

Environment

??????

SQL?Server?2000/2005

????

Author

??????

Michael?Sorens

????

Since

??????

CleanCode?0.9.17

??

At this point, you’ve covered the key XML elements for instrumenting a SQL file with documentation comments. You’ll find a few others in the XSLT file (translate.xsl), most of which are documented here.

Step 2?Run the Extractor
After you instrument your SQL files, the next step is to run the XML extractor sqldoc2xml.pl, which extracts the documentation from each SQL file into its own XML file. To execute sqldoc2xml.pl:

• Load Perl (5.6 or later). For Windows systems ActiveState offers a free Perl interpreter. For Linux, Perl comes standard.
• Load thel cleancode-perl open source library.
• Set your PERL5LIB environment variable to reference both the library from the previous bullet plus the lib-external directory provided with the sample project files. (See the perlrun manual page.)

Test the installation by invoking the program with no arguments or with -h or -H for usage information and documentation, as you see here:

   > cd sqlDocProjectcode   > perl sqldoc2xml.pl --h   Usage:       sqldoc2xml [ -lhH ] dirs target-directory      Options:       -h  Option help.       -H  Long help (manual page).       -l  List actions but do not make any changes (simulation).      

Following that syntax, experiment with the “sandbox” included in the sample project:

   > cd sqlDocProjectsandboxsqldoc2xml      > mkdir test-output-NEW      > perl ....codesqldoc2xml.pl test test-output-NEW

The sqldoc2xml.pl invocation reads the test directory as input and sends output to test-output-NEW. The sandbox includes a previously generated output directory called test-output, which you could compare your results to. This sample shows how sqldoc2xml handles an entire directory tree.

To actually process the real SQL files included with the sample project, use arguments similar to this:

   > perl sqldoc2xml.pl src XMLsrc   

This command extracts the XML documentation comments you have embedded in SQL source files in the src source directory and creates similarly named files in the XMLsrc target directory.

Step 3?Copy Support Files
With the XML output from sqldoc2xml in place in the XMLsrc directory, you can run XmlTransform. First, you need to make sure the XSLT file (translate.xsl, in the code directory in the sample download) that specifies how to map the XML to XHTML resides in the same place as the XML files. Simply copy that file into the XMLsrc directory and proceed with the next step.

Step 4?Run the Transformer
To execute XmlTransform:

• Load Java (version 1.5 or later). You only the need Java run-time engine, which you probably already have. If not, download it from Sun.
• Load the Xerces library for XML parsing.
• Load the Xalan library for XSLT transformations.
• Load the cleancode-java open source library.
• Add the following JAR files to your Java classpath: the CleanCode library (cleancode.jar), the Xerces library (xml-apis.jar and xercesImpl.jar), and the Xalan library (serializer.jar and xalan.jar).

Test the installation by invoking the program with no arguments or with ?help for further usage information. If you invoke XmlTransform with no parameters, you get a warning message:

   > cd sqlDocProjectcodecustomJar   > java com.cleancode.xml.XmlTransform   XmlTransform> +++ WARNING: must select at least one of these operations:   xslTransform, validateInputToSchema, validateOutputToSchema   Use '--help' option to see allowable parameters.   

Invoked with the ?help option, XmlTransform displays the myriad command-line options available. The main options are shown here, but you can see the full list in the related article “XmlTransform?A General-Purpose XSLT Pre-Processor:”

     contentsBaseName       -- base name for contents file (e.g. 'index')     contentsToParent       -- boolean indicating to place contents                               in parent or same dir     debug                  -- enable all diags if true (including libraries)     diagList               -- string of single-character diags to activate     dirList                -- comma-separated list of dirs to process     enable                 -- do processing if true; just report if false     generateContents       -- boolean switch to generate contents     generatorNode          -- tag name of node to replace with generator info     groupIdXpath           -- simple Xpath pointing to group id node in each file     groupPlaceHolder       -- tag name in contents file to put file list     help                   -- show this list     inExtension            -- extensions of files to process     inputSchemaSource      -- global Schema file for input validation     outExtension           -- extensions for translated files     outputSchemaSource     -- global Schema file for output validation     processAll             -- process without checking date stamps if true     sourcePath             -- root of source XML tree     startDepth             -- number of directories from the top of your tree back                               to your own relative root     targetPath             -- root of target XML tree     validateInputToSchema  -- boolean switch to validate input     validateOutputToSchema -- boolean switch to validate output     validateXslBySchema    -- boolean switch to validate needed XSL files     xslName                -- primary XSL file in each directory     xslParmList            -- comma-separated list of XSL parameters     xslSchema              -- name of Schema file for XSL files     xslTransform           -- boolean switch to do XSL translation

As you can see, XmlTransform has a lot of options. You may be familiar with the epigram, “With great power comes great responsibility.” The corollary here is: With great flexibility comes a great number of options. While I’m not going to go into great detail on XmlTransform, its options, and their effects in this article, you can find out more about how the general-purpose XmlTransform tool works, as well as more information on the various options in the article, “XmlTransform?A General-Purpose XSLT Pre-Processor.”

To experiment with the “sandbox” included in the sample project, try running these commands:

   > cd sqlDocProjectsandboxXmlTransformsameLevelIndexTest   > java com.cleancode.xml.XmlTransform @xdoc.conf      > cd sqlDocProjectsandboxXmlTransformparentLevelIndexTest   > java com.cleancode.xml.XmlTransform @xdoc.conf      > cd sqlDocProjectsandboxXmlTransformparentGroupIndexTest   > java com.cleancode.xml.XmlTransform @xdoc.conf   

Each of these commands transforms files from a source directory (srcdir) to a destination directory (destdir). Because of the large number of options available for XmlTransform, the options are stored in the different directories specified above in files named xdoc.conf. Each command uses a different options file, which changes the output as follows:

• The sameLevelIndexText example shows non-grouped items on the contents page; the contents page is stored in the same directory as the contents.
• The parentLevelIndexText example shows non-grouped items on the contents page as above, but the contents page itself is stored in the parent directory.
• The parentGroupIndexText example shows grouped items on the contents page; again, the contents page itself is stored in the parent directory.

Which version you’d want to use depends on how you prefer to arrange your source files and how you prefer to arrange your top-level index pages (those that provide lists of links to the documentation details for your SQL-stored procedures and functions).

XmlTransform and SQL Documentation
XmlTransform’s options are the key to controlling its output. Because there are so many options, it’s best to use an options file (as in the three examples near the end of the previous section) rather than typing options directly on the command line. The sample project includes XmlTransform_params.dat for this purpose. To actually process your files, use a command similar to this:

   > cd sqlDocProjectcode   > java com.cleancode.xml.XmlTransform @XmlTransform_params.dat   

The @ character preceding the option file name tells XmlTransform to treat the options in the file just as if they were typed on the command line itself.

To set up the options file, then, you need to determine which options you need by answering the following set of questions:

• What operation(s) do you want to perform? You may elect to validate the input files, validate the output files, perform a transformation from source to target, or generate contents files.

      Example:   --validateInputToSchema=false   --validateOutputToSchema=false   --xslTransform=true   --generateContents=true   

• Where are your source files? (This should be an absolute path to the root of your source XML directory tree.)

   Example:   --sourcePath=/MyProjects/sqlDocProject/code/XMLsrc   

• Where do you want to store the output HTML files? (This should be an absolute path to the root of your target directory tree.)

   Example:   --targetPath=/MyProjects/sqlDocProject/code/newdoc   

• What supplemental parameters have you defined in your XSLT file for which you need to supply values? The sample XSLT file translate.xsl defines three common parameters: a revision date (revdate), a copyright date (copyright), and a release version (relVersion). Having these parameters in the XSLT mapping means that if and when you need to change any of those, you change it in the XSLT file, rerun XmlTransform, and it updates every file in your hierarchy. The xslParmList option is a simple, comma-separated list of items.

   Example:   --xslParmList=revdate:2007.11.01,copyright:2007,relVersion:0.95   

• If you elect to generate contents (index) pages, do you want the contents page for each directory within the directory or elevated to its parent? Suppose you have a folder /source/abc that contains a contents template file /source/abc/_index.xml. This template file is unusual in several respects. It must be named “_index.xml”; there must be one such file per directory; and it actually goes through an extra step of translation, from a “precursor” XML file to a regular XML file, i.e. the contents page (still in XML, so still considered a source file). You can elect to place this generated contents page in the parent folder, which causes the contents file to assume the name of the folder. That allows you to have several subdirectories that elevate their contents to a common parent without contention. That is, the index page (/source/abc/_index.xml) maps to /source/abc.xml (and eventually to /target/abc.html). If you instead choose to place contents pages in the same directory as the set of files to which they link, then all the contents pages will be named “index.xml.” In that case, /source/abc/_index.xml would map to /source/abc/index.xml and ultimately to /target/abc/index.html. Elevating contents to the parent directory fits more naturally to a hierarchical model where you want to navigate up, down, and sideways in a web site. Putting index/content pages in the same directory, however, is a widely used convention, so XmlTransform provides the option to choose whichever you prefer.

   Example:   --contentsToParent=false   

• If you elected to generate contents pages, where do you want the contents inserted in your template? The precursor contents page contains XML with a special placeholder specifying where to insert the dynamically generated contents list. You can select an arbitrary XML element as a placeholder for the relevant group of files. That element?with no children and possibly one attribute (see next question)?controls where XmlTransform inserts summary information for the relevant group of files.

   Example:   --groupPlaceHolder=cc:files      

• If you selected to generate contents pages, and you wish to subdivide the files into more than one group, where is the group identifier in each file? In the simplest case, you need only provide a single element to list all files in one section in the contents page. You may include a group attribute and set its value to an empty string. However, by specifying a non-empty value for the group attribute you may subdivide the files into smaller groups. For SQL documentation, one natural group separation is between stored procedures and functions. In the contents template you specify a group attribute on the placeholder element for each group you wish to use. Here is a portion of the _index.xml file for the SQL documentation set:

   
StoredProcedures
      
Functions
      

You also need to instrument each source file to identify what group it belongs to. Stored procedure files should use, for example, sp while function files should use fn. These values are not fixed; you may use different values if you prefer as long as they match the group attribute of the cc:files elements (shown above).

You specify the name and location of these XML nodes with the groupIdXpath option. This example of a generated intermediate file (index.xml) shows how the template (_index.xml) is filled out with the appropriately tagged elements:

   
StoredProcedures
                     SP_map.html                        C:/usr/ms/devel/sql/doc/SP_map.xml                           
Functions
                     FN_list2table.html                        C:/usr/ms/devel/sql/doc/FN_list2table.xml                                    FN_list2tableIndexed.html                        C:/usr/ms/devel/sql/doc/FN_list2tableIndexed.xml                        

Observe how the list of three files has been split between the two groups based on the directives in each file that match the group attributes specified in the contents template file.

   Example:   --groupIdXpath=cc:cleanCodeDoc/cc:head/cc:group   

	?
Figure 4. Generating Groups on a Contents Page: Here are the main steps involved in the group generation process.

Figure 4 clarifies this process. Look for the elements in the representative set of source files (1). Those groups are called out in the contents template (2). In its first pass, XmlTransform weaves them together, filling out each group in the contents template with information from the source files identifying themselves as a member of that group (3). Its second pass then takes the filled out contents file and runs it through an XSLT transformation (4), along with all the other “regular” source files. This converts your extended XHTML language into XHTML renderable by any browser (5).

• If you elected to generate contents pages, where do you want to place the identification of the generated contents file? This is important because XmlTransform tries not to inadvertently overwrite material that it did not generate in the first place. The first time you run XmlTransform there is no issue, as no intermediate contents files exist. For subsequent runs, the generated index.xml file will exist. To avoid overwriting a handcrafted file of the same name, it looks for the presence of a node specified by the generatorNode option.

   Example:   --generatorNode=cc:generator   

• How much information do you want XmlTransform to report, and where do you want it delivered? XmlTransform uses a flexible diagnostic logging system that allows you to review as much or as little output about the execution as you wish. First you specify values for the possible diagnostics; XmlTransform has six available, designated A through F. The most useful approach is to specify powers of two so that you may combine the diagnostics as you please by bitwise ORing. Additionally, there are a couple other diagnostics that provide some useful output, one to show version numbers for each loaded module, and one to show values of all program options.

   Example:   // to show all module versions   VERSION_DIAG=0x800      // to show all configuration options   INPUTOPTIONS_DIAG=0x400      // to show program-specific details   XMLTRANSFORM_A_DIAG=0x2   XMLTRANSFORM_B_DIAG=0x4   XMLTRANSFORM_C_DIAG=0x8   XMLTRANSFORM_D_DIAG=0x10   XMLTRANSFORM_E_DIAG=0x20   XMLTRANSFORM_F_DIAG=0x40   

After defining values for each diagnostic, you then set the run-time diagnostic level (VERSION_DIAG) to a combination of these. A value of 10, for example, will activate diagnostics A and C (because 10 [1010 in binary] is the sum of the values for XMLTRANSFORM_C_DIAG [1000 in binary] and XMLTRANSFORM_A_DIAG [0010 in binary] ):

   Example:   DIAG_LEVEL=0x00A   

The final component for diagnostic setup is specifying where to deliver the output. You have four channels (stdout, stderr, logfile, and webpage) which may each be routed to two streams (standard output and standard error). Again, these are specified by enabling the appropriate bit combinations:

      Example:   // send to log file (8) and stdout (1)   OUTPUT_DIAG=9   // send to log file (8) and stderr (2)   OUTPUT_ERR=0xA

Run, XML, Run
After answering all the questions in the preceding section, collect the resulting options in your options file, and you’re ready to execute XmlTransform:

   > cd sqlDocProjectcode    > java com.cleancode.xml.XmlTransform       @XmlTransform_params.dat

Here’s the diagnostic output for the SQL documentation generation:

   Processing . (level=0)   XmlTransform.isGeneratedFile> scanning candidate contents:       index.xml   DocumentPlus> VERSION DocumentPlus 9   [contents] processing index.xml:       one or more files newer than output file   Generating contents file (3 files):       devxsqlDocsqlDocProjectXMLsrcindex.xml   Scanning contents template   Contents file complete:      devxsqlDocsqlDocProjectXMLsrcindex.xml   XmlTransform.dirHasChanged> files added in       /devx/sqlDoc/sqlDocProject/api/sql   [XSL] processing index.html:       input file is newer than output file   Loading XSL for level 0: translate.xsl   XslQualifier> VERSION XslQualifier 9   [XSL] index.html[~~FN_list2table.html]:       converted @ Wed Oct 24 14:05:38 PDT 2007    [XSL] processing FN_list2table.html:       output file does not yet exist   [XSL] FN_list2table.html[      index.html~~FN_list2tableIndexed.html]:       converted @ Wed Oct 24 14:05:39 PDT 2007   [XSL] processing FN_list2tableIndexed.html:       input file is newer than output file   [XSL] FN_list2tableIndexed.html[      FN_list2table.html~~SP_map.html]:       converted @ Wed Oct 24 14:05:39 PDT 2007   [XSL] processing SP_map.html:       files added or deleted in current directory   [XSL] SP_map.html[FN_list2tableIndexed.html~~]:       converted @ Wed Oct 24 14:05:39   PDT 2007   === Summary ===   start time: Wed Oct 24 14:05:38 PDT 2007   end time  : Wed Oct 24 14:05:39 PDT 2007   elapsed   : 0.79 seconds   max depth reached: 0   dirs processed   : 1   files processed  : 4   files transformed: 4   failed transform : 0   files validated  : 0   failed validation: 0   

There are only three SQL files and one contents file in the sample project so the output is quite brief. The diagnostic level set above (0x00A) provides the output shown. It includes an explanation for why each file is regenerated?I have highlighted these in bold. Try enabling different bits to see other available diagnostic output. You might, for instance, want to enable the INPUTOPTIONS_DIAG (0x400) to list all options used by the program, including both the ones you set and the default values for those you did not. Adding it to the current DIAG_LEVEL yields 0x40A. The XmlTransform API provides details on what each of the A through F diagnostics report on, as well as describing the complete set of XmlTransform options.

After generating your documentation set, you need only install it on your web server. When code changes inevitably come along, you can then quickly and easily regenerate an up-to-date documentation set with just a couple of simple commands.