Creating a Twitter Data Reader
In order to explore the potential of the Twitter XML data, I built a RESTful data application around an XML database (eXist-db, though others will work as well) that would illustrate both how to use the Twitter API and, as importantly, how to make such an API work using virtual collections. The web application itself is relatively simple. It periodically queries the Twitter API for new status messages (tweets) from a given user's account (mine in this case) and downloads them to the database as XML, combining the XML and Atom formats to give access to both.
The application's site features a table that provides paging and shows the most recent tweets (see Figure 1). This page is generated using an XProc pipeline that calls XQuery scripts to format the output, reinterpreting a RESTful URL such as http://myaddress/exist/xrx/twitter.table to render the Twitter combined feed as an XHTML table. Changing this to twitter.list will change the output to a list of entries (see Figure 2), while twitter.xml will retrieve the raw XML content for that page (this is still in development at the time of this writing).
Figure 1. Twitter Table: The application's site features a table that provides paging and shows the most recent tweets.
Figure 2. Twitter List: Changing the RESTful URL to twitter.list will change the output to a list of entries.
If you click on a person's screen name (their @name), the app will use the same presentation face to show all entries in descending chronological order by that person (see Figure 3).
Finally, if you click on a hash entry (such as #sxsw for the South by Southwest conference), it will display in reverse chronological order all tweets that include that hash (see Figure 4).
Figure 3. Entries for @guykawasaki: The application's site features a table that provides paging and shows the most recent tweets.
Figure 4. Entries for #sxsw: Changing the RESTful URL to twitter.list will change the output to a list of entries.
Setting up eXist
The benefit of the XML formats may not be obvious to the AJAX crowd. XML support generally is minimal in most browser systems. Even where it does exist, the XML tools are often complicated and awkward to use for development. However, the ability to consume XML content does make far more sense on the server side, especially with regard to XML databases and similar tools, particularly when used in conjunction with XQuery.
The open source eXist database is a good platform to explore such applications. Written in Java, eXist has been around since 2000, though has seen most of its development (and adoption) since 2006. The current stable branch (1.2.5) is generally recommended for most production sites, while the newer 1.3.x line represents the bulk of the current development (and requires that you use SVN to install the application).
eXist can be installed on Windows, Linux, Solaris, and Macintosh computers, though it requires that Java 1.5 is currently available on the system. Additionally, eXist seems to work best with 1.5 to 2GB of memory available and at least 100MB of free disk-space. The default installation can be launched directly from the eXist installer, which provides a graphical interface for setting up the database and authorization codes.
The code contained here makes use of one feature that is somewhat developed in the 1.2 branch but is more completely handled in the 1.3.x branch: the ability to perform URL rewriting (in which an XQuery becomes the "router" for a specific set of services). If you wish to use the 1.3 version (not covered here), then you have to specifically pull it from an SVN development branch until 1.3 is formally released as a "stable" product. To do so, point your SVN client to:
svn co https://exist.svn.sourceforge.net/svnroot/exist exist
Once downloaded, move the folder to the desired location (such as the Program Files folder in Windows, or /opt in Linux), switch to superuser status, and run $exist/installer/scripts/setup.bat or $exist/installer/scripts/setup.sh respectively, depending upon the flavor of your operating system.
After eXist is installed and started (which should happen automatically), you can see the eXist startup page at http://localhost:8080/exist (replace localhost with the domain name or IP address of the host computer in order to get access).
You can run eXist as a service in the background. In Windows, simply select Install eXist as a Service from the Start Menu options. In Linux, it's generally better to run bin/startup.sh in order to run it in process, or $exist/bin/startup.sh& to run it in the background.
The final step is enabling XPath extensions. In the 1.2 builds, for performance reasons many of the advanced XPath extensions are not enabled. To enable them, stop the server (best done via the Java client in Windows or by launching it in Linux via $exist/bin/client.sh) and then open the $exist/conf.xml file for editing. In this file, about two-thirds of the way down, you'll find the code in Listing 4.
The optional modules are currently commented out, but you can enable them by removing the comment tags (>!-- -->). Once you've done this, save the file and then run $exist/build.bat (or $exist/build.sh in Linux) in order to rebuild the JAR file. Note that you should do this only after you have done the initial setup and run eXist at least once.