Extend your Java Application with Embedded Languages

hat happens when a potential customer goes through the feature list for your application and finds just one or two missing items? Do you have a mechanism for them to extend your application to handle those features? Or will you lose the sale to an application that can be extended? Embedded scripting languages can turn a static vertical application into a dynamic application platform that will adapt rapidly to changing customer requirements.

And lucky for you Java is an extremely popular platform for embedded scripting languages. Why? Because the garbage collection and reflection capabilities of the Java language and framework make it easy to build parsers, compilers, and interpreters that work on any platform and can make use of the Java framework.

In this article I will discuss how you can use these embedded languages to add real customer value to your application, and then demonstrate the use of three different embedded languages in an example email application.

Let’s start with the basics. A scripting language engine parses, compiles, and executes scripts at runtime. The host application then feeds the scripting engine chunks of script code to compile and execute. This script code can make method calls back to the host application through objects that have been injected into the script context.

One of the most popular examples of built-in scripting language support is Microsoft’s support for custom macros in Office. These macro scripts access injected objects that represent the active document to add words or make changes to spreadsheets with arbitrarily complex logic. Whole business applications have been built entirely in the scripting framework. Which proves the benefit of a dynamic scriptable application framework in a business environment.

All of the scripting engines discussed in this article are capable of providing the same level of facility.

The Customer Perspective
The customer advantages of scripting support are clear:

  • The application can be extended and customized to add missing features or to achieve a higher level of integration with the business environment.
  • Business logic or flow can be customized without returning to the vendor for modifications.
  • Repetitive tasks can be automated to increase productivity.

Support for a scripting language in an application can also aid us as developers:

  • We can rapidly prototype new features in the scripting language and then roll them back into the Java when it makes sense for efficiency reasons.
  • It’s simple to implement a shell?operating within the context of a running application?that allows you to inspect and modify the application’s object model.
  • It’s much easier to provide an application facility for what-if scenarios with run-time evaluated formulae.

That’s just a taste of the power that embedded scripting languages can supply us as developers as well as our customers and end users.

Scripting Engines: The Process Flow
A standard architecture for the flow of control between an application and a scripting engine is shown in Figure 1.

Figure 1. Embedding the Engine: The diagram shows how a scripting engine can be embedded into an application using proxy objects.

The application hosts a scripting engine. The application provides the engine with a set of scripts, which it then compiles into its script context. When the user instructs the application to execute a command that is implemented by a macro, the application follows this workflow:

  • It sets up the script environment with proxy objects that represent the relevant portions of the application’s object model.
  • The scripting engine is instructed to execute the method or function that corresponds to the application event.
  • The running script then implements the appropriate command behavior by using the proxy objects to modify the application object model.
  • The method or function in the script ends and control is returned to the application.

This sequence is exactly what Microsoft Word and Excel do when they execute a macro.

Another common use for a scripting engine is for providing a modular extension mechanism. One classic example is an application that uses scripts to provide a customer with extensible input or output mechanism (see Figure 2).

Figure 2. Translation: In this architecture the scripting engine acts as a translation mechanism.

In this architecture the application uses the script as an extensible data translation mechanism. A practical example would be an Address Book application. The application can ship with several canned import mechanisms for the most common data sources, but it can also ship with an extensible scripting mechanism that the customer can use to build support for any data source.

One small tweak in this model is the removal of the proxy objects. None of the scripting engines demonstrated in this article are dependent on proxy objects; they are strictly optional. They use Java’s reflection capability to talk directly to any object. Proxies are not required as they are when integrating C++ object models with scripting engines. Proxies do provide value in that they restrict the access that a script can have to the object model. So while they are not necessary, proxies may help you sleep a little better at night, knowing that your customers don’t have access to every method in your object model.

Evaluating Scripting Engines
What are we looking for in a scripting engine?

  • The scripting syntax should be familiar and simple.
  • The language should be latently typed to make it easier for customers (and developers) to use.
  • There should be a way to inject objects into the script context.
  • The script should be able to read and write data and invoke methods on the injected objects.
  • The injected objects should appear and work naturally within the language.
  • There should be robust error handling to make it easy to debug the scripts.
  • It should be easy to print out debug messages in case debugging is necessary and there is no support for single-step debugging.
  • The engine should be thread-safe and provide for multiple script contexts.
  • The engine and language should be powerful enough to talk to frameworks outside of the application. This allows the script to act as a bridge between applications.

The three scripting engines in this article meet and exceed these criteria. So let’s get introduced:

  • Jython?This is a Java implementation of the very popular Python programming language. The engine provides access to the power of Python and its code base as well as seamless access to any of the Java framework APIs. It’s a highly evolved project (now in version 2.1), with a stable, well-tested code base. O’Reilly publishes a book specifically on Jython as well as the fundamental resource Programming Python.
  • BeanShell?This language is similar in form and syntax to Java, which makes it an ideal language for developers. The BeanShell interpreter is loosely typed by default but can be set to support only strongly typed macros where variables are predefined. Like Jython, the BeanShell interpreter is actively developed and extremely stable. The language syntax is amply documented on the BeanShell Web site, as is the JavaDoc of the interpreter’s API. BeanShell also has the advantage of being contained within a single .jar file, which makes it very easy to integrate.
  • Groovy?Groovy’s fan base is growing fast thanks to a very clean syntax, similar in many way to the popular Ruby programming language. Groovy is attempting to make a standard of itself as JSR 241. Groovy is still an emerging language, but it has a novel syntax and a lot of a promise.

Now that you know a little about the scripting engines, I can introduce the example application, a flexible spam killer.

Sample App: A Flexible Spam Killer
The example application for this article is an e-mail application with spam filtering. Spam changes fast so we want to make it easy for people to extend their filtering to block out new types of spam. The application should have a simple filter configuration dialog, while also providing a way of creating a much more focused filter through a scripting language.

To test the application you’ll need two pieces of mail. One telling you that you’ve been accepted for a new job, which you want. And one trying to sell you Vialis, which you don’t want. Vialis just came out, so the pre-configured filters in the application don’t catch it. To cut down on the Vialis spam you obviously want to identify any mail that references ‘Vialis’ in the message body as spam.

The application architecture is fairly simple; it maintains simple Mail objects:

public class Mail{    public String subject;    public String message;        public Mail( String subject, String message ) {         this.subject = subject;        this.message = message;    }}

The application will send these mail message objects to the script. The job of the script is to determine whether the message is spam. When a mail comes in the system invokes the script with the mail message object (see Figure 3). The method will return true if the message is spam.

Figure 3. Spam Chute: The diagram shows the flow of the spam filter through the scripting engine.

My first implementation of the scriptable spam filter is in Groovy.

Groovy
After installing Groovy and adding its JAR files to the project you can access the groovy.lang namespace and use the Groovy interpreter:

import groovy.lang.GroovyShell;import groovy.lang.Binding;import groovy.lang.Closure;import java.io.File;public class MailApp{     private GroovyShell _shell;        public MailApp() { }

The first step is to load the interpreter with the script, which is located in the file user.groovy. Groovy makes creating a Groovy interpreter and loading a file as simple as two commands:

    public void loadInterpreter() throws Exception    {        // Create the groovy shell        _shell = new GroovyShell( new Binding() );        // Read in the script and evaluate it        _shell.evaluate( new File( "user.groovy" ) );    }

With the interpreter in hand you can now wrap the invocation of the isSpam closure (the Groovy name for a function) in a simple boolean isSpam call. First you get a reference to the isSpam closure. Then invoke the closure with the mail object and get the return value. You’ll then return the boolean return value from the script:

    public boolean isSpam( Mail mail )    {        // Get the isSpam closure        Closure isSpam = (Closure)_shell.getVariable( "isSpam" );        // Run the closure on the mail object        Boolean output = (Boolean)isSpam.call( mail );        // Coerce the return value back to a basic boolean        return output.booleanValue();    }

The main function creates the mail application, loads the interpreter, and then tests both the good and bad messages to see if they are spam:

    public static void main( String []args ) throws Exception    {         MailApp m = new MailApp();                m.loadInterpreter();                Mail badMail = new Mail( "Hi!", "You will love our vialis!" );        Mail goodMail = new Mail( "Hi!", "You get the job!" );                System.out.println( "Bad mail is spam = " + m.isSpam( badMail ) );        System.out.println( "Good mail is spam = " + m.isSpam( goodMail ) );    }}

The Groovy script file is shown below:

import java.util.regex.Patternimport java.util.regex.MatcherisSpam = { mail |        p = Pattern.compile( "(vialis)" );        m = p.matcher( mail.message );        return m.find();}

It would be difficult for it to be simpler. You create an isSpam variable, which is a closure (function). It takes one argument: the mail message object. Your custom code then compiles a regular expression, matches it against the message of the mail, and returns true if it finds a match.

From this example you can see the fundamentals of embedding scripting languages:

  • Creating an instance of the interpreter.
  • Loading a script into the interpreter.
  • Executing functions or methods within the script.
  • Parsing the return value from the engine.

You could evaluate the script on each invocation, but that would not be as efficient.

Python
The first step in using Jython is installing the interpreter, which is done through a Swing UI application, then including the Jython JARs in your project. With that done you can reference the basic PythonInterpreter class and the PyObject class, which represents a Jython object.

import org.python.util.PythonInterpreter; import org.python.core.PyObject; public class MailApp{     private PythonInterpreter _interp;        public MailApp() { }

The first step is to create an instance of the interpreter and then load the script file.

    public void loadInterpreter()    {        // Create the interpreter        _interp = new PythonInterpreter();        // Load the script file        _interp.execfile( "user.py" );    }

You can invoke the spamFilter function by getting the function from the interpreter context and then using the __call__ method to call the function with the mail object. Once you have the result of the call in hand you can convert it to an integer and return true or false based on the value.

    public boolean isSpam( Mail mail )    {                // Get the function object            PyObject spamFilter = _interp.get( "spamFilter" );                // Set a module global to our mail object        _interp.set( "mail", mail );                        // Run the spamFilter with our mail object            PyObject result = spamFilter.__call__( _interp.get( "mail" ) );                        // Print the result        Integer realResult = (Integer)result.__tojava__( Integer.class );        // Coerce the integer return value to a boolean            return realResult.intValue() == 0 ? false : true;    }

Notice that in the code above I didn’t have to proxy or wrap the mail object. I just send it into the context and Jython does all of the work to reflect against the object and integrate it into the environment.

The final Java section of the example invokes the spamFilter method on the example mail objects.

    public static void main( String []args ) throws Exception    {         MailApp m = new MailApp();                m.loadInterpreter();                Mail badMail = new Mail( "Hi!", "You will love our vialis!" );        Mail goodMail = new Mail( "Hi!", "You get the job!" );                System.out.println( "Bad mail is spam = " + m.isSpam( badMail ) );        System.out.println( "Good mail is spam = " + m.isSpam( goodMail ) );    }}

The Python (or Jython) code for the spamFilter is shown below:

import redef spamFilter( mail ):        filter = re.compile( r"vialis" )        if ( filter.search( mail.message ) ):                return 1        return 0

This would run just fine under Python. Which is one of the great advantages of Jython. I should also point out that I used the Python regular expression library here but I just as easily could have used the Java regular expression library to do the matching. Of course, if I had done that, the script would not be executable by Python.

BeanShell
BeanShell is the easiest to install, and has the simplest interface of the three embedded languages I’ve demonstrated in this article. The installation amounts to putting a single .jar file in your project or extensions directory. Then to use it involves just one object.

import bsh.Interpreter;import java.io.BufferedReader;import java.io.FileReader;import java.io.File;public class MailApp{     private Interpreter _interp;        public MailApp() { }

The first step is to build the interpreter and to load it with the script source. Unlike the other languages you have to do the file I/O yourself, but that’s no big deal.

    public void loadInterpreter() throws Exception    {        // Create the interpreter        _interp = new Interpreter();        // Read in the script        File inFile = new File( "user.bsh" );        BufferedReader reader = new BufferedReader( new FileReader( inFile ) );            StringBuffer userScript = new StringBuffer();        String line;            while( ( line = reader.readLine() ) != null ) { userScript.append( line ); }            // Evaluate the script into the current context        _interp.eval( userScript.toString() );    }

Next, wrap your invocation of the isSpam function by injecting the mail object into the context. Then evaluate the invocation of the call and coerce the return value to a boolean.

    public boolean isSpam( Mail mail ) throws Exception    {        // Set the active mail object        _interp.set( "mail", mail );        // Run the script and get the result back        Boolean isSpam = (Boolean)_interp.eval( "isSpam( mail );" );        // Coerce the value to the boolean return value        return isSpam.booleanValue();    }

I couldn’t find a way to access the function to call it directly. Because I use eval I pay a hefty performance penalty with the compilation of the text to running code.

The final Java step, as with all of the previous examples, is to test the spam filter by sending it the two different e-mails.

    public static void main( String []args ) throws Exception    {         MailApp m = new MailApp();                m.loadInterpreter();                Mail badMail = new Mail( "Hi!", "You will love our vialis!" );        Mail goodMail = new Mail( "Hi!", "You get the job!" );                System.out.println( "Bad mail is spam = " + m.isSpam( badMail ) );        System.out.println( "Good mail is spam = " + m.isSpam( goodMail ) );    }}

The BeanShell code is very similar in flavor to Java:

import java.util.regex.Pattern;import java.util.regex.Matcher;boolean isSpam( mail ){        p = Pattern.compile( "(vialis)" );        m = p.matcher( mail.message );        return m.find();}

Because it’s a scripting language BeanShell supports macro (a simple list of commands), procedural, and object-oriented programming paradigms. In this case I chose to use a function but the API could have easily been OO. In fact you can define interfaces in Java and implement them in BeanShell.

As you can see from the different variations the languages may change but the basic structures and data flow remains the same.

Hints and Tips
Embedded languages are powerful and as with all powerful things they can be misused and abused. Here’s some hints and tips that may help you avoid the pitfalls:

  • Pay attention to performance?I did some performance testing with these three examples and the results were a little shocking. Jython was only three times slower than hard coding. Groovy was six times slower. And BeanShell, most likely because of the eval in the code, was much slower than Groovy. There are two lessons to learn: The first is that you need to use embedded languages only when you need flexibility over performance, and second, that you need to optimize how you use the interpreter to avoid excessive compilation.
  • Sourcing the Scripts?I used flat files in this example, but there is no reason that you can’t construct script code as a string in memory, or use XML to store script fragments. The only particular advantage of storing scripts as flat text files is that you can get code coloring on the language if your editor supports it.
  • Dropping the context?You can refresh the script context with new code by dropping the interpreter and replacing it with a new one where you load in new code. If you are writing a desktop application you should probably drop and reload the context if the timestamp on the script file has changed. This allows the user to modify the scripts without having to restart your application.
  • Don’t make everything extensible?You should make it clear in your architecture and to your users where and how the application should be extended. If you take the position that ‘everything is extensible’ then you will have problems maintaining compatibility between releases. In addition, your users won’t have any clue as to the recommended way to extend the system. So they probably won’t extend the system. Apache has been successful because the architecture of the framework made it clear how and where it should be extended. As such, the Apache implementers can make modifications without too much fear of breaking the extension modules.
  • Inverting the design?Another potential extensible application architecture uses the script as the central processing flow. The value of the application is in the objects that it injects into the script context. An example would be a version of Ant where the build.xml file is replaced with a script. The Ant tasks would simply be objects that are injected into the script context.
  • Not just Java?The are some embedded scripting languages for C and C++. Most notable is the Mozilla JavaScript engine. Also VB.NET can be used in a dynamic way through use of the .NET Framework.

With some examples and some tips and tricks in hand we can wrap up this article and get on with implementing this stuff!

There are lots of places where you can apply a scripting language in an application. Below are just a few ideas to whet your appetite.

  • In an ASP model you could allow the user to provide custom business logic through macros, which are run in a sandbox with only limited access to customer records.
  • As an extension layer to input or output processing.
  • To provide an extensible reporting mechanism.
  • In an editor (e.g. Eclipse) to provide a macro mechanism.
  • Providing application integration for your customers.
  • As a rapid-prototyping environment for user interface construction.
  • Providing an extension mechanism where the user can add new components.
  • If your product is a library you can have your QA team use a scripting language in a standalone mode to stress-test the library.

This list just scratches the surface of the potential customer-facing functionality that can be provided by macros. However, the most fun reason, which I haven’t mentioned, is that you get to learn a fun new language and to experiment with it within the familiar context of your own application.

Unlike C++ there are no barriers or hard work involved in using scripting languages with your Java application. So pick up one of these languages and take it for a spin. Before you know it your application will go from static, slow-to-compile beast to an agile, extensible, rapid-prototyping platform.

Share the Post:
Share on facebook
Share on twitter
Share on linkedin

Overview

Recent Articles: