Dynamic in-memory compilation using javax.tools

Once I needed to calculate expressions dynamically, so one option I got was dynamic in-memory compilation. I searched, and to my surprise, almost at the end of Java 6 (I am expecting Java 7 to be out soon…), I noticed this feature under javax.tools package. May be I am the last one to notice this!!. Similar feature is present in .net as well. But I had to leave this idea, because may be I thought there is some memory overhead, designed whole expression calculator application. But still let's see how dynamic compilation is possible in java.

This dynamic compiler API is included with Java 6 under javax.tools package.

How does it work?
javax.tools package has all the required interfaces and classes. Here, we will see how to compile a simple “HelloWorld” program source code stored in an in-memory String variable.
Able to compile a piece of source code stored in a string variable, WOW! this is interesting! isn’t it?
Follow the sequence of steps mentioned below. I explained these steps with the required code-snippets at that point. The full version of source code is available at the end of the article.

Important API's
The most important classes in this API are,

• JavaCompiler - This is used to create a compilation task
• JavaCompiler.CompilationTask – The compilation task, on which we execute compile operation using it’s call method
• JavaFileManager:Manages how the compiler read and writes to the files
• JavaFileObject: The file object that abstracts the java source and class files
• DiagnosticListener: This listens to the compilation diagnostic events
• ToolProvider: Which is used to get the compiler object from the underlying platform.

We will discuss these classes further in the example below. Let’s start…

Looking at the Example
1. Build the source code to compile; we can read it from file system, retrieve from database, or generate it dynamically in memory!!

Get the source code to be dynamically compiled ready:

StringBuilder src = new StringBuilder();
        src.append("public class DynaClass {\n");
        src.append("    public String toString() {\n");
        src.append("        return \"Hello, I am \" + ");
        src.append("this.getClass().getSimpleName();\n");
        src.append("    }\n");
        src.append("}\n");

Create a JavaFileObject instance for each of the compilation unit.

If the source is not from file system, then we need to write a class implementing from JavaFileObject interface. Java 6 provides a sample implementation of this in the form of SimpleJavaFileObject. We can extend from this and customize it as per our needs.

CharSequenceJavaFileObject implements the SimpleJavaFileObject interface and represents the source code we want to compile. Normally instances of SimpleJavaFileObject would point to a real file in the file system, but in our case we want it to represent a StringBuilder createdy by us dynamically. Let’s see how it goes:

import java.net.URI;

import javax.tools.SimpleJavaFileObject;
import javax.tools.JavaFileObject.Kind;

public class CharSequenceJavaFileObject extends SimpleJavaFileObject {

    /**
    * CharSequence representing the source code to be compiled
    */
    private CharSequence content;

    /**
    * This constructor will store the source code in the
    * internal "content" variable and register it as a
    * source code, using a URI containing the class full name
    *
    * @param className
    *            name of the public class in the source code
    * @param content
    *            source code to compile
    */
    public CharSequenceJavaFileObject(String className,
        CharSequence content) {
        super(URI.create("string:///" + className.replace('.', '/')
            + Kind.SOURCE.extension), Kind.SOURCE);
        this.content = content;
    }

    /**
    * Answers the CharSequence to be compiled. It will give
    * the source code stored in variable "content"
    */
    public CharSequence getCharContent(
        boolean ignoreEncodingErrors) {
        return content;
    }
}

If the source code is from file system, then create JavaFileObject instances from the File objects read from the file system.

/*Java source files read from file system*/
File []files = new File[]{file1, file2} ;
Iterable<? extends JavaFileObject> compilationUnits1 =
           fileManager.getJavaFileObjectsFromFiles(Arrays.asList(files1));

Though I am keeping the file object as in-memory one, ie not from file system.

Representing the compiled byte code

Next we must define the class representing the output of the compilation – compiled byte code. It is needed by the ClassFileManager which we will describe later. Compiler takes the source code, compiles it and splits out a sequence of bytes which must be stored somewhere. Normally they would be stored in a .class file but in our case we just want to make a byte array out of it. Here is a class that fulfills our needs:

import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.OutputStream;
import java.net.URI;

import javax.tools.SimpleJavaFileObject;

public class JavaClassObject extends SimpleJavaFileObject {

    /**
     * Byte code created by the compiler will be stored in this
     * ByteArrayOutputStream so that we can later get the
     * byte array out of it
     * and put it in the memory as an instance of our class.
     */
    protected final ByteArrayOutputStream bos =
        new ByteArrayOutputStream();

    /**
     * Registers the compiled class object under URI
     * containing the class full name
     *
     * @param name
     *            Full name of the compiled class
     * @param kind
     *            Kind of the data. It will be CLASS in our case
     */
    public JavaClassObject(String name, Kind kind) {
        super(URI.create("string:///" + name.replace('.', '/')
                + kind.extension), kind);
    }

    /**
     * Will be used by our file manager to get the byte code that
     * can be put into memory to instantiate our class
     *
     * @return compiled byte code
     */
    public byte[] getBytes() {
        return bos.toByteArray();
    }

    /**
     * Will provide the compiler with an output stream that leads
     * to our byte array. This way the compiler will write everything
     * into the byte array that we will instantiate later
     */
    @Override
    public OutputStream openOutputStream() throws IOException {
        return bos;
    }
}

At some point of the compilation, compiler will call openOutputStream() method of our JavaClassObject class and write there the compiled byte code. Because the openOutputStream() method returns a reference to the bos variable, everything will be written there, so that afterwards we will be able to get the byte code from it.

FileManager - putting the bytecode into JavaClassObject

We will also need something like a “file manager” that will tell the compiler to put the compiled byte code into an instance of our JavaClassObject class instead of putting it to a file. Here it is:

import java.io.IOException;
import java.security.SecureClassLoader;

import javax.tools.FileObject;
import javax.tools.ForwardingJavaFileManager;
import javax.tools.JavaFileObject;
import javax.tools.StandardJavaFileManager;
import javax.tools.JavaFileObject.Kind;

public class ClassFileManager extends
ForwardingJavaFileManager {
    /**
     * Instance of JavaClassObject that will store the
     * compiled bytecode of our class
     */
    private JavaClassObject jclassObject;

    /**
     * Will initialize the manager with the specified
     * standard java file manager
     *
     * @param standardManger
     */
    public ClassFileManager(StandardJavaFileManager
            standardManager) {
        super(standardManager);
    }

    /**
     * Will be used by us to get the class loader for our
     * compiled class. It creates an anonymous class
     * extending the SecureClassLoader which uses the
     * byte code created by the compiler and stored in
     * the JavaClassObject, and returns the Class for it
     */
    @Override
    public ClassLoader getClassLoader(Location location) {
        return new SecureClassLoader() {
            @Override
            protected Class<?> findClass(String name)
            throws ClassNotFoundException {
                byte[] b = jclassObject.getBytes();
                return super.defineClass(name, jclassObject
                        .getBytes(), 0, b.length);
            }
        };
    }

    /**
     * Gives the compiler an instance of the JavaClassObject
     * so that the compiler can write the byte code into it.
     */
    public JavaFileObject getJavaFileForOutput(Location location,
            String className, Kind kind, FileObject sibling)
    throws IOException {
        jclassObject = new JavaClassObject(className, kind);
        return jclassObject;
    }
}

Function getClassLoader() will be called by us to get a ClassLoader instance for instantiating our compiled class. It returns an instance of SecureClassLoader modified by the function findClass(), which in our case gets the compiled byte code stored in the instance of JavaClassObject, defines a class out of it with the function defineClass() and returns it.

Writing our dynamic compiler

import java.util.ArrayList;
import java.util.List;

import javax.tools.JavaCompiler;
import javax.tools.JavaFileManager;
import javax.tools.JavaFileObject;
import javax.tools.ToolProvider;

import com.vaani.compiler.files.CharSequenceJavaFileObject;
import com.vaani.compiler.files.ClassFileManager;


public class DynamicCompiler {
    private JavaFileManager fileManager ;
    private String fullName;
    private String sourceCode;
    
    /**
     * @param fullName_
       Full name of the class that will be compiled. 
       If class should be in some package,
       fullName should contain it too
       (ex. "testpackage.DynaClass")
     * @param SrcCode_
       Here we specify the source code of the class to be compiled

     */
    
    public DynamicCompiler(String fullName_, String SrcCode_){
        fullName = fullName_;
        sourceCode=SrcCode_;
        fileManager = initFileManager();
    }
    
    public JavaFileManager initFileManager(){
        if(fileManager!=null)
            return fileManager;
        else {
            JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();
             fileManager = new
                ClassFileManager(compiler
                    .getStandardFileManager(null, null, null));
             return fileManager;
        }
            
    }
    
    public void compile(){
           // We get an instance of JavaCompiler. Then
        // we create a file manager
        // (our custom implementation of it)
          JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();
          

            // Dynamic compiling requires specifying
            // a list of "files" to compile. In our case
            // this is a list containing one "file" which is in our case
            // our own implementation (see details below)
            List<JavaFileObject> jfiles = new ArrayList<JavaFileObject>();
            jfiles.add(new CharSequenceJavaFileObject(fullName, sourceCode));

            // We specify a task to the compiler. Compiler should use our file
            // manager and our list of "files".
            // Then we run the compilation with call()
            compiler.getTask(null, fileManager, null, null,
                null, jfiles).call();

    }
    
    public void run() throws InstantiationException, 
                IllegalAccessException, ClassNotFoundException{
          // Creating an instance of our compiled class and
        // running its toString() method
        Object instance = fileManager.getClassLoader(null)
            .loadClass(fullName).newInstance();
        System.out.println(instance);
    }
}

As you see the code we want to compile is stored in the variable sourceCode. Also our dynamic compiler object takes 2 parameters, the fully qualified classname and its sourceCode. In the constructor we get both initialized and then we initialize filemanager as well, which will hold the classes in it.

Lets, first focus on the compile method. After we define it, we print it to the console, get an instance of the compiler, put the source code into an object representing a source file.. The real compilation starts when we call the call() method of the compilation task. Then we get the Class representing our compiled class from the file manager, instantiate our class and print it to the console, using the toString() function that we implemented in the code.

There are three classes used in the code that are not available in the JDK and hence we have to implement them by ourselves – CharSequenceJavaFileObject, JavaClassObject and ClassFileManger.
These classes have already been implemented by us and explained as well. 

In the run method, we call getClassLoader() to get the class we need to get instance of.

Running the program

The main method

import com.vaani.compiler.DynamicCompiler;

import com.vaani.compiler.src.SourceCodes;

public class DynaCompTest {


    public static void main(String[] args) throws Exception {
        // Full name of the class that will be compiled.
        // If class should be in some package,
        // fullName should contain it too
        // (ex. "testpackage.DynaClass")
        String fullName = SourceCodes.strDynaClassFullName;

        // Here we get and specify the source code of the class to be compiled     
        String src = SourceCodes.getDynaClassSource();
        
        DynamicCompiler uCompiler = new DynamicCompiler(fullName, src);
        uCompiler.compile();
        uCompiler.run();

    }
}

Output

Now that we have all our classes ready, lets compile them and run the program. We should get an output like this:

public class DynaClass {
    public String toString() {
        return "Hello, I am " + this.getClass().getSimpleName();
    }
}

Hello, I am DynaClass

Application of dynamic compilation
As introduced in the beginning, I needed this for expression calculator, for expressions such as “y=2*(sin(x)+4.0)”. Using dynamic compilation you don’t have to parse it any more by yourself, you could just compile it and get a fast, optimized function representing this expression. You can read about it (and much more about dynamic compilation generally) here.

Some other usage is creating dynamic classes for accessing data stored in JavaBeans. Normally you would have to use reflection for it, but reflection is very slow and its generally better to avoid using it when possible. Dynamic compilation allows you to minimize the use of reflection in a library that handles JavaBeans. How? We will try to show it in one of our next posts, so stay tuned!