Asynchronous calls and remote callbacks using Lingo Spring Remoting

http://www.jroller.com/sjivan/entry/asynchronous_calls_and_callbacks_using

Asynchronous calls and remote callbacks using Lingo Spring Remoting

As mentioned in my previous blog entry, Lingo is the only Spring Remoting implementation that supports asynchronous calls and remote callbacks. Today I'll cover all the nitty gritty details of the async/callback related functionality along with the limitations and gotchas.

Asynchronous method invocation and callback support by Lingo is an awesome feature and there are several usecases where these are an absolute must. Lets consider a simple and rather common use case : You have a server side application (say an optimizer) for which you want you write a remote client API. The API has methods like solve() which are long running and methods like cancel() which stops the optimizer solve.

A synchronous API under such circumstances is not really suitable since the solve() method could take a really long time to complete. It could be implemented by having the client code spawn their own thread and do its own exception management but this becomes really kludgy. Plus you have to worry out network timeout issues. You might be thinking "I'll just use JMS if I need an asynchronous programming model". You could use JMS but think about the API you're exposing. Its going to be a generic JMS API where the client is registering JMS listeners, and sending messages to JMS destinations using the JMS API. Compare this to a remote API where the client is actually working with the Service interface itself.

Lingo combines the elegance of Spring Remoting with the ability to make asynchronous calls. Lets continue with our Optimizer example and implement a solution using Lingo and Spring. OptimizerService interface

public interface OptimizerService {
    void registerCallback(OptimizerCallback callback) throws OptimizerException;
 
    void solve();
 
    void cancel() throws OptimizerException;
}

The solve() method is asynchronous while the cancel() and registerCallback(..) methods are not. Asynchronous methods by convention must not have a return value and also must not throw exceptions. The registerCallback(..) method registers a client callback with the Optimizer. In order to make an argument be a remote callback, the argument must implement java.util.EventListener or java.rmi.Remote. In this example the OptimizerCallback interface extends java.util.EventListener. If the argument does not implement either of these interfaces, it must implement java.io.Serializable and it will then be passed by value.

OptimizerCallback interface

public interface OptimizerCallback extends EventListener {
 
    void setPercentageComplete(int pct);
 
    void error(OptimizerException ex);
 
    void solveComplete(float solution);
}

The callback API has a method for the Optimizer to set the percentage complete, report an error during the solve() process (remember that the solve() method is asynchronous so it cannot throw an exception directly) and finally the solveComplete(..) callback to inform the client that the solve is complete along with the solution.

OptimizerService implementation

public class OptimizerServiceImpl implements OptimizerService {
 
    private OptimizerCallback callback;
    private volatile boolean cancelled = false;

 
    private static Log LOG = LogFactory.getLog(OptimizerServiceImpl.class);
 
    public void registerCallback(OptimizerCallback callback) {
        LOG.info("registerCallback() called ...");
        this.callback = callback;
    }

 
    public void solve() {
        LOG.info("solve() called ...");
        float currentSolution = 0;
 
        //simulate long running solve process
        for (int i = 1; i <= 100; i++) {
            try {
                currentSolution += i;
                Thread.sleep(1000);
                if (callback != null) {
                    callback.setPercentageComplete(i);
                }
                if (cancelled) {
                    break;
                }
            } catch (InterruptedException e) {
                System.err.println(e.getMessage());
            }
        }
        callback.solveComplete(currentSolution);

 
    }
 
    public void cancel() throws OptimizerException {
        LOG.info("cancel() called ...");
        cancelled = true;
    }
}

The solve() method sleeps for a while and makes the call setPercentageComplete(..) on the callback registered by the client. The code is pretty self explanatory here.

Optimizer Application context - optimizerContext.xml We now need to export this service using Lingo Spring Remoting. The typical Lingo Spring configuration as described in the Lingo docs and samples is :

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE beans PUBLIC "-//SPRING//DTD BEAN//EN" "http://www.springframework.org/dtd/spring-beans.dtd">
<beans>    <bean id="optimizerServiceImpl" class="org.sanjiv.lingo.server.OptimizerServiceImpl" singleton="true"/>
    <bean id="optimizerServer" class="org.logicblaze.lingo.jms.JmsServiceExporter" singleton="true">        <property name="destination" ref="optimizerDestination"/>        <property name="service" ref="optimizerServiceImpl"/>        <property name="serviceInterface" value="org.sanjiv.lingo.common.OptimizerService"/>        <property name="connectionFactory" ref="jmsFactory"/>    </bean>
    <!-- JMS ConnectionFactory to use -->    <bean id="jmsFactory" class="org.activemq.ActiveMQConnectionFactory">        <property name="brokerURL" value="tcp://localhost:61616"/>        <property name="useEmbeddedBroker">            <value>true</value>        </property>    </bean>
    <bean id="optimizerDestination" class="org.activemq.message.ActiveMQQueue">        <constructor-arg index="0" value="optimizerDestinationQ"/>    </bean></beans>

In this example, I'm embedding a JMS broker in the Optimizer process. However you are free to use an external JMS broker and change the JMS Connection Factory configuration appropriately.

Note : The above optimizerContext.xml it the typical configuration in the Lingo docs/examples
but is not the ideal configuration. It has some serious limitations which I'll cover in a bit
along with the preferred  "server" configuration.

OptimizerServer The "main" class that exports the OptimizerService simply needs to instantiate the "optimizerServer" bean in the optimizerContent.xml file.

public class OptimizerServer {
 
    public static void main(String[] args) {
        if (args.length == 0) {
            System.err.println("Usage : java org.sanjiv.lingo.server.OptimizerServer <config file>");
            System.exit(-1);
        }
        String applicationContext = args[0];

 
        System.out.println("Starting Optimizer ...");
        FileSystemXmlApplicationContext ctx = new FileSystemXmlApplicationContext(applicationContext);
 
        ctx.getBean("optimizerServer");
 
        System.out.println("Optimizer Started.");

 
        ctx.registerShutdownHook();
    }
}

The Client In order for the client to lookup the remote OptimizerService, we need to configure the client side Spring application context as follows : Client Application Context - clientContext.xml

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE beans PUBLIC "-//SPRING//DTD BEAN//EN" "http://www.springframework.org/dtd/spring-beans.dtd">
<beans>    <bean id="optimizerService" class="org.logicblaze.lingo.jms.JmsProxyFactoryBean">        <property name="serviceInterface" value="org.sanjiv.lingo.common.OptimizerService"/>        <property name="connectionFactory" ref="jmsFactory"/>        <property name="destination" ref="optimizerDestination"/>
        <!-- enable async one ways on the client -->        <property name="remoteInvocationFactory" ref="invocationFactory"/>    </bean>
    <!-- JMS ConnectionFactory to use -->    <bean id="jmsFactory" class="org.activemq.ActiveMQConnectionFactory">        <property name="brokerURL" value="tcp://localhost:61616"/>    </bean>
    <bean id="optimizerDestination" class="org.activemq.message.ActiveMQQueue">        <constructor-arg index="0" value="optimizerDestinationQ"/>    </bean>
    <bean id="invocationFactory" class="org.logicblaze.lingo.LingoRemoteInvocationFactory">        <constructor-arg>            <bean class="org.logicblaze.lingo.SimpleMetadataStrategy">                <!-- enable async one ways -->                <constructor-arg value="true"/>            </bean>        </constructor-arg>    </bean></beans>

Now all a client needs to do to is obtain a handle of the remote OptimizerService by looking up the bean "optimizerService" configured in clientContext.xml.

OptimizerCallback implementation Before going over the sample Optimizer client code, lets first write a sample implementation of the OptimizerCallback interface - one which the client will register with the remote Optimizer by invoking the registerCallback(..) method.

public class OptimizerCallbackImpl implements OptimizerCallback {
 
    private boolean solveComplete = false;
    private OptimizerException callbackError;
    private Object mutex = new Object();

 
    public void setPercentageComplete(int pct) {
        System.out.println("+++ OptimzierCallback :: " + pct + "% complete..");
    }
 
    public void error(OptimizerException ex) {
        System.out.println("+++ OptimzierCallback :: Error occured during solve" + ex.getMessage());
        callbackError = ex;
        solveComplete = true;
        synchronized (mutex) {
            mutex.notifyAll();
        }
    }

 
    public void solveComplete(float soltion) {
        System.out.println("+++ OptimzierCallback :: Solve Complete with answer : " + soltion);
        solveComplete = true;
        synchronized (mutex) {
            mutex.notifyAll();
        }
    }
 

    public void waitForSolveComplete() throws OptimizerException {
        while (!solveComplete) {
            synchronized (mutex) {
                try {
                    mutex.wait();
                    if (callbackError != null) {
                        throw callbackError;
                    }
                } catch (InterruptedException e) {
                    e.printStackTrace();
                    break;
                }
            }
        }
    }
}

OptimizerClient

public class OptimizerClient {
 
    public static void main(String[] args) throws InterruptedException {

 
        if (args.length == 0) {
            System.err.println("Usage : java org.sanjiv.lingo.client.OptimizerClient <clientContext.xml>");
            System.exit(-1);
        }
 
        String applicationContext = args[0];
        FileSystemXmlApplicationContext ctx = new FileSystemXmlApplicationContext(applicationContext);
 
        OptimizerService optimizerService = (OptimizerService) ctx.getBean("optimizerService");
        OptimizerCallbackImpl callback = new OptimizerCallbackImpl();

 
        try {
            optimizerService.registerCallback(callback);
            System.out.println("Client :: Callback Registered.");
 
            optimizerService.solve();
            System.out.println("Client :: Solve invoked.");
 
            Thread.sleep(8 * 1000);
            System.out.println("Client :: Calling cancel after 8 seconds.");

 
            optimizerService.cancel();
            System.out.println("Client :: Cancel finished.");
            //callback.waitForSolveComplete();
 
        } catch (OptimizerException e) {
            System.err.println("An error was encountered : " + e.getMessage());
        }
    }
}

The test client registers a callback and calls the asynchronous method solve(). Note that the solve method in our sample OptimizerService implementation takes ~100 seconds to complete. The client then prints out the message "Client :: Solve invoked.". If the solve() call is indeed invoked asynchronously by Lingo under the hoods, this message should be printed to console immediately and not after 100 seconds. The client then calls cancel() after 8 seconds have elapsed.

Here's the output when we run the Optimizer Server and Client

Notice that the solve method has been called asynchronously and after 8 seconds the client makes the cancel() call however the server does not seem to be receiving this call and continues with its setPercentageComplete(..) callback.

I asked this question on the Lingo mailing list but did not get a response. This misbehaviour was pretty serious because what this meant was that while an asynchronous call like solve() was executed asynchronously by the client, the client was not able to make another call like cancel() until the solve() method completed execution on the server... which defeats the purpose of a method like cancel().

Lingo and ActiveMQ are open source so I rolled up my sleeves and ran the whole thing through a debugger. Debugging multithreaded applications can get tricky but after spending several hours I was able to get the to bottom of this issue.

Recollect that we exported the OptimizerSericve using the class org.logicblaze.lingo.jms.JmsServiceExporter in optimizerContext.xml. On examining the source, I found that this class creates a single JMS Session which listens for messages on the configured destination ("optimizerDestinationQ" in our example) and when messages are received, it invokes a Lingo listener which does the translation of the inbound message into a method invocation on the exported OptimizerServiceImpl service object.

The JMS spec clearly states

A Session object is a single-threaded context for producing and consuming messages.
...
It serializes execution of message listeners registered with its message consumers.

Basically a single JMS Session is not suitable for receiving concurrent messages. I understood why the cancel() method wasn't being invoked until the solve() method completed. But this behavior still didn't make sense from an API usage perspective.

Fortunately Spring 2.0 added support classes for receiving concurrent messages which is exactly what we need (yep, Spring rocks!). There are a few different support classes like DefaultMessageListenerContainer, SimpleMessageListenerContainer, and ServerSessionMessageListener .

The ServerSessionMessageListenerContainer "dynamically manages JMS Sessions, potentially using a pool of Sessions that receive messages in parallel". This class "builds on the JMS ServerSessionPool SPI, creating JMS ServerSessions through a pluggable ServerSessionFactory".

I tried altering optimizerContext.xml to use this class optimizerContextPooledSS.xml

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE beans PUBLIC "-//SPRING//DTD BEAN//EN" "http://www.springframework.org/dtd/spring-beans.dtd">
<beans>    <bean id="optimizerServiceImpl" class="org.sanjiv.lingo.server.OptimizerServiceImpl" singleton="true">    </bean>
    <bean id="optimizerServerListener" class="org.logicblaze.lingo.jms.JmsServiceExporterMessageListener">        <property name="service" ref="optimizerServiceImpl"/>        <property name="serviceInterface" value="org.sanjiv.lingo.common.OptimizerService"/>        <property name="connectionFactory" ref="jmsFactory"/>    </bean>
    <bean id="optimizerServer" class="org.springframework.jms.listener.serversession.ServerSessionMessageListenerContainer">        <property name="destination" ref="optimizerDestination"/>        <property name="messageListener" ref="optimizerServerListener"/>        <property name="connectionFactory" ref="jmsFactory"/>    </bean>
    <!-- JMS ConnectionFactory to use -->    <bean id="jmsFactory" class="org.activemq.ActiveMQConnectionFactory">        <property name="brokerURL" value="tcp://localhost:61616"/>        <property name="useEmbeddedBroker">            <value>true</value>        </property>    </bean>
    <bean id="optimizerDestination" class="org.activemq.message.ActiveMQQueue">        <constructor-arg index="0" value="optimizerDestinationQ"/>    </bean></beans>

Unfortunately the behavior was still the same - cancel() was not executing on the server until solve() completed. I posted this question on the Spring User list but did not get a response. This class uses the ServerSessionPool SPI so I'm not sure if there is a problem with the Spring class, the ActiveMQ implementation of this SPI or something that I've done wrong.

Anyway I was able to successfully configure the DefaultMessageListenerContainer class and observed the desired behavior. In contrast to ServerSessionMessageListenerContainer, DefaultMessageListenerContainer "creates a fixed number of JMS Sessions to invoke the listener, not allowing for dynamic adaptation to runtime demands". While ServerSessionMessageListenerContainer would have been ideal, DefaultMessageListenerContainer is good enough for most use cases as you'd typically want to have some sort of thread pooled execution on the server anyways.

optimizerContextPooled.xml

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE beans PUBLIC "-//SPRING//DTD BEAN//EN" "http://www.springframework.org/dtd/spring-beans.dtd">
<beans>
    <bean id="optimizerServiceImpl" class="org.sanjiv.lingo.server.OptimizerServiceImpl" singleton="true">    </bean>
    <bean id="optimizerServerListener" class="org.logicblaze.lingo.jms.JmsServiceExporterMessageListener">        <property name="service" ref="optimizerServiceImpl"/>        <property name="serviceInterface" value="org.sanjiv.lingo.common.OptimizerService"/>        <property name="connectionFactory" ref="jmsFactory"/>    </bean>
    <bean id="optimizerServer" class="org.springframework.jms.listener.DefaultMessageListenerContainer">        <property name="concurrentConsumers" value="20"/>        <property name="destination" ref="optimizerDestination"/>        <property name="messageListener" ref="optimizerServerListener"/>        <property name="connectionFactory" ref="jmsFactory"/>    </bean>
    <!-- JMS ConnectionFactory to use -->    <bean id="jmsFactory" class="org.activemq.ActiveMQConnectionFactory">        <property name="brokerURL" value="tcp://localhost:61616"/>        <property name="useEmbeddedBroker">            <value>true</value>        </property>    </bean>
    <bean id="optimizerDestination" class="org.activemq.message.ActiveMQQueue">        <constructor-arg index="0" value="optimizerDestinationQ"/>    </bean>
</beans>

Note : Although some Lingo examples have the destination created as a Topic(ActiveMQTopic)
with the org.logicblaze.lingo.jms.JmsServiceExporter class, you must use a Queue when
using multiple JMS sessions for concurrent message retreival as a Topic will be received
by all listeners which is not what we want.

Here's the result when using applicationContextPooled.xml

You can download the complete source for this here and run the sample server and client. JRoller doesn't allow uploading .zip files so I've uploaded the sample as a .jar file instead. The source distribution has a Maven 1.x project file. To build, simply run "maven". To run the optimizer sever without pooled JMS listeners, run startOptimizer.bat under dist/bin/. To run with pooled JMS listeners, run startOptimizerPooled.bat and to run the test client, run startClient.bat

I am using this architecture to provide a remote API for our C++ optimizer. The C++ optimizer has a thin JNI layer which loads the Spring application context file and the OptimizerServiceImpl has a bunch of native methods which is tied to the underlying C++ optimizer functionality using the JNI function RegisterNatives(). Do you Lingo? I'd like to hear how others are using Lingo/Spring Remoting.

Make sure the times on both machines are in sync otherwise messages are thrown in the dead letter queue without error.