Monday, February 21, 2011

Retries with the Spring HTTP Invoker

I've been doing work on a project that uses Spring Remoting with the HTTP Invoker.  It makes remoting a piece of cake, and since it's HTTP, it's almost trivial to introduce a load balancer for HA/Scalability.  I didn't realize that it was supporting retries until I saw this in the logs:

2011-02-07 19:59:37,541 [http-8080-2] INFO  org.apache.commons.httpclient.HttpMethodDirector[439] - I/O exception (java.net.ConnectException) caught when processing request: Connection refused
2011-02-07 19:59:37,541 [http-8080-2] INFO  org.apache.commons.httpclient.HttpMethodDirector[445] - Retrying request

I was a bit surprised to discover this, since it is not documented, and I was about to implement my own retry logic.  The retry support actually has nothing to do with Spring Remoting, as it is built into HttpClient, which is used by the CommonsHttpInvokerRequestExecutor.  By default, the HttpClient code will attempt up to three retries. The retry code is on line 433 of HttpMethodDirector.java (v 1.34 2005/01/14 19:40:39 olegk). The code is smart about what type of exceptions it will retry. The most typical exception that would get retried is an IOException (e.g. java.net.ConnectException). The retry code does not involve any delays, which if it did would be a big reason to disable it. But the code can block up to the connect timeout, under certain conditions, for example if the server is under heavy load.  This of course translates to a potential 3*timeout delay.  If you would like to suppress retries, or change the retry value, here's how to do it. Note, you will need to create the invoker programmatically to do this.

HttpInvokerProxyFactoryBean fb = new HttpInvokerProxyFactoryBean();

CommonsHttpInvokerRequestExecutor client = new CommonsHttpInvokerRequestExecutor();

client.getHttpClient().getParams().getDefaults().setParameter(HttpMethodParams.RETRY_HANDLER, new  DefaultHttpMethodRetryHandler(0, false));

fb.setHttpInvokerRequestExecutor(client);

The first argument to DefaultHttpMethodRetryHandler is the retry value.  You could also configure the invoker in the Spring config, and then get a reference via the Spring context.  Alternatively, you can use SimpleHttpInvokerRequestExecutor instead of CommonsHttpInvokerRequestExecutor. The former uses the java.net.HttpURLConnection, with no retry support, while the latter uses Apache HttpClient.

This is a good time to set your connection and read timeouts

client.getHttpConnectionManager().getParams().setConnectionTimeout(10000);
client.setReadTimeout(20000);

There, that's it. I hope. Note: I didn't exactly test this, since retries seem to be a good thing for me, however, it seems like it should work.