Feb 23, 2010 / CLUSTER, TOMCAT

Clustering Tomcat

In this article, I will show you how to use Apache/Tomcat in order to set up a load balancer. I know this has been done a zillion time before, but I will use this setup in my next article (teaser, teaser) so at least I will have it documented somewhere.

Apache Tomcat is the reference JSP/container since its inception. Despite a lack of full JEE support, it certainly has its appeal. The reasons behind using a full-featured commercial JEE application server are not always technical ones. With lightweight frameworks such as Spring being mainstream, it is not unusual to think using Tomcat in a production environment. Some companies did it even before that.

When thinking production, one usually think reliability and scalability. Luckily, both can be attained with Apache/Tomcat through the set up of a load-balancing cluster. Reliability is thus addressed so that if a Tomcat fails, following requests can be directed to a working Tomcat. Requests are dispatched to each Tomcat according to a predefined strategy. If the need be, more Tomcat can be added at will in order to scale.

In the following example, I will set up the simplest clustering topology possible: an Apache front-end that balances 2 Tomcat instance on the same physical machine.

Set up Apache

The first step is to configure Apache to forward your requests to Tomcat. There are basically 2 options in order to do this (I ruled out the pre-shipped load-balancer webapp):

use mod_jk, the classic Apache/Tomcat module
use mod_proxy, another Apache module

I’m not a system engineer, so I can’t decide on facts whether to use one or the other: I will use mod_jk since I’ve already used it before.

Download the mod_jk that is adapted to your Apache and Tomcat versions
Put it in the 'modules' folder of your Apache installation
Update your httpd.conf configuration to load it with Apache
```
LoadModule jk_module modules/mod_jk-1.2.28-httpd-2.2.3.so
```

Configure Apache. Put these directive in the httpd.conf:

JkWorkersFile	conf/worker.properties
JkShmFile	logs/mod_jk.shm
JkLogLevel	info
JkLogFile logs/mod_jk.log
JkMount		/servlets-examples/* lb

This configuration example is minimal but needs some comments:

Parameter Description

Parameter	Description
`JkWorkersFile`	Where to look for the module configuration file (see below)
`JkShmFile`	Where to put the shared memory file
`JkLogLevel`	Module log level (debug/error/info)
`JkLogFile`	Log file location. It is the default but declaring it avoid the Apache warning
	`JkMount`

JkWorkersFile

Where to look for the module configuration file (see below)

JkShmFile

Where to put the shared memory file

JkLogLevel

Module log level (debug/error/info)

JkLogFile

Log file location. It is the default but declaring it avoid the Apache warning

JkMount

Since mod_jk can be used in non-clustered setups, there could be any JkMount, each forwarding to its own worker (see below). In our case, it means any request beginning with /servlets-examples/ (the trailing slash is needed) will be forwarded to the 'lb' worker.

Configure the workers

Workers are destination routes as viewed by Apache. They’re are referenced by an unique label in the httpd.conf and parameteirzed under the same label in the worker.properties file.

My workers.properties is the following:

worker.list=lb
worker.worker1.port=8010
worker.worker1.host=localhost
worker.worker1.type=ajp13
worker.worker2.port=8011
worker.worker2.host=localhost
worker.worker2.type=ajp13
worker.lb.type=lb
worker.lb.balance_workers=worker1,worker2

I define 3 workers in this file: lb, worker1 and worker2. The 'lb' worker is the load-balancing worker: it is virtual and it balances the latter two. Both are configured to point to a real Tomcat instance.

Now, with the Apache configuration in mind, we see that requests beginning with /servlets-examples/ will be managed by the load balancer worker which will in turn forward to a random worker.

One can also put weight on workers hosted by more powerful machines so that these are more heavily loaded than less powerful ones. In our case, both are hosted on the same machine so it has no importance whatsoever.

Configure the Tomcat instances

The last step consist of the configuration of Tomcat instances. In order to do so, I shamelessly copied entire Tomcat installations (I’m on Windows). While editing the server.xml of the Tomcat instances, three points are worth mentioning:

The Engine tag has a jvmRoute attribute. It’s value should be the same as the worker’s name used in both httpd.conf and worker.properties. Otherwise, sessions will be recreated for each request
Look out for duplicated port numbers if all Tomcat instances are on the same machine. For example, use an incremental rule to configure every stream on a different port
Be sure that the tcpListenPort attribute of the Receiver is unique across all Tomcat instances

Use it!

With the previous set up, one can now start both Tomcat and Apache, then browse to the servlet-examples webapp, and more precisely to the Session page. Look there for Tomcat 5.5 and there for Tomcat 6. The servlet-example page page displays the associated session ID:

ID de Session: 324DAD12976045D197435033A67C025D.worker2
Crée le: Tue Feb 23 23:15:13 CET 2010
Dernier accès: Tue Feb 23 23:31:47 CET 2010

Notice that on my Tomcat instance, the worker’s name is part of the session ID.

If everything went fine, two interesting things should take place: first, when refreshing the page, the session ID should not change because of the sticky session (enabled by default). Morevoer, if I shutdown the Tomcat instance associated with the worker (the second in my case), and if I try to refresh the page, I still can access my application, but under a new session.

Thus, I lose all the information I stored under my session! In my following article, I will study how on can try to remedy to this.

To go further:

Follow @nicolas_frankel