Running Tomcat as a resilient service on Ubuntu 14.04

I’m not a system administrator, I’m a consultant and a developer. However, in order to put my ideas in practice and show the world that they work, I had to be able to set up fully working prototypes. I rented a server and started to deploy applications the same way I did for development on my PC.

Naive deployment by a naive consultant

That might sound very obvious to some of you, but that was not exactly a bulletproof setup. The applications tended to crash once in a while, and I rebooted them manually, as soon as I found out. The frequency of my tweets about my work increased, and I couldn’t take the risk of a potential client clicking on one of my links and see an error where there should be a cool app. So I took measures, and checked how to do it right.

Objective: invincible processes

The objective was clear: since I couldn’t figure out the root cause of the failures of the application server, and they occurred only a few times a week in the worst case, I had to find a way to have the application server reboot whenever it failed.

I was already familiar with the concept of services on Windows, I consequently looked for a similar mechanism on Linux. And more specifically, Ubuntu 14.04, as it is what my server runs (August 2015). In Ubuntu 14.04, three systems offered this sort of facility: System V, Upstart and systemd. System V didn’t seem to be the future, Upstart was very well documented but specific to Ubuntu, and systemd wasn’t fully implemented yet (only in version 15.04 and later). Upstart good documentation convinced me.

So, enough talking, here is how I setup Tomcat to be more robust:

1. Installing Tomcat

Package manager vs. official binaries

There are two ways to install Tomcat:

• Installing it through the package manager (apt)
• Downloading the binaries from Apache and stash them in /usr/share

I tested the latter but stuck to the former. When you install it through the package manager:

• upgrading is easier
• a bunch of environment variables are set up for you
• …it’s just well integrated

So, here is the magic command: apt-get install tomcat7

Tomcat all over the place

The way Tomcat gets installed can be confusing, as the files are spread in various directories across the system:

• The core scripts (catalina.sh, etc.) and libraries sit in /usr/share/tomcat7. That’s CATALINA_HOME.
• The core configuration (server.xml,tomcat-users.xml, etc.) sits in /etc/tomcat7/.
• The logs are stored in /var/log/tomcat7.
• The cache is stored in /var/cache/tomcat7.
• The default base (CATALINA_BASE) is /var/lib/tomcat7, and it has symlinks to logs (/var/log/tomcat7), cache (/var/cache/tomcat7) and conf (/etc/tomcat7/).

Ubuntu provides a fine feature via the tomcat7-user package: the possibility for any user to create light weight instances of Tomcat in a directory where they have writing rights. More about this feature and Tomcat in Ubuntu documentation.

Disabling System V scripts for Tomcat

On top of the Tomcat files, a few scripts are also created to set up Tomcat as a System V service so that it can start at boot time. It’s good, but we also want resilience, which System V doesn’t provide (AFAIK). You have:

• The service script is /etc/init.d/tomcat7
• Symlinks to this script are created in /etc/rc0.d/, /etc/rc1.d/, and so on to make Tomcat start up system boot.

If you want to use Upstart, you should remove the tomcat scripts from the System V startup configuration to prevent conflicts: update-rc.d -f tomcat7 remove

Setting up the Upstart service

Each Upstart service is created as soon as it’s described in a valid .conf file stored in /etc/init/ directory. Be careful when creating a new Upstart service .conf file, its name should differ from any script stored in /etc/init.d/ otherwise expect strange behaviors. For instance, now you have installed Tomcat, you should not create a tomcat7.conf file since /etc/init.d/tomcat7 exists. I called mine tomcat.conf, and here it is in extenso and with inline comments:

Check the validity of your service file by running init-checkconf /etc/init/yourservice.conf.

If the syntax is OK, start your service with initctl start yourservice. If it starts normally, you see a message like yourservice start/running, process 1234. Congrats!

If things go wrong, either retry keeping an eye on the system logs (tail -f /var/log/syslog) or check the logs of your scripts.

The final test

When it started, the console gave you the process ID of your service (in my example it’s 1234). Let’s see what happens if we brutally kill your new shiny service to simulate an unexpected stop: kill 1234.

If you did a good job, the service will automatically restart. If it’s Tomcat or another Web application thingy (Apache, nginx, node, etc.), you can check if it’s up by running netstat -nlp. You should see it in the first list, with a new process ID.

---

I wrote this article to leave a trace of my experiments. If it can help others, even better! As soon as my hosting provider proposes Ubuntu 16.04, I’ll upgrade, switch to systemd and write a similar article.

Do you think this article is clear enough? How does my method compare with yours?

Sources :

• Upstart cookbook
• Example of working upstart conf file for Tomcat