We recently had a scenario where an apache reverse proxy needed to be deployed in front of a pair of tomcat servers. Due to security concerns, this reverse proxy was hosting mod_security and acting as a web application firewall (WAF)
However, a critical requirement was that the tomcat applications would be able to see the original IP address of the client. This presented a problem because unlike squid, apache has no configurable option to act as a fully transparent proxy. In other words, once traffic was redirected through the apache reverse proxy, the traffic forwarded to the tomcat server was forwarded with it’s source IP address changed to the proxy, effectively hiding the public IP the client used to connect to the site.
The first solution that sprang to mind was the “X-Forwarded-For” headers, which is an HTTP header inserted into the original HTTP GET request whose value is equal to the client’s public IP. Turns out apache reverse proxy inserts this header by default, and even so the tomcat application could not extract the client’s IP. We somehow needed to instruct the tomcat server itself to provide the application with the correct client IP.
The solution that worked in my case was the RemoteIP tomcat valve. Official documentation lives here:
It’s quite simple to configure in that all that needs to be done is to modify tomcat server.xml to recognise original client IP rather than the proxy IP by adding the following to server.xml:
make sure to change 127.0.0.1 to the address of the apache reverse proxy.
The application could now recognise the original client IP.
PS as per the tomcat documentation, the apache equivalent of the above method is using the mod_remoteip
In my previous post I mentioned that the check_jmx plugin I’ve used for monitoring JMX does not output perfdata data, which is useful for programs such as Centreon and Nagios which use this data to graph the output of the plugin.
The developers have made an excellent job with this plugin, and I hope they take the below script and incorporate into their official script
I’ve written an “add-on” script which basically wraps around the official script and outputs the data in a perfdata friendly way, while keeping the nagios plugin OK/WARNING/CRITICAL states. The script isn’t perfect in the sense it could do with some input validation and a more helpful help, but it does work
Below are the contents of the script “check_jmx_perfdata”, and assumes the plugin is located at /usr/local/nagios/libexec/check_jmx:
# IMPORTANT, do not forget the ” ” when using this command as per example below
# example usage:
# ./check_jmx_perfdata “-U service:jmx:rmi:///jndi/rmi://localhost:9003/jmxrmi –O #Catalina:type=ThreadPool,name=http-8080 -A currentThreadCount”
# grab the original output from the check_jmx plugin
# set the variables, where metric = attribute being monitored
# value = numeric value of the attribute
# result = OK/WARNING/CRITICAL/UNKNOWN state
metric=$(echo $plugin_output | cut -d ” ” -f 4)
value=$(echo $plugin_output | cut -d ” ” -f 6)
result=$(echo $plugin_output | cut -d ” ” -f 2)
# output the result with the added perfdata output
echo “$plugin_output | $metric=$value”
case “$result” in
Instead of calling the check_jmx script from nagios, you will call the above script, using the exact same syntax but using the inverted commas. So for exmaple if you used:
check_jmx -U service:jmx:rmi:///jndi/rmi://localhost:9003/jmxrmi -O Catalina:type=Manager,path=/servlets-examples,host=localhost -A activeSessions
You’d now use:
check_jmx_perfdata “-U service:jmx:rmi:///jndi/rmi://localhost:9003/jmxrmi -O Catalina:type=Manager,path=/servlets-examples,host=localhost -A activeSessions”
Requirement : monitoring vital Tomcat statistics such as active & idle threads, memory consumption and so on.
JMX (java monitoring extensions) is quite well documented – so a google search should bring you up to speed, but in a nutshell, JMX is almost like a java-centric SNMP. Java app developers can make certain attributes such as number of threads used, etc, available via “MBeans” (java-style SNMP OIDs). So this lends itself quite well to monitoring.
To enable JMX on a CentOS based tomcat server:
- Navigate to /etc/tomcat/tomcat5.conf and add the following lines to the end of the “JAVA_OPTS” variable:
The options are self-explanatory. The above does not use authentication
- Make sure the file jmxremote.password exists in $JAVA_HOME/jre/lib/management – Edit the password file to set the password, even if (as above) authentication is actually disabled.
- Also make sure the password file is read only for the same owner as the java process. You can create the password file from the template file. (jmxremote.password.template)
- SYSTEM is the default owner of java process in Windows environment.
- If your remote app is running behind a NAT device, you need to add the additional line when starting the app to specify the remote hostname or ip address:
-Djava.rmi.server.hostname=<IP or hostname>
Please note, the IP or hostname must be the public IP of the tomcat server
To monitor JMX using nagios:
- download “jmxquery” script:
- Unzip the archive, and modify the script contained therein:
* change the JAVA_HOME variable
(hint: to set JAVA_HOME in the above two check the file /etc/tomcat/tomcat5.conf)
- note : the above script does not yet support outputting perfmon data for nagios graphing. I’ll post an add-on script in my next post.
- note on troubleshooting: if after following all instructions the check_jmx plugin returns “no route to host” , check the /etc/hosts file on the tomcat server to make sure you have the correct IP to hostname mapping for the server.
- install the package named “jakarta webapps” for tomcat app examples you may use to test JMX
- examples of JMX (check also http://tomcat.apache.org/tomcat-5.5-doc/monitoring.html), note the below should be on a single line, also if monitoring remotely remember to change “localhost” to the appropriate IP
— Heap Memory:
./check_jmx -U service:jmx:rmi:///jndi/rmi://localhost:9003/jmxrmi -O java.lang:type=Memory -A HeapMemoryUsage -K used -I HeapMemoryUsage
— Active Sessions:
./check_jmx -U service:jmx:rmi:///jndi/rmi://localhost:9003/jmxrmi -O Catalina:type=Manager,path=/servlets-examples,host=localhost -A activeSessions
— Max Active Sessions:
./check_jmx -U service:jmx:rmi:///jndi/rmi://localhost:9003/jmxrmi -O Catalina:type=Manager,path=/servlets-examples,host=localhost -A maxActiveSessions
— Thread Count:
./check_jmx -U service:jmx:rmi:///jndi/rmi://localhost:9003/jmxrmi -O java.lang:type=Threading -A ThreadCount
As an alternative, you may also use jmxterm (needs some scripting):
- embedding in script:
- extra details:
At this stage you may be asking, what else can I monitor over JDK? This is where “jconsole” comes into play. To use jconsole:
- download JDK
- open jconsole.exe (in windows case)
- enter the ip and port of the tomcat server to monitor.
This will give you a couple of useful graphs:
And a summary of the java machine:
But, most importantly in our case, it gives you a whole list of monitor able attributes on your tomcat server:
Tomcat is one of the more involving servers to setup. It contains a rather large server.xml file that can be daunting at first. If you “break down” the massive xml file into containers however, it becomes a lot more understandable. Below is a diagram I drew up during my studies:
For a better (printable) view:
The diagram should help to make it clearer. The tomcat5.conf file contains variables which control the JAVA server and filepaths referred to from within server.xml
The server.xml file itself is split into “levels” which I call “containers” in the diagram. Each container can contain other “child” containers, which are influenced by the attributes specified within the “parent” container. So for example, if the debug valve in the Engine container is switched on, it will log traffic to all the hosts, since the hosts container is a child of the engine container.
Some of the more important container attributes are:
The path to this Host’s webapps directory (the directory in which webapp directories and/or WAR files reside). This can be a path relative to CATALINA_HOME, or an absolute path.
Another fully qualified hostname for the same Host container.
This is the path (relative or absolute) to the webapp’s unpacked directory or WAR file. If you specify a relative path, the path is relative to the Host’s appBase directory. Do not set a value for docBase that contains the value of appBase at the beginning of the value. For example, if appBase=”deploy”, do not choose a docBasevalue such as “deployment-webapp“. Doing so will lead to deployment errors.
The URI path relative to the root of the web server (”/“) where this webapp should be mapped. Set this to an empty string (“”) to denote that the webapp should be the root webapp. This attribute cannot be set unless the Context element is inside the server.xml file.
<Engine name=”Catalina” defaultHost=”localhost”>
<Host name=”localhost” appBase=”webapps”>
<Context path=”” docBase=”ROOT”/>
<Context path=”/orders” docBase=”/home/ian/orders”
<Host name=”www.example.com” appBase=”/opt/example.com/webapps”>
<Context path=”” docBase=”ROOT”/>