I’ve been hunting down PermGen memory leaks lately with YourKit profiler. It’s been an interesting experience, tracking down these little buggers is frequently harder than you might think.

To save others the pain, I’ve documented all the problems I’ve found, and how I solved them, in this blog entry. I’ll keep updating this as I find new ones.

Java Bean Introspection

The Java bean introspector keeps a cache that doesn’t get flushed. It’s hard to know if this is a problem for you without profiling – your application probably doesn’t use the introspector, but some libraries that you use might.

To fix this problem you need to create a ServletContextListener in your app and add the following clean-up code to the “contextDestroyed” method.

[sourcecode language='java']
Introspector.flushCaches();
[/sourcecode]

If you’re using Spring, you should add the IntrospectorCleanupListener to your web.xml instead (this calls Instrospector.flushCaches() for you, as well as doing some additional cleanup).

Commons Pool Eviction Timer

In old versions of commons pool, the eviction timer was not being cleaned up when the webapp shut down. This issue has been fixed in the version 1.4 of commons-pool, so if you use commons-pool, make sure you’re using at least version 1.4.

As a side note, if you’re using commons-dbcp (which uses commons-pool) you should also make sure you’re using at least version 1.2.2, as it fixes a swag of issues some of which we’ve seen in our production systems.

MySQL Connector/J Statement Cancellation Timer

Version 5.1.6 (and earlier) of the MySQL JDBC driver (Connector/J) has a problem whereby the statement cancellation timer in the ConnectionImpl class is never cancelled, resulting in the timer thread hanging around even after you’ve unloaded your app. If you use the MySQL JDBC driver this will be a problem for you.

I’ve raised a bug report with the MySQL folks. In the meantime you can create a ServletContextListener in your app and add the following clean-up code to the “contextDestroyed” method.

[sourcecode language='java']
try {
if (ConnectionImpl.class.getClassLoader() == getClass().getClassLoader()) {
Field f = ConnectionImpl.class.getDeclaredField(”cancelTimer”);
f.setAccessible(true);
Timer timer = (Timer) f.get(null);
timer.cancel();
}
}
catch (Exception e) {
System.out.println(”Exception cleaning up MySQL cancellation timer: ” + e.getMessage());
}
[/sourcecode]

JDBC Driver Manager

The JDBC Driver Manager is notorious for causing PermGen memory leaks. When a JDBC driver starts up it gets registered with the global DriverManager, but never de-registered. If you are using any sort of JDBC driver, this will be a problem for you.

To fix this problem you need to create a ServletContextListener in your app and add the following clean-up code to the “contextDestroyed” method.

[sourcecode language='java']
try {
for (Enumeration e = DriverManager.getDrivers(); e.hasMoreElements(); ) {
Driver driver = (Driver) e.nextElement();
if (driver.getClass().getClassLoader() == getClass().getClassLoader()) {
DriverManager.deregisterDriver(driver);
}
}
}
catch (Throwable e) {
System.out.println(”Unable to clean up JDBC driver: ” + e.getMessage());
}
[/sourcecode]

DOM4J ThreadLocal leaks

Older versions of DOM4J store data in ThreadLocal variables that never get cleaned up. This problem has been fixed in recent versions of DOM4J. If you’re using DOM4J make sure you use at least version 1.6.1.

Xerces XML Libraries

Tomcat uses the SAXParser to read in configuration files. If you include Xerces in your webapp’s WEB-INF/lib directory Tomcat will use this rather than the one in the JDK. This is bad because Tomcat maintains a reference to this parser, which will prevent your webapp’s ClassLoader from being collected.

The fix is to remove the xerces and XML parser API libs from your webapp. If you’re running Java 5 or better these are included in the JRE so you don’t need them. If you’re using an older JRE, put the jars into the container’s shared library folder (”common/lib” in Tomcat).

Quartz Scheduler ShutdownHook

The Quartz ShutdownHookPlugin registers a hook with the Java runtime that doesn’t get cleaned up if you simply unload your webapp (it will only get cleaned up when you shut down the JVM).

The solution is to not use the ShutdownHookPlugin at all. If you’re using the Spring Quartz wrappers, you simply configure the Spring bean factory to clean up Quartz when the bean factory is destroyed. Otherwise it’s a simple job to add a ServletContextListener to clean up Quartz when your application shuts down.

Tomcat and Commons Logging

Earlier versions of Tomcat had a problem with leaking PermGen memory when a webapp included commons-logging in its WEB-INF/lib. Later versions fixed this problem, so make sure you’re running at least version 5.5.16.

Other Resources

Here are some other resources that you might find useful in finding/fixing your PermGen leaks:

• Spring Wiki

Good luck!

Tags: commons pool, dbcp, dom4j, java, jdbc, memory leak, mysql, permgen, quartz, Software Development, tomcat, xerces, yourkit

This entry was posted on Friday, May 9th, 2008 at 1:57 pm and is filed under Software Development, Technology. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.