Everyday a surprise, there are no exceptions today.

According to james' s email, system crahsed last night.  I checked tomcat log and found Exception as follows：

in thread “com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0” java.lang.OutOfMemoryError: PermGen space

I have to say, I hate "out of memory". It is really hard to report the situation to Jonny, he think 8G memory is far enough for our application.

After a short time research, I got explanation from internet: The permanent generation is used to hold reflective of the VM itself such as class objects and method objects. These reflective objects are allocated directly into the permanent generation, and it is sized independently from the other generations. Generally, sizing of this generation can be ignored because the default size is adequate. However, programs that load many classes may need a larger permanent generation. This error occurs when the JVM runs out of space in the permanent generation heap. Since the defaults for the JVM are quite low, the first step should be to increase the default using -XX:MaxPermSize.

The final Solution is :  configure MaxPermSize in /etc/profile or /etc/bash.bashrc

the java option will be: JAVA_OPTS=-Xms512m -Xmx1024m -XX:PermSize=256m -XX:MaxPermSize=256m

If you do not want to restart server, you can activate configuration by command like

source /etc/bash.bashrc