15

We are using Java 8 Spring Boot 2 for our micro-service app. Running a load test locally noticed the heap consuming memory but never releasing back. I am running the app with G1 garbage collector and also did a manual GC from JVisualVM, but still the allocated memory never gets released.

I took the heap dump and analysed it, and I can clearly see the large byte array created by System Class Loader listed as leak suspect. I see the byte array instance holds my HTTP request to endpoint "/test". But the load test already completed and threads went back down to where they were before running load test.

Not sure why byte array loaded by system class loader contains all these elements and taking all this heap for no reason.

JVisualVM

Leak Suspects

byte Array

Objects with Outgoing Ref

/test endpoint is the only method in a @RestController class

@RequestMapping(value = "/test", method = RequestMethod.GET)
@CrossOrigin(origins = "*")
public void test() {
    logger.info("Testing1...");
}

Below are the Spring Boot application.properties related to server:

server.port=8090
server.tomcat.max-threads=200
server.tomcat.accept-count=100
server.tomcat.min-spare-threads=20
server.error.whitelabel.enabled=false
server.max-http-header-size=2097152
6
  • Can you post the implementation /test endpoint ? Commented Oct 16, 2019 at 19:44
  • @maitreyak I have posted the method which is just one liner Commented Oct 16, 2019 at 19:54
  • how many http worker threads do you have? i suspect the webserver pre-allocates IO buffers per thread. Commented Oct 16, 2019 at 20:03
  • @radai I have posted my server properties also. I have min 20 and max 200 Commented Oct 16, 2019 at 20:09
  • if you pin it at 1 thread, do the numbers change ? Commented Oct 16, 2019 at 20:16

2 Answers 2

14

Tomcat caches a number of objects to make it go faster. With the setting server.max-http-header-size=2097152 you made one of those cached objects claim 2 MB of memory and keep it. In this case it is the Http11OutputBuffer and you can see here that it claims (in your case) the 2 MB of memory. The Http11OutputBuffer is used by the Http11Processor which you can see here.

The documentation has the following to say about processorCache:

The protocol handler caches Processor objects to speed up performance. This setting dictates how many of these objects get cached. -1 means unlimited, default is 200. If not using Servlet 3.0 asynchronous processing, a good default is to use the same as the maxThreads setting. If using Servlet 3.0 asynchronous processing, a good default is to use the larger of maxThreads and the maximum number of expected concurrent requests (synchronous and asynchronous).

So my suggestion is to set the server.max-http-header-size to something more reasonable, e.g. 8KB (the default) and slowly double that when testing shows you really need that (related: Tomcat throws "400 Bad request" when total header size is larger than server.max-http-header-size).

Sign up to request clarification or add additional context in comments.

1 Comment

@TZ-EZ AFAIK it doesn't, but maybe the documentation has something to say about it? Also clearing cache would increase load when a spike of a lot of request need to be processed - that is the opposite of what you want during a spike and something that a filled cache can prevent.
2

Get a dump before running the stress test and a dump after running it. Eclipse MAT lets you compare the histogram between two dumps, so you'll know that a memory leak occurs.

I recommend using Java Mission Control (JMC) to monitor the JVM, where you can take a closer look at memory consumption (heap and not heap). Note the difference between memory in use and committed memory.

Newer versions of Java have more sophisticated algorithms where the JVM returns memory to the OS. For Java 8 one option is to use the Eclipse Open J9 JVM.

Recommended reading:

https://openjdk.java.net/jeps/346

https://jelastic.com/blog/elastic-jvm-vertical-scaling/

1 Comment

Also an updated presentation related to Java elasticity and unused memory release slideshare.net/mobile/jelastic/…

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.