Post

2 followers Follow
0
Avatar

Spring Boot Restful service with incremental memory without load.

Dear PCF team,

I have deployed Spring Boot Restful service [spring-boot-starter-parent - 1.3.0.RELEASE] along with spring-boot-starter-cloud-connectors, spring-boot-starter-data-rest, spring-boot-starter-jdbc, spring-boot-starter-test, spring-boot-configuration-processor, spring-context-support[4.1.2.RELEASE] and spring-cloud-starter-parent[Brixton.M4] maven dependencies. Also connected with MySQL and AutoScalar marketplace services.

This restful service is working excellent with respect to SLA and memory utilization.

However, I have observed following concerns:

1. The application has been deployed with initial memory however after couple of minutes (say 15-20 minutes), it is keep growing by 2-5MBs every couple-of-minutes. However there is no load (service call).

2. After a huge work load, the application get reached on certain memory (say 600MB). But the Memory is not coming down after the load (even in idle state for a while ). However due to above point 1, the memory is keep growing.

The JMX is showing that GC is calling both minor and major while work load.

And following is the Spring Metrics through spring-boot-starter-actuator:

{"_links" : {"self" : {"href" : "https:///Promotion/metrics"}},"mem" : 376320,"mem.free" : 199048,"processors" : 4,"instance.uptime" : 6457682,"uptime" : 6464241,"systemload.average" : 0.05,"heap.committed" : 376320,"heap.init" : 382976,"heap.used" : 177271,"heap" : 376320,"threads.peak" : 22,"threads.daemon" : 20,"threads.totalStarted" : 27,"threads" : 22,"classes" : 9916,"classes.loaded" : 9916,"classes.unloaded" : 0,"gc.ps_scavenge.count" : 47,"gc.ps_scavenge.time" : 344,"gc.ps_marksweep.count" : 0,"gc.ps_marksweep.time" : 0,"httpsessions.max" : -1,"httpsessions.active" : 0,"datasource.primary.active" : 0,"datasource.primary.usage" : 0.0,"gauge.response.metrics" : 7.0,"gauge.response.BuyMoreSaveMore.getDiscDetails" : 14.0,"counter.status.200.metrics" : 7,"counter.status.200.BuyMoreSaveMore.getDiscDetails" : 850}

Shobhit Sharma

Please sign in to leave a comment.

3 comments

1
Avatar

It's really hard to say what's happening with the given information.  Here are some suggestions to perhaps understand what is going on better.

  1. Updated to the latest available Spring Boot version.  As of me writing this, 1.3.7 is the current GA release.  This will make sure that you are not running into an issue that has already been fixed.
  2. Likewise make sure you're using the latest SCS support.  With SCS 1.1 this get's easier as you no longer use the SCS starter.  New instructions.  This does require your services to be SCS 1.1 as well though, so you'd have to upgrade the tile in conjunction with your services.  The reason I mention this is because there is a known leak in SCS 1.0 Service Registry that causes a very slight memory growth in metaspace and an eventual OOME: metaspace.
  3. If running on PCF, make sure you're using Java build pack 3.8+.  There were changes to the default memory calculations in this release and it makes Java applications running on PCF more stable.
  4. Hook up a profiler to your application.  This will help you to debug 95% of memory problems.  It's easier when you can run your application locally, but you can still do it when running your app on PCF.  Here's instructions for doing this with YourKit.

Hope that helps!

Daniel Mikusa 1 vote
0
Avatar

Thanks Dan,

I have already tried AppDynamics "Auto Memory Leakage" option and found nothing. So I am escaping point 4 (yourkit).

Also, I have tried the latest Java_Buildpack (https://github.com/cloudfoundry/java-buildpack.git) but no luck; still memory is keep increasing without any load/service call (option 3).

Now I going to apply option 1 & 2. Hope for the best!!

 

 

Shobhit Sharma 0 votes
0
Avatar

>I have already tried AppDynamics "Auto Memory Leakage" option and found nothing. So I am escaping point 4 (yourkit).

No offense to AppDynamics, but I wouldn't rely on that to give you a definitive answer here.  There are so many more things that could be happening beyond a traditional memory leak or even beyond what heuristics might reliably catch.  

You should really get some heap dumps and look through them manually for issues using YourKit, jvisualvm or Eclipse Memory Analyzer (or some other similar tool).  If memory growth is really happening as fast as you say, it should be easy enough to grab three or four heap dumps over the course of 30 minutes and then look through to see what is causing the memory usage to go up.  

Plus if you do end up finding something, these tools will help to track down where the problem exists.  Output from these tools and heap dumps or a minimal example application that replicates the problem would be critical to getting a fix should there be something wrong in SCS or Spring Framework.

Hope that helps!

Daniel Mikusa 0 votes