Pivotal Knowledge Base

Follow

P-Invitations app has Out Of Memory (OOM) error after upgrading Elastic Runtime

Environment

Pivotal Cloud Foundry (PCF): 1.10.x 

Symptom

When apply changes to Elastic Runtime or an upgrade of Elastic Runtime in 1.10.x, you see similar entries in the push-apps-manager errand output:

Waiting for app to start...
Start unsuccessful

TIP: use 'cf logs p-invitations --recent' for more information
FAILED

You will also see the following at the end of the push-apps-manager output:

Errand 'push-apps-manager' completed with error (exit code 64) 

Navigating to the system Org, system Space by running cf target -s system -o system with the CF CLI you will noticed that the p-invitations app is not running. If you run cf logs p-invitations --recent , the logs might contain the following error:

2017-05-26T10:28:22.697-04:00 [CELL/1] [OUT] Destroying container
2017-05-26T10:28:22.727-04:00 [API/0] [OUT] Process has crashed with type: "web"
2017-05-26T10:28:22.734-04:00 [API/0] [OUT] App instance exited with guid af69afe4-a85f-45a5-843c-a8855a6282c5 payload: {"instance"=>"", "index"=>1, "reason"=>"CRASHED", "exit_description"=>"2 error(s) occurred:\n\n* 2 error(s) occurred:\n\n* Exited with status 1 (out of memory)\n* cancelled\n* cancelled", "crash_count"=>6, "crash_timestamp"=>1495808902707525622, "version"=>"31f7517c-5500-46ea-a38a-d20c69729254"}

If you see an entry similar to this, then your p-invitations app is crashing due to being Out Of Memory (OOM) as described by Cloud Foundry. 

Cause

Recent changes in PCF 1.10 have cause the p-invitations app to run with more memory than the originally allocated 128M at times. To be able to get past this issue we simply need to scale up the memory for the p-invitations system app then you should be able to proceed with your operations without encountering this error again. 

Instructions

To get past this issue, follow these steps:

Step 1

Run cf target -s system -o system to verify you'll be in the correct space where the p-invitations app is staged:

$ cf target -s system -o system
api endpoint: https://api.<domain>
api version: 2.75.0
user: admin
org: system
space: system

Step 2

Run cf app p-invitations to verify p-invitations is only running with 128M of memory, example output:

Showing health and status for app p-invitations in org system / space system as admin...
name: p-invitations
requested state: stopped
instances: 0/2
usage: 128M x 2 instances
routes: p-invitations.<domain>
last uploaded: Thu 08 Jun 17:43:16 EDT 2017
stack: cflinuxfs2
buildpack: nodejs_buildpack
restate since cpu memory disk details
#0 crashed 2017-06-08T21:43:51Z 0.0% 0 of 128M 0 of 1G
#1 crashed 2017-06-08T21:43:51Z 0.0% 0 of 128M 0 of 1G

Step 3

Run cf scale p-invitations -m 256M, you will see similar output to the following if successful:

$ cf scale p-invitations -m 256M

This will cause the app to restart. Are you sure you want to scale p-invitations?> yes

Scaling app p-invitations in org system / space system as admin...
OK
Stopping app p-invitations in org system / space system as admin...
OK

Starting app p-invitations in org system / space system as admin...

0 of 2 instances running, 2 starting
2 of 2 instances running

App started

OK

App p-invitations was started using this command `npm start`

Showing health and status for app p-invitations in org system / space system as admin...
OK

requested state: started
instances: 2/2
usage: 256M x 2 instances
urls: p-invitations.run-02.haas-59.pez.pivotal.io
last uploaded: Thu Jun 8 21:43:16 UTC 2017
stack: cflinuxfs2
buildpack: nodejs_buildpack

state since cpu memory disk details
#0 running 2017-06-09 08:50:45 AM 0.0% 36.2M of 256M 80.6M of 1G
#1 running 2017-06-09 08:50:45 AM 0.0% 54.9M of 256M 80.7M of 1G

Step 4

Once you have scaled the memory for p-invitations up to 256M, you should be able to apply changes again or re-run your upgrade in Ops Manager without encountering this error. 

Additional Information

In future releases beyond Elastic Runtime 1.10.12, the p-invitations will begin by being staged with 256M to circumvent this issue.

Comments

Powered by Zendesk