Pivotal Knowledge Base

Follow

Ops Manager slows down as Ruby bundle process is spiking memory and CPU

Environment

Pivotal Cloud Foundry

Symptom

Operations Manager is slow loading pages and eventually crashes.

Operation Manager has process 'bundle' using 100% CPU and slowly increasing upto 100% memory usage.

The following process is increasing in memory:

tempest+ 1463 11.7 76.3 6797856 6244960 ? Sl 16:14 36:19 /home/tempest-web/tempest/web/vendor/bundle/ruby/2.3.0/bin/thin -C config/thin.production.yml start
ubuntu 5744 0.0 0.0 10480 2240 pts/4 S+ 21:23 0:00 grep 1463

Cause 

This issue is caused by bad IP range reservation specified under networks tab in OpsManager director. Bad ranges can cause OpsManager to iterate forever over that range and spike the CPU and memory utilization of ruby bundle process. This leads to degraded performance and eventually crashes of OpsManager

Resolution

The IP reservations set in OpsManager network settings need to be reviewed for problems. 

You can do this in GUI in OpsManager Director > Networks tab. (you may need to perform `service tempest-web restart` on Ops Manager first to restart process)

Alternatively, you can verify network settings in installation.yml by decrypting it from this location:

sudo -u tempest-web RAILS_ENV=production /home/tempest-web/tempest/web/scripts/decrypt /var/tempest/workspaces/default/installation.yml /tmp/installation.yml 

vim /tmp/installation.yml

Now you need to verify your network IP ranges and reservations for bad settings.

Examples of bad network settings:

IP reservation range goes backwards (from high number to low): 10.1.1.250-10.1.1.2

IP reservation has mismatching subnets: 10.72.94.141-10.72.82.255 
                                                             110.X.X.X-10.X.X.X

Network has very large CIDR block: 10.72.0.0/5

---

Once the bad network setting is identified and addressed, the performance should immediately be fixed upon saving changes in OpsManager.

Additional Information

Refer How to manually edit your installation YAML file on the Ops Manager VM.

Comments

Powered by Zendesk