Pivotal Cloud Foundry
Operations Manager is slow loading pages and eventually crashes.
Operation Manager has process 'bundle' using 100% CPU and slowly increasing upto 100% memory usage.
The following process is increasing in memory:
tempest+ 1463 11.7 76.3 6797856 6244960 ? Sl 16:14 36:19 /home/tempest-web/tempest/web/vendor/bundle/ruby/2.3.0/bin/thin -C config/thin.production.yml start
ubuntu 5744 0.0 0.0 10480 2240 pts/4 S+ 21:23 0:00 grep 1463
This issue is caused by bad IP range reservation specified under networks tab in OpsManager director. Bad ranges can cause OpsManager to iterate forever over that range and spike the CPU and memory utilization of ruby bundle process. This leads to degraded performance and eventually crashes of OpsManager
The IP reservations set in OpsManager network settings need to be reviewed for problems.
You can do this in GUI in OpsManager Director > Networks tab. (you may need to perform `service tempest-web restart` on Ops Manager first to restart process)
Alternatively, you can verify network settings in installation.yml by decrypting it from this location:
sudo -u tempest-web RAILS_ENV=production /home/tempest-web/tempest/web/scripts/decrypt /var/tempest/workspaces/default/installation.yml /tmp/installation.yml
Now you need to verify your network IP ranges and reservations for bad settings.
Examples of bad network settings:
IP reservation range goes backwards (from high number to low): 10.1.1.250-10.1.1.2
IP reservation has mismatching subnets: 10.72.94.141-10.72.82.255
Network has very large CIDR block: 10.72.0.0/5
Once the bad network setting is identified and addressed, the performance should immediately be fixed upon saving changes in OpsManager.