Pivotal Knowledge Base

Follow

MySQL Monitor VM shows High Memory Usage

Environment

Pivotal Cloud Foundry (PCF) versions 1.10, 1.11, 1.12, 2.0

Symptom

MySQL Monitor Node shows high memory usage, caused by the replication_canary process and possibly the black box process after some time. 

Cause

The replication_canary process is the cause of the high memory usage. A bug causes this process to overwrite log files rather than appending to them, which then causes problems when a 'logrotate' occurs on the 'replication-canary.stdout.log' file. The black box process may also contribute to the high memory usage as it follows the large log file created by replication_canary.

Resolution

Upgrade to the version of MySQL for Elastic Runtime (ERT) or Pivotal Application Service (PAS) which includes the mysql-monitoring release version 8.14.0. Below are the prerequisites:

  • Elastic Runtime versions 1.11.25+, 1.12.13+
  • PAS versions 2.0.4+, 2.1+
  • MySQL for PCF versions 1.9.18+, 1.10.11+

Note: If you are unable to upgrade, try the following steps manually:

  1. BOSH SSH to the mysql_monitor VM
  2. Edit /var/vcap/jobs/replication-canary/bin/replication-canary_ctl and remove the logLevel property.
  3. Restart the 'monit' services for the replication-canary and black box.

This should prevent the high memory utilization, however, the workaround will need to be carried out if the VM is recreated/upgraded.

Additional Information

Refer to these documents for guidance

Comments

Powered by Zendesk