Pivotal Cloud Foundry (PCF) versions 1.10, 1.11, 1.12, 2.0
MySQL Monitor Node shows high memory usage, caused by the replication_canary process and possibly the black box process after some time.
The replication_canary process is the cause of the high memory usage. A bug causes this process to overwrite log files rather than appending to them, which then causes problems when a 'logrotate' occurs on the 'replication-canary.stdout.log' file. The black box process may also contribute to the high memory usage as it follows the large log file created by replication_canary.
Upgrade to the version of MySQL for Elastic Runtime (ERT) or Pivotal Application Service (PAS) which includes the mysql-monitoring release version 8.14.0. Below are the prerequisites:
- Elastic Runtime versions 1.11.25+, 1.12.13+
- PAS versions 2.0.4+, 2.1+
- MySQL for PCF versions 1.9.18+, 1.10.11+
Note: If you are unable to upgrade, try the following steps manually:
- BOSH SSH to the mysql_monitor VM
- Edit /var/vcap/jobs/replication-canary/bin/replication-canary_ctl and remove the logLevel property.
- Restart the 'monit' services for the replication-canary and black box.
This should prevent the high memory utilization, however, the workaround will need to be carried out if the VM is recreated/upgraded.
Refer to these documents for guidance