|Pivotal Cloud Foundry||All|
CFOPS backups fail for elastic-runtime tile.
ubuntu@pivotal-ops-manager:/dd-bkp/17-Apr-2017$ cfops backup --opsmanagerhost <ops-mgr-URL> --adminuser admin --adminpass <password> --opsmanageruser ubuntu --opsmanagerpass <password> -d /<directory>/ --tile elastic-runtime
2017/04/17 09:56:42 I0315 09:56:42.294729 29772 elasticruntime.go:144] Exporting %s nfs_server
2017/04/17 09:56:42 I0315 09:56:42.295988 29772 elasticruntime.go:144] Exporting %s mysql
2017/04/17 09:56:42 E0315 09:56:42.296628 29772 elasticruntime.go:69] Error backing up db open /<directory>/mysql.backup: permission denied
2017/04/17 09:56:42 E0315 09:56:42.296652 29772 createCliCommand.go:52] there was an error: failed to backup database running backup on elastic-runtime tile:tile
The issue exists because the VM
vcap credentials are not in sync with the vcap credentials displayed in the Ops Manager Credentials tab. The password known to Ops Manager is different than that used by VM.
This can also happen when the option "Use default BOSH password" is selected under the Ops Manager tile -> Security tab.
Hence, when CFOPS is trying to SSH to the VM, it fails with permission denied.
ubuntu@pivotal-ops-manager:/dd-bkp/15-Mar-2017$ cfops version
cfops version v3.0.5
ubuntu@pivotal-ops-manager:/dd-bkp/15-Mar-2017$ cfops list-tiles
nfs_server using the
vcap credentials of the VM.
- Login to Ops Manager dashboard and make a note of the
vcapcredentials from the Elastic Runtime tile -> Credentials tab
- Make a note of the
nfs_serverIP Address from the Elastic Runtime tile -> Status tab
- Using a terminal window, try to SSH into the
nfs_serverIP address with the vcap credentials noted above
Failing to login to
nfs_server VM using above
vcap credentials confirms the password sync issue.
This can be tested for any other VM under the Elastic Runtime tile.
There are two possible causes:
- During the upgrade process, the passwords were reset. However, the upgrade process failed due to an error before updating the passwords in installation YAML.
- The option "Use default BOSH password" is selected under Ops Manager tile -> Security tab.
To fix the password sync issue, we need to recycle and regenerate the passwords by using the Bosh default password settings and setting them back to Generate passwords for Bosh.
Follow these steps to sync the vcap credentials with actual VMs.
- Login to Ops Manager
- Under, Ops Manager tile -> Security tab, select the option "Use default BOSH password"
- Click Apply changes on the Installation dashboard
- Once this is completed, go back to Ops Manager tile -> Security tab and select the option "Generate passwords" to re-generate the vcap credentials
- Click Apply changes to commit these changes
- Once done, try to login again to
vcapcredentials generated on the Credentials tab of ERT
- Once this is working, run the CFOPS backup of Elastic Runtime tile as desired
./cfops backup --opsmanagerhost <ops-mgr-URL> --adminuser <admin-user> --adminpass <admin-password> --opsmanageruser ubuntu
--opsmanagerpass <password> -d /<directory>/ --tile elastic-runtime