Pivotal Cloud Foundry 1.9
When attempting an add a certificate where there are already existing certificates within the ops manager>security section, UAA jobs will fail on apply changes.
Deployment failed at UAA update
Started updating instance uaa
Started updating instance uaa > uaa/4efb3606-b2a8-48fb-a10d-70c654ec71e3 (0) (canary). Failed: Action Failed get_task: Task 16ab6cf5-6c6d-43d9-5c84-76d49197508e result: 1 of 3 pre-start scripts failed. Failed Jobs: uaa. Successful Jobs: route_registrar, consul_agent. (00:00:11)
Error 450001: Action Failed get_task: Task 16ab6cf5-6c6d-43d9-5c84-76d49197508e result: 1 of 3 pre-start scripts failed. Failed Jobs: uaa. Successful Jobs: route_registrar, consul_agent.
Within the uaa logs pre-start.stdout.log you will see:
A failed run looks like this (note lack of java certificates being processed):
uaa-pre-start - completed at Fri Feb 24 19:56:43 UTC 2017
uaa-pre-start - starting at Wed Mar 15 19:49:57 UTC 2017
uaa-pre-start - starting at Wed Mar 15 20:03:38 UTC 2017
Note of uaa-pre-start script starting, but no importing of certificates.
A successful run looks like this:
uaa-pre-start - starting at Fri Feb 24 19:55:48 UTC 2017
Processing certificates for Java cacerts file
Processed certificate 1 of 175
This appears to be a bug introduced here where
set -e was added to the uaa pre-start scripts
We modified uaa pre-start script with
set -ex to see where it fails and because
diff returns exit code
1 given there are differences and
set -e is enabled the bash shell will exit pre-start and not log any errors. Diff always returns a return of 1 on mismatched results, causing the script to fall through to exit when adding certificates.
uaa/4efb3606-b2a8-48fb-a10d-70c654ec71e3:~$ /var/vcap/jobs/uaa/bin/pre-start ++ date + echo 'uaa-pre-start - starting at Fri Mar 17 15:01:52 UTC 2017' uaa-pre-start - starting at Fri Mar 17 15:01:52 UTC 2017 + CERT_FILE=/etc/ssl/certs/ca-certificates.crt + CONF_DIR=/var/vcap/jobs/uaa/config + CACHE_DIR=/var/vcap/data/uaa/cert-cache + mkdir -p /var/vcap/data/uaa/cert-cache + CERT_CACHE_FILE=/var/vcap/data/uaa/cert-cache/cacerts-cache.txt + TRUST_STORE_FILE=/var/vcap/data/uaa/cert-cache/cacerts ++ grep 'END CERTIFICATE' /etc/ssl/certs/ca-certificates.crt ++ wc -l + CERTS=177 + '[' -a /var/vcap/data/uaa/cert-cache/cacerts-cache.txt ']' + '[' -a /var/vcap/data/uaa/cert-cache/cacerts ']' + diff /var/vcap/data/uaa/cert-cache/cacerts-cache.txt /etc/ssl/certs/ca-certificates.crt
- Bosh ssh into each UAA vm
- On each UAA VM, sudo mv /var/vcap/data/uaa/cert-cache/cacerts-cache.txt /root
- Apply Changes via Ops Manager.
Upgrade to ERT 1.9.13 or later (UAA 24.6)