Pivotal Knowledge Base

Follow

Loading PCC in Web browser failed with "Service Temporarily Unavailable"

Environment

Product Version
Pivotal HD 2.x
OS RHEL 6.x
Others  

Symptom

When attempting to load PCC in a web browser it failed with the error "Service Temporarily Unavailable" as shown in the screenshot below.

 

Cause

Pivotal Command Center HTTPS service ("thin server" processes") was not running which caused failure of loading PCC website.

RCA

1. Checked PCC service status and found HTTPS service was not running. And there wasn't any running "thin server" process in system

[root@mdw ~]# service commander status
nodeagent is running
Jetty is running
httpd is running
Pivotal Command Center HTTPS is not running
Pivotal Command Center Background Worker is running
[root@mdw ~]# ps -ef|grep thin
root 785490 744950 0 03:45 pts/43 00:00:00 grep thin

2. "thin server" logs indicated "No space left on device" error which caused "thin server" processes to exit abnormally

[root@mdw ~]# head /usr/local/pivotal-cc/log/thin.5501.log
/usr/local/pivotal-cc/vendor/bundle/ruby/1.9.1/gems/sequel-3.41.0/lib/sequel/adapters/postgres.rb:208: warning: Failed to set the default_internal encoding to UTF8: 'ERROR: failed to acquire resources on segment 0
'
/usr/local/pivotal-cc/vendor/bundle/ruby/1.9.1/gems/sequel-3.41.0/lib/sequel/adapters/postgres.rb:208: warning: Failed to set the default_internal encoding to UTF8: 'ERROR: failed to acquire resources on segment 0
'
log writing failed. No space left on device - /usr/local/pivotal-cc/log/production.log
log writing failed. No space left on device - /usr/local/pivotal-cc/log/production.log
log writing failed. No space left on device - /usr/local/pivotal-cc/log/production.log
......

3. Restart of PCC service could not solve the issue

4. Further check found some stale (empty) PID files for "thin server" process, which blocked new "thin server" processes from starting up

[root@mdw ~]# ls -l /var/tmp
total 0
-rw-r--r-- 1 gpadmin gpadmin 0 Aug 18 09:11 commander.5500.pid
-rw-r--r-- 1 gpadmin gpadmin 0 Aug 18 09:11 commander.5501.pid
-rw-r--r-- 1 gpadmin gpadmin 0 Aug 18 09:11 commander.5502.pid
-rw-r--r-- 1 gpadmin gpadmin 0 Aug 18 09:11 commander.5503.pid
-rw-r--r-- 1 gpadmin gpadmin 0 Aug 18 09:11 commander.5504.pid
-rw-r--r-- 1 gpadmin gpadmin 0 Aug 18 09:11 commander.5505.pid
-rw-r--r-- 1 gpadmin gpadmin 0 Aug 18 09:11 commander.5506.pid
-rw-r--r-- 1 gpadmin gpadmin 0 Aug 18 09:11 commander.5507.pid
-rw-r--r-- 1 gpadmin gpadmin 0 Aug 18 09:11 commander.5508.pid
-rw-r--r-- 1 gpadmin gpadmin 0 Aug 18 09:11 commander.5509.pid

5. As mentioned in this article "Pivotal Command Center HTTPS is not running" will be shown when there is not enough "thin server" processes up and running

Resolution

1. Check "thin server" logs for any errors and fix them. In this case, do some cleanup to free more disk space in root partition (/)

2. Remove any stale PID files for "thin server" process before start commander service

Additional Information

PCC HTTPS is not running  

 

Comments

Powered by Zendesk