Pivotal Knowledge Base

Follow

Loading PCC will hang without showing login page in web browser

Environment

  • PHD 2.0.1

Symptom

After upgrade to PHD 2.0.1 customer failed to load Pivotal Command Center(PCC) in any web browser. It's being hanged at loading stage and no login page could be shown up.

"service command status" output indicated "Pivotal Command Center HTTPS is not running". But PCC HTTPS port was being listened on.

[root@big1gwh001 ~]# service commander status
nodeagent is running
Jetty is running
httpd is running
Pivotal Command Center HTTPS is not running
Pivotal Command Center Background Worker is running

[root@big1gwh001 ~]# netstat -anp|grep 5443
tcp 0 0 :::5443 :::* LISTEN 12161/httpd

Cause

There are some error messages in /usr/local/greenplum-cc/log/thin*.log which indicate "port is in use" for connection request to PCC web server.

[root@big1gwh001 log]# less thin.5509.log
/usr/local/greenplum-cc/vendor/bundle/ruby/1.9.1/gems/eventmachine-1.0.0/lib/eventmachine.rb:526:in `start_tcp_server': no acceptor (port is in use or requires root privileges) (RuntimeError)
from /usr/local/greenplum-cc/vendor/bundle/ruby/1.9.1/gems/eventmachine-1.0.0/lib/eventmachine.rb:526:in `start_server'
from /usr/local/greenplum-cc/vendor/bundle/ruby/1.9.1/gems/thin-1.5.0/lib/thin/backends/tcp_server.rb:16:in `connect'
from /usr/local/greenplum-cc/vendor/bundle/ruby/1.9.1/gems/thin-1.5.0/lib/thin/backends/base.rb:55:in `block in start'
from /usr/local/greenplum-cc/vendor/bundle/ruby/1.9.1/gems/eventmachine-1.0.0/lib/eventmachine.rb:187:in `call'
from /usr/local/greenplum-cc/vendor/bundle/ruby/1.9.1/gems/eventmachine-1.0.0/lib/eventmachine.rb:187:in `run_machine'
from /usr/local/greenplum-cc/vendor/bundle/ruby/1.9.1/gems/eventmachine-1.0.0/lib/eventmachine.rb:187:in `run'
from /usr/local/greenplum-cc/vendor/bundle/ruby/1.9.1/gems/thin-1.5.0/lib/thin/backends/base.rb:63:in `start'
from /usr/local/greenplum-cc/vendor/bundle/ruby/1.9.1/gems/thin-1.5.0/lib/thin/server.rb:159:in `start'
from /usr/local/greenplum-cc/vendor/bundle/ruby/1.9.1/gems/thin-1.5.0/lib/thin/controllers/controller.rb:86:in `start'
from /usr/local/greenplum-cc/vendor/bundle/ruby/1.9.1/gems/thin-1.5.0/lib/thin/runner.rb:187:in `run_command'
from /usr/local/greenplum-cc/vendor/bundle/ruby/1.9.1/gems/thin-1.5.0/lib/thin/runner.rb:152:in `run!'
from /usr/local/greenplum-cc/vendor/bundle/ruby/1.9.1/gems/thin-1.5.0/bin/thin:6:in `<top (required)>'
from /usr/local/greenplum-cc/vendor/bundle/ruby/1.9.1/bin/thin:23:in `load'
from /usr/local/greenplum-cc/vendor/bundle/ruby/1.9.1/bin/thin:23:in `<main>'


After stop command center service, there were still some "think server" process running which were started up several months ago (back to Jan 15).

[root@big1gwh001 log]# ps -ef|grep thin|grep -v grep
gpadmin 21233 1 0 Jan15 ? 00:53:03 thin server (127.0.0.1:5500)
gpadmin 21244 1 0 Jan15 ? 00:52:46 thin server (127.0.0.1:5501)
gpadmin 21254 1 0 Jan15 ? 00:53:19 thin server (127.0.0.1:5502)
gpadmin 21266 1 0 Jan15 ? 00:53:46 thin server (127.0.0.1:5503)
gpadmin 21276 1 0 Jan15 ? 00:53:32 thin server (127.0.0.1:5504)
gpadmin 21289 1 0 Jan15 ? 00:53:14 thin server (127.0.0.1:5505)
gpadmin 21299 1 0 Jan15 ? 00:53:18 thin server (127.0.0.1:5506)
gpadmin 21310 1 0 Jan15 ? 00:53:17 thin server (127.0.0.1:5507)
gpadmin 21323 1 0 Jan15 ? 00:53:38 thin server (127.0.0.1:5508)
gpadmin 21334 1 0 Jan15 ? 00:52:52 thin server (127.0.0.1:5509)


Those stale "thin server" processes which provide HTTP service for PCC were not functioning normally and block new connections to PCC web service.

Fix

Stop command center service first and kill those stale "thin server" processes. Then start up command service again, everything returns to good. The login page can be loaded in browser successfully.

[root@big1gwh001 log]# ps -ef|grep thin|grep -v grep|awk {'print $2'}|xargs kill
[root@big1gwh001 log]# ps -ef|grep thin|grep -v grep
[root@big1gwh001 log]#

[root@big1gwh001 log]# service commander start

Starting PHD Manager Services...
Starting httpd: [ OK ]
Preparing nodeagent: [ OK ]
Starting nodeagent: [ OK ]

Starting Jetty [ OK ]
Waiting for Jetty to mount applications [ OK ]
Starting Pivotal Command Center HTTPS [ OK ]
Starting Pivotal Command Center Background Worker [ OK ]

[root@big1gwh001 log]# service commander status
nodeagent is running
Jetty is running
httpd is running
Pivotal Command Center HTTPS is running
Pivotal Command Center Background Worker is running
[root@big1gwh001 log]#

[root@big1gwh001 log]# ps -ef|grep thin
gpadmin 32966 1 0 15:36 ? 00:00:04 thin server (127.0.0.1:5500)
gpadmin 32976 1 0 15:36 ? 00:00:05 thin server (127.0.0.1:5501)
gpadmin 32987 1 0 15:36 ? 00:00:04 thin server (127.0.0.1:5502)
gpadmin 33010 1 0 15:36 ? 00:00:04 thin server (127.0.0.1:5503)
gpadmin 33021 1 0 15:36 ? 00:00:05 thin server (127.0.0.1:5504)
gpadmin 33032 1 0 15:36 ? 00:00:05 thin server (127.0.0.1:5505)
gpadmin 33044 1 0 15:36 ? 00:00:05 thin server (127.0.0.1:5506)
gpadmin 33056 1 0 15:36 ? 00:00:04 thin server (127.0.0.1:5507)
gpadmin 33067 1 0 15:36 ? 00:00:04 thin server (127.0.0.1:5508)
gpadmin 33080 1 0 15:36 ? 00:00:04 thin server (127.0.0.1:5509)

Comments

Powered by Zendesk