Pivotal Knowledge Base

Follow

Starting PXF Service Fails giving a "NOT RUNNING" Status

Environment

 Product  Version
 Pivotal HDB  1.3.x, 2.x
 OS  RHEL 6.x

Symptom

When attempting to start a PXF service on Ambari web, its instance on one node of the cluster was not starting up. If tried to start it on that node separately, it failed with a "NOT RUNNING" status.

Error Message:

[root@sdw4 ~]# service pxf-service start
/var/gphd/pxf /
Using CATALINA_BASE: /var/gphd/pxf/pxf-service
Using CATALINA_HOME: /opt/vmware/vfabric-tc-server-standard/tomcat-7.0.55.A.RELEASE
Using CATALINA_TMPDIR: /var/gphd/pxf/pxf-service/temp
Using JRE_HOME: /usr/jdk64/jdk1.7.0_67
Using CLASSPATH: /opt/vmware/vfabric-tc-server-standard/tomcat-7.0.55.A.RELEASE/bin/bootstrap.jar:/opt/vmware/vfabric-tc-server-standard/tomcat-7.0.55.A.RELEASE/bin/tomcat-juli.jar
Using CATALINA_PID: /var/gphd/pxf/pxf-service/logs/tcserver.pid
Tomcat started.
Status: NOT RUNNING 

Cause

PXF service reserved port 51200 was being used by another process. 

RCA

Reviewed PXF service's Catalina logs and found the following error messages:

10-Feb-2017 05:12:52.793 SEVERE [main] org.apache.catalina.core.StandardService.initInternal Failed to initialize connector [Connector[HT
TP/1.1-51200]]
org.apache.catalina.LifecycleException: Failed to initialize component [Connector[HTTP/1.1-51200]]
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:106)
at org.apache.catalina.core.StandardService.initInternal(StandardService.java:559)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:102)
at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:821)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:102)
at org.apache.catalina.startup.Catalina.load(Catalina.java:638)
at org.apache.catalina.startup.Catalina.load(Catalina.java:663)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:280)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:454)
Caused by: org.apache.catalina.LifecycleException: Protocol handler initialization failed
at org.apache.catalina.connector.Connector.initInternal(Connector.java:980)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:102)
... 12 more
Caused by: java.net.BindException: Address already in use <null>:51200
at org.apache.tomcat.util.net.JIoEndpoint.bind(JIoEndpoint.java:411)
at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:646)
at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:434)
at org.apache.coyote.http11.AbstractHttp11JsseProtocol.init(AbstractHttp11JsseProtocol.java:119)
at org.apache.catalina.connector.Connector.initInternal(Connector.java:978)
... 13 more
Caused by: java.net.BindException: Address already in use
at java.net.PlainSocketImpl.socketBind(Native Method)
at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:376)
at java.net.ServerSocket.bind(ServerSocket.java:376)
at java.net.ServerSocket.<init>(ServerSocket.java:237)
at java.net.ServerSocket.<init>(ServerSocket.java:181)
at org.apache.tomcat.util.net.DefaultServerSocketFactory.createSocket(DefaultServerSocketFactory.java:49)
at org.apache.tomcat.util.net.JIoEndpoint.bind(JIoEndpoint.java:398)
... 17 more

A further check of port state found that the port 51200 was taken by the curl process already.

[root@sdw4 logs]# netstat -anp|grep 51200
tcp 0 0 127.0.0.1:51200 127.0.0.1:51200 ESTABLISHED 49546/curl
[root@sdw4 logs]# ps -ef|grep 49546
root 49546 49545 0 02:02 ? 00:00:00 curl --silent --connect-timeout 1 -I http://localhost:51200

Resolution

Kill the stale curl process and start the PXF service again.  

Comments

Powered by Zendesk