Pivotal Knowledge Base

Follow

Ambari Web UI became Slow or Unresponsive

Environment

 Product  Version
 Pivotal HD  3.x
 OS  RHEL 6.x

Symptom

After running out of disk space on one node in a HD cluster, the Ambari Web UI can become very slow or unresponsive even after the disk space issue has been resolved.

Cause 

The Ambari server process was busy contacting Ambari metrics collector and degraded the performance.

RCA 

1. As shown in /var/log/ambari-server/ambari-server.log, the Ambari server is constantly connecting to Ambari metric collector but is failing:

[root@AYCAPSU01AS203 ambari-server]# tail -f ambari-server.log
03 Apr 2017 11:36:31,906 ERROR [qtp-client-9752] MetricsRequestHelper:87 - Error getting timeline metrics. Can not connect to collector, socket error.
03 Apr 2017 11:36:35,891  WARN [qtp-ambari-agent-9809] HeartBeatHandler:222 - Old responseId received - response was lost - returning cached response
03 Apr 2017 11:36:40,968  WARN [qtp-ambari-agent-9787] HeartBeatHandler:222 - Old responseId received - response was lost - returning cached response
03 Apr 2017 11:36:48,105 ERROR [qtp-client-9750] MetricsRequestHelper:87 - Error getting timeline metrics. Can not connect to collector, socket error.
03 Apr 2017 11:37:04,295 ERROR [qtp-client-9755] MetricsRequestHelper:87 - Error getting timeline metrics. Can not connect to collector, socket error.
03 Apr 2017 11:37:11,348  WARN [qtp-ambari-agent-9811] HeartBeatHandler:222 - Old responseId received - response was lost - returning cached response
03 Apr 2017 11:37:20,487 ERROR [qtp-client-9755] MetricsRequestHelper:87 - Error getting timeline metrics. Can not connect to collector, socket error.

2. Ambari metrics collector was being hosted on the node where the disk space issue had occurred.

3. Therefore, the Java process of Ambari metric collector was not running in a good state due to the disk space issue and failed to process a connection request from the Ambari server.

Resolution

Restart Ambari metrics service via the Ambari Web UI. 

If the Ambari Web UI is not responsive, restart Ambari server first and quickly logon to the Web UI and then restart the Ambari metrics service immediately.  

Comments

Powered by Zendesk