Pivotal Knowledge Base

Follow

Ambari user interface is extremely slow

Environment

Product Version
Pivotal HD 3.0.x
OS RHEL 6.x
Ambari 1.7.1

Symptom

Ambari Web user interface is extreme slow when checking service performance metrics and ambari-server.log shows Ganglia host/server is not live or known.

Error Message:

Messages from ambari-server.log:

:
04:42:59,509 INFO [qtp334484437-166379] GangliaPropertyProvider:452 - Ganglia host is not live
04:43:00,693 INFO [qtp334484437-166379] GangliaPropertyProvider:452 - Ganglia host is not live
04:43:05,500 INFO [qtp334484437-166379] GangliaPropertyProvider:452 - Ganglia host is not live
04:43:06,905 INFO [qtp334484437-166379] GangliaPropertyProvider:452 - Ganglia host is not live
04:43:11,503 INFO [qtp334484437-166379] GangliaPropertyProvider:452 - Ganglia host is not live
04:43:12,394 WARN [qtp334484437-166379] GangliaReportPropertyProvider:116 - Attempting to get metrics but the Ganglia server is unknown. Resource=Resource : Cluster
Properties:
{Clusters={cluster_name=phdcluster, version=PHD-3.0}} : Cluster=phdcluster
04:43:12,396 WARN [qtp334484437-166379] GangliaReportPropertyProvider:116 - Attempting to get metrics but the Ganglia server is unknown. Resource=Resource : Cluster
Properties:
{Clusters={cluster_name=phdcluster, version=PHD-3.0}} : Cluster=phdcluster
:

Cause 

With Ambari versions before 2.1.2, HBase per region metrics are enabled by default. When a cluster has too many regions this will cause severe performance issue. One example cluster showed, it had 2600 regions and each region had 30 metrics which had 2600*30 = 78000 metrics to be maintained and showed the performance issue.

Resolution

1. Locate below two files under /var/lib/ambari-server/resources/stacks/PHD/2.0.6/services/HBASE/package/templates 

hadoop-metrics2-hbase.properties-GANGLIA-MASTER.j2
hadoop-metrics2-hbase.properties-GANGLIA-RS.j2

2. Add below two entries to each file

*.source.filter.class=org.apache.hadoop.metrics2.filter.GlobFilter
hbase.*.source.filter.exclude=*Regions*

3. Do the rolling restart of Hbase region servers



Comments

Powered by Zendesk