Pivotal Knowledge Base

Follow

ICM doesn't handle service start/stop/restart of hadoop-hdfs-zkfc in PHD 2.0.1

Environment

PHD 2.0.1.0

Pivotal command center(PCC) 2.2.1

Problem

hadoop-hdfs-zkfc service is installed on active and standby namenode to enable namenode HA auto failover. In PHD 2.0.1.0/PCC 2.2.1, the ICM client doesn't handle service start/stop/restart of hadoop-hdfs-zkfc.

To confirm the problem.

1: Check if hadoop namenode HA is deployed or reconfigured by icm_client. Export cluster configuration by running

icm_client fetch-configuration -l <cluser_ID> -o <output_dir>

If clusterConfig.xml under <output_dir> has contents of the HA namenode configuration, it means hadoop namenode HA was enabled through icm_client deployment or reconfigure.

 2: Restart the cluster by running

 icm_client restart -l <cluser_ID>

Monitor /var/log/message on admin node, you can see icm restarted other stacks but not hadoop-hdfs-zkfc. Also, from either namenode, you can monitor the hadoop-hdfs-zkfc by running

service hadoop-hdfs-zkfc
 
You will see the service status was not changed.
 
Impacts
 
HDFS namenode HA won't function properly if zkfc is not running. It may impact other hadoop cluster component as well. For example when submitting oozie workflows, you need to allow super user oozie to impersonate other users or groups. This is done by add a few parameters in core-site.html and restart the hadoop stack include hadoop-hdfs-zkfc. If it fails to do that, you will see problem described in this article
 
Workaround
 
This problem  has been reported in a Pivotal internal Jira HD-11123. To workaround this problem, you can make this service start with reboot by running
chkconfig hadoop-hdfs-zkfc on
 
 
 
 
 
 
 
 

Comments

Powered by Zendesk