Pivotal Knowledge Base

Follow

How to manually start Pivotal HD components if Ambari is down?

Environment

Product Version
Pivotal HD 3.x
Ambari 1.7 / 2.x

Purpose

In case Ambari is running into issues and can no longer start Hadoop components it may be necessary to manually start Hadoop components in order to give immediate relief. This knowledge bases explains the order in which the components should be started and the commands that can be used to start them.

Procedure

Components should be restarted in this order:

  • Ranger
  • Knox
  • ZooKeeper
  • HDFS
  • YARN
  • HBase
  • Hive Metastore
  • HiveServer2
  • WebHCat
  • HAWQ / Pivotal HDB
  • PXF
  • Oozie 
  • Hue
  • Spark
  • Ambari Metrics

The commands below can be run as root to restart the individual components, failure to follow the order given above may cause some commands to fail. The commands will need to be run on each host that has the component installed. 

NOTE: If the Ambari database is up the mapping between hosts and components can be obtained by logging in the Ambari database and running the SQL:

select host_name,component_name from hostcomponentdesiredstate,hosts where hostcomponentdesiredstate.host_id=hosts.host_id order by component_name; 

RANGER:

service ranger-admin start
service ranger-usersync start 

KNOX:

su -l knox -c "/usr/phd/current/knox-server/bin/gateway.sh start"

ZOOKEEPER:

su -l zookeeper -c "export ZOOCFGDIR=/usr/phd/current/zookeeper-server/conf ; export ZOOCFG=zoo.cfg; source /usr/phd/current/zookeeper-server/conf/zookeeper-env.sh ; /usr/phd/current/zookeeper-server/bin/zkServer.sh start"

HDFS:

If the cluster is configured for NameNode HA - start the JournalNodes:

su -l hdfs -c "/usr/phd/current/hadoop-hdfs-journalnode/../hadoop/sbin/hadoop-daemon.sh --config /etc/hadoop/conf/ start journalnode"

Start the NameNodes(s):

su -l hdfs -c "/usr/phd/current/hadoop-hdfs-namenode/../hadoop/sbin/hadoop-daemon.sh --config /etc/hadoop/conf/ start namenode"

If the cluster has NameNode HA configured, start the Zookeeper Failover Controller (ZKFC) by executing the following command on all NameNode machines:

su -l hdfs -c "/usr/phd/current/hadoop-hdfs-namenode/../hadoop/sbin/hadoop-daemon.sh start zkfc"

If not running NameNode HA, execute the following command on the Secondary NameNode host machine:

su -l hdfs -c "/usr/phd/current/hadoop-hdfs-secondarynamenode/../hadoop/sbin/hadoop-daemon.sh --config /etc/hadoop/conf/ start secondarynamenode"

On all DataNodes:

su -l hdfs -c "/usr/phd/current/hadoop-hdfs-datanode/../hadoop/sbin/hadoop-daemon.sh --config /etc/hadoop/conf/  start datanode"

YARN:

ResourceManager:

su -l yarn -c "/usr/phd/current/hadoop-yarn-resourcemanager/sbin/yarn-daemon.sh --config /etc/hadoop/conf/ start resourcemanager"

NodeManagers:

su -l yarn -c "/usr/phd/current/hadoop-yarn-nodemanager/sbin/yarn-daemon.sh --config /etc/hadoop/conf/ start nodemanager" 

History Server host machine:

su -l mapred -c "/usr/phd/current/hadoop-mapreduce-historyserver/sbin/mr-jobhistory-daemon.sh start historyserver"

App Timeline Server:

su -l yarn -c "/usr/phd/current/hadoop-yarn-timelineserver/sbin/yarn-daemon.sh --config /etc/hadoop/conf/ start timelineserver"

HBase:

HBase Master host machine:

su -l hbase -c "/usr/phd/current/hbase-master/bin/hbase-daemon.sh start master; sleep 25"

On all RegionServers:

su -l hbase -c "/usr/phd/current/hbase-regionserver/bin/hbase-daemon.sh start regionserver"

HBASE Thrift API:

/usr/bin/hbase thrift start

HBASE REST API:

/usr/phd/current/hbase/bin/hbase-daemon.sh start rest --infoport 8085

HIVE:

Hive Metastore:

service mysqld start
su -l hive -c "nohup /usr/phd/current/hive-metastore/bin/hive --service metastore>/var/log/hive/hive.out 2>/var/log/hive/hivemetastore.log &"

HiveServer2:

su -l hive -c "nohup /usr/phd/current/hive-server2/bin/hiveserver2 >/var/log/hive/hiveserver2.out 2> /var/log/hive/hiveserver2.log &"

WebHCat Server: 

su -l hcat -c '/usr/phd/current/hive-webhcat/sbin/webhcat_server.sh --config /etc/hive-webhcat/conf/ start'

HAWQ / PIVOTAL HDB: 

On the HAWQ master:

HAWQ 1.3.1 and earlier - See the Pivotal HDB documentation for further details.

su -l gpadmin -c "gpstart"

Pivotal HDB 2.0 - See the Pivotal HDB 2.0 Documentation for further details:

su -l gpadmin -c "hawq start cluster"

PXF:

service pxf-service start

OOZIE:

su -l oozie -c "/usr/phd/current/oozie-server/bin/oozied.sh start"

HUE:

/etc/init.d/hue start

SPARK:

/usr/phd/current/spark/sbin/start-history-server.sh

AMBARI-METRICS:

ambari-metrics-collector start
ambari-metrics-monitor start

NOTE: Once the Ambari issues are resolved, it is recommended to stop and start all services via Ambari in order to make sure the Hadoop components and Ambari are in sync. 

Comments

Powered by Zendesk