Pivotal Knowledge Base

Follow

After adding Pivotal HDB segments via Ambari, the new segments will not start

Environment

Product Version
Pivotal HD (PHD) 3.0.x
Pivotal HDB 1.3.x
Ambari 1.7.1 / 2.1.2

Symptom

When trying to start newly added segments via Ambari, they appear to come up for approximately 60 seconds, then show as stopped. Production is not affected but the following symptoms will be seen:

  • Ambari will show a number of HAWQ segments as not live:

  •  In Ambari on the newly added node, the HAWQ Segment(s) will be stopped:

 

  • When trying to start, the "Start HAWQ Segment" will complete and the HAWQ segment will show up for about 60 seconds. Then it will show as stopped again as in the above screenshot:

 

  • The segments directory on the affected node will be empty:
[root@dn4 ~]# ls -ltr /data/hawq/segments/
total 0
[root@dn4 ~]#

Cause

This issue is seen because adding HAWQ segments in Ambari version 1.7.1 or 2.1.2 is not supported.

Although Ambari can install the software and set up some operating system configurations on new nodes, the segments will not be correctly initialized, causing the startup of the segments to fail.

Resolution

HAWQ and AMBARI need to be put back in sync to make sure both report the same number of segments. So there are two options:

A) Initialize the segments via CLI (Recommended)

Follow the Pivotal HDB documents to correctly expand the Pivotal HDB cluster. 

Note: Read all the steps in the procedure before starting so as to fully understand the impact of every step taken. 

Note: The exchange of the root SSH keys as part of the expand procedure should already have been completed while adding the nodes in the Hadoop cluster via Ambari.

OR

 

B) Remove the segments from Ambari (Not recommended):

If the segments do need to be removed (not recommended), the segment entry in Ambari can be removed via the Ambari API and packages removed by following these steps:

1. Remove the HAWQ SEGMENT from Ambari by running this command from the Ambari node:

curl -u admin:admin -H "X-Requested-By: ambari" -X DELETE http://<AMBARI_FQDN>:8080/api/v1/clusters/<CLUSTER_NAME>/hosts/<AFFECTED_HOST>/host_components/HAWQSEGMENT

Warning: this will remove the HAWQ segment in Ambari, so ensure that the correct host is chosen in the URL.

 2. Locate the HAWQ package.

[root@dn4 ~]# yum list installed hawq
Loaded plugins: fastestmirror, security
Loading mirror speeds from cached hostfile
* base: anorien.csc.warwick.ac.uk
* extras: centos.mirroring.pulsant.co.uk
* updates: www.mirrorservice.org
Installed Packages
hawq.x86_64 1.3.1.1-19631 @PADS

3. Remove the HAWQ package.

root@dn4 ~]# yum remove hawq.x86_64
Loaded plugins: fastestmirror, security
Setting up Remove Process
Resolving Dependencies
--> Running transaction check
---> Package hawq.x86_64 0:1.3.1.1-19631 will be erased
--> Finished Dependency Resolution Dependencies Resolved <...> Installed size: 458 M Is this ok [y/N]: y
Downloading Packages:
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Erasing : hawq-1.3.1.1-19631.x86_64 1/1
Verifying : hawq-1.3.1.1-19631.x86_64 1/1 Removed:
hawq.x86_64 0:1.3.1.1-19631 Complete!

4. Remove any segment or master directories that may have been created on the node.

 

 

Comments

Powered by Zendesk