Pivotal Knowledge Base

Follow

After upgrading to Ambari 2.1.2 from Ambari 1.7.1, HAWQ will not start when restarting all services

Environment

Product Version
Pivotal HD 3.0.x
Ambari 1.7.1 / 2.1.2
Pivotal HDB 1.3.x

Symptom

After upgrading from Ambari 1.7.1 to Ambari 2.1.2 if a start of all services is attempted, the restart will fail because HAWQ failed to start with the following errors in the GUI:  

Error Message:

If the log is reviewed in the Ambari web GUI the following log entries will be seen: 

  tries=tries, try_sleep=try_sleep)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of ' hdfs dfs -chown -R gpadmin:gpadmin /hawq_data' returned 1. chown: Call From amb171.lab/192.168.177.153 to amb171.lab:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused

Cause

  • HAWQ tries to start before the NAMENODE is started 
  • The restart order is mixed up because /var/lib/ambari-server/resources/stacks/PHD/3.0/role_command_order.json no longer has any mention of HAWQ, it was replaced during the Ambari upgrade.

Resolution

On the Ambari server node as root complete the following steps:

1. Copy the backed up role_command_order.json file to the production stack (in the command below stacks_10_03_16_16_18 will be replaced by the specific directory name on the cluster): 

cp /var/lib/ambari-server/resources/stacks_10_03_16_16_18.old/PHD/3.0/role_command_order.json /var/lib/ambari-server/resources/stacks/PHD/3.0/

2. Restart ambari-server:

ambari-server restart

3. Attempt to start all services again.

NOTE: If HAWQ 1.3.1.0 is installed PXF may still fail to start, if that is the case see After upgrading to Ambari 2.1.2 from Ambari 1.7.1 PXF will not start with the error "'format_stack_version' is not defined"

Internal Comments

Notes: Discussed in https://jira.eng.pivotal.io/browse/GPSQL-3265 

Comments

Powered by Zendesk