Pivotal Knowledge Base


After upgrading to Ambari 2.1.2 from Ambari 1.7.1, HAWQ will not start when restarting all services


Product Version
Pivotal HD 3.0.x
Ambari 1.7.1 / 2.1.2
Pivotal HDB 1.3.x


After upgrading from Ambari 1.7.1 to Ambari 2.1.2 if a start of all services is attempted, the restart will fail because HAWQ failed to start with the following errors in the GUI:  

Error Message:

If the log is reviewed in the Ambari web GUI the following log entries will be seen: 

  tries=tries, try_sleep=try_sleep)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of ' hdfs dfs -chown -R gpadmin:gpadmin /hawq_data' returned 1. chown: Call From amb171.lab/ to amb171.lab:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused


  • HAWQ tries to start before the NAMENODE is started 
  • The restart order is mixed up because /var/lib/ambari-server/resources/stacks/PHD/3.0/role_command_order.json no longer has any mention of HAWQ, it was replaced during the Ambari upgrade.


On the Ambari server node as root complete the following steps:

1. Copy the backed up role_command_order.json file to the production stack (in the command below stacks_10_03_16_16_18 will be replaced by the specific directory name on the cluster): 

cp /var/lib/ambari-server/resources/stacks_10_03_16_16_18.old/PHD/3.0/role_command_order.json /var/lib/ambari-server/resources/stacks/PHD/3.0/

2. Restart ambari-server:

ambari-server restart

3. Attempt to start all services again.

NOTE: If HAWQ is installed PXF may still fail to start, if that is the case see After upgrading to Ambari 2.1.2 from Ambari 1.7.1 PXF will not start with the error "'format_stack_version' is not defined"

Internal Comments

Notes: Discussed in https://jira.eng.pivotal.io/browse/GPSQL-3265 


Powered by Zendesk