|Ambari||1.7.1 / 2.1.2|
After upgrading from Ambari 1.7.1 to Ambari 2.1.2 if a start of all services is attempted, the restart will fail because HAWQ failed to start with the following errors in the GUI:
If the log is reviewed in the Ambari web GUI the following log entries will be seen:
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
resource_management.core.exceptions.Fail: Execution of ' hdfs dfs -chown -R gpadmin:gpadmin /hawq_data' returned 1. chown: Call From amb171.lab/192.168.177.153 to amb171.lab:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
- HAWQ tries to start before the NAMENODE is started
- The restart order is mixed up because /var/lib/ambari-server/
resources/stacks/PHD/3.0/role_ command_order.json no longer has any mention of HAWQ, it was replaced during the Ambari upgrade.
On the Ambari server node as root complete the following steps:
1. Copy the backed up role_command_order.json file to the production stack (in the command below stacks_10_03_16_16_18 will be replaced by the specific directory name on the cluster):
cp /var/lib/ambari-server/resources/stacks_10_03_16_16_18.old/PHD/3.0/role_command_order.json /var/lib/ambari-server/resources/stacks/PHD/3.0/
2. Restart ambari-server:
3. Attempt to start all services again.
NOTE: If HAWQ 18.104.22.168 is installed PXF may still fail to start, if that is the case see After upgrading to Ambari 2.1.2 from Ambari 1.7.1 PXF will not start with the error "'format_stack_version' is not defined"
Notes: Discussed in https://jira.eng.pivotal.io/browse/GPSQL-3265