Pivotal Knowledge Base

Follow

PXF and History Server cannot start when HDFS is stored in Isilon

Environment

Product Version
Pivotal HD 3.0.x
Ambari  2.1.2

Symptom

When Pivotal HD is attached to Isilon the restart of HIVESERVER2 or the History server may fail with to start with the following message:

2016-08-22 00:40:33,845 - checked_call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET '"'"'http://namenode:50070/webhdfs/v1/user/hcat?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmp91qV1C 2>/tmp/tmpIgMfIh''] {'logoutput': None, 'quiet': False} 

Cause

Isilon is using a non-default webHDFS port and Ambari has not been configured to use the correct port. This can happen after an Ambari upgrade from 1.7.1 to 2.1.2 as Ambari 1.7.1 did not check the webHDFS service was started before starting HIVE or History server.

Resolution

1. Confirm the port that is being used for WebHDFS (default is 8082) by logging into the Isilon and running the command: 

isi_gconfig -t web-config  webhdfs_port

2. In Ambari go into HDFS / Configs / Advanced and modify the configuration dfs.namenode.http-address to use the correct port for webHDFS as found in step one: 

dfs.namenode.http-address=isinn.domain.local:8082

3. Restart HDFS as recommended by Ambari 

4. Attempt to start HIVESERVER2 and HISTORY SERVER once again.

 

Comments

Powered by Zendesk