Pivotal Knowledge Base

Follow

HDB query fails after NameNode HA was enabled

Environment

Product Version
Pivotal HDB 1.3.x
OS RHEL 6.x
Others  

Symptom

After enabling NameNode HA on the associated HDFS, HAWQ query fails with below error messages.

Error Message:

gpadmin=# select * from test;
ERROR: Append-Only Storage Read could not open segment file 'hdfs://elephant/hawq_data/gpseg1/16385/16554/16555.1' for relation 'test' (cdbappendonlystorageread.c:369) (seg1 slice1 hdw1.phd.local:40001 pid=489421) (cdbdisp.c:1572)
DETAIL: InvalidParameter: Cannot parse URI: hdfs://elephant, missing port or invalid HA configuration Caused by: HdfsConfigNotFound: Config key: dfs.ha.namenodes.elephant not found

Cause 

name node HA has been correctly enabled on the HDFS side and on the HDB side Filespace names also have been updated accordingly. However, $GPHOME/etc/hdfs-client.xml on all nodes haven't been updated yet.

Resolution

Follow the steps below:

1. Edit $GPHOME/etc/hdfs-client.xml (it is usually /usr/local/hawq/etc/hdfs-client.xml) on all HDB nodes (masters & segments) and update the properties as below. Assuming NameNode HA has already been successfully enabled, you can find the necessary info from /etc/hadoop/conf/hdfs-site.xml. You may update it from the master node and then copy it over to all other nodes using scp (or gpscp).

<property>
<name>dfs.nameservices</name>
<value>elephant</value>
</property>
<property>
<name>dfs.ha.namenodes.elephant</name>
<value>nn1,nn2</value>
</property> <property>
<name>dfs.namenode.rpc-address.elephant.nn1</name>
<value>hdm1.phd.local:8020</value>
</property> <property>
<name>dfs.namenode.rpc-address.elephant.nn2</name>
<value>hdm1.phd.local:8020</value>
</property> <property>
<name>dfs.namenode.http-address.elephant.nn1</name>
<value>hdm1.phd.local:50070</value>
</property> <property>
<name>dfs.namenode.http-address.elephant.nn2</name>
<value>hdm1.phd.local:50070</value>
</property>

2. Once the above edits are done, then restart the HDB

$ gpstop -r


Additional Information

For more related information, please check - http://hawq.docs.pivotal.io/docs-hawq/topics/HAWQFilespacesandHighAvailabilityEnabledHDFS.html#configuregphomeetchdfsclientxml 

Comments

Powered by Zendesk