Pivotal Knowledge Base

Follow

HDB PXF error Couldn't find the REST server on the data-node

Environment

  • PHD 2.1.0.0
  • HDB 1.2.1.0

Symptom

gpadmin=# CREATE EXTERNAL TABLE external_table_hawq(name text) 
gpadmin-# LOCATION('pxf://192.168.3.5:50070/data/external_table_hawq.txt?PROFILE=HdfsTextSimple')FORMAT 'TEXT'; 
CREATE EXTERNAL TABLE 
gpadmin=# select * from external_table_hawq; 
NOTICE: External scan using PXF protocol will utilize 66 out of 132 segment databases 
NOTICE: External scan using PXF protocol will utilize 66 out of 132 segment databases 
ERROR: Couldn't find the REST server on the data-node: 10.10.10.10

Cause

When executing a PXF query hawq tries to match the host running the segment process to the host where HDFS datablock ( fragment ) resides. Hawq will send a API call to the PXF service running on the namenode and receive a list of fragments containing ip addresses. Hawq will then compare the ip address from the fragments list to the resolved hostnamed from its gp_segment_configuration table. If the namenode and hawq master resolve the hostnames to different IP addresses then you will see the above error.  This is commonly found in environments that have dual network interfaces with inconsistent host to ip address mappings. 

Troubleshooting Tips

Download GetFragmants.jar ( source code here ) and execute it against a file in hdfs.  This tool needs to be run from the hadoop namenode in order see what hostnames the namenode is resolving the fragment hosts too.  . If the ip subnet returned by the pxf fragment is different from hawq then the most common cause of this issue is namenode resolves the IP addresses differently from then the hawq master

 [gpadmin@hdm1 ~]$ /usr/java/default/bin/java -classpath GetFragments.jar:`hadoop classpath` io.pivotal.support.pxf.api.utilities.GetFragments /tmp/file.txt
 Setting up request params
 creating fragmenter
 getting fragments
 dumping fragments
 {"PXFFragments":[{"userData":null,"replicas":["192.168.3.6","192.168.3.7","192.168.3.8"],"metadata":"rO0ABXcQAAAAAAAAAAAAAAAAAAABdHVyABNbTGphdmEubGFuZy5TdHJpbmc7rdJW5+kde0cCAAB4cAAAAAN0AA5oZHcyLnBoZC5sb2NhbHQADmhkdzMucGhkLmxvY2FsdAAOaGR3MS5waGQubG9jYWw=","sourceName":"/tmp/file.txt","index":0}]}

Comments

Powered by Zendesk