Pivotal Knowledge Base

Follow

How to eliminate error message "WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform..." with gphdfs

Environment

Product Version
Pivotal Greenplum (GPDB) 4.3
OS RHEL 6.x

Purpose

When querying external tables via gphdfs, sometimes a warning message such as "Unable to load native-hadoop library" is displayed.

The error message is shown below:

ERROR: external table gphdfs protocol command ended with error. 16/05/17 18:20:02 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable (seg1 slice1 GPDB1:30001 pid=28120)
DETAIL: 16/05/17 18:20:04 WARN shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded. ......

Cause

The reason for this issue is that the path for native-hadoop library files is not set correctly in hadoop_env.sh.

For example, here is the code snippet of $GPHOME/lib/hadoop/hadoop_env.sh, which has two issues:

  1. "${HADOOP_HOME}/lib/native" should be "${HADOOP_HOME}/hadoop/lib/native"
  2. On some Hadoop release (e.g. PHD 3.0.1) there is no sub-directory "${JAVA_PLATFORM}" under "${HADOOP_HOME}/hadoop/lib/native"
# setup 'java.library.path' for native-hadoop code if necessary
JAVA_LIBRARY_PATH=''
if [ -d "${HADOOP_HOME}/build/native" -o -d "${HADOOP_HOME}/lib/native" ]; then
JAVA_PLATFORM=`CLASSPATH=${CLASSPATH} ${JAVA} -Xmx32m ${HADOOP_JAVA_PLATFORM_OPTS} org.apache.hadoop.util.PlatformName | sed -e "s/ /_/g"` if [ -d "$HADOOP_HOME/build/native" ]; then
JAVA_LIBRARY_PATH=${HADOOP_HOME}/build/native/${JAVA_PLATFORM}/lib
fi if [ -d "${HADOOP_HOME}/lib/native" ]; then
if [ "x$JAVA_LIBRARY_PATH" != "x" ]; then
JAVA_LIBRARY_PATH=${JAVA_LIBRARY_PATH}:${HADOOP_HOME}/lib/native/${JAVA_PLATFORM}
else
JAVA_LIBRARY_PATH=${HADOOP_HOME}/lib/native/${JAVA_PLATFORM}
fi
fi
fi

Resolution

Correct the native-hadoop library path as shown below:

if [ -d "${HADOOP_HOME}/build/native" -o -d "${HADOOP_HOME}/hadoop/lib/native" ]; then
JAVA_PLATFORM=`CLASSPATH=${CLASSPATH} ${JAVA} -Xmx32m ${HADOOP_JAVA_PLATFORM_OPTS} org.apache.hadoop.util.PlatformName | sed -e "s/ /_/g"` if [ -d "$HADOOP_HOME/build/native" ]; then
JAVA_LIBRARY_PATH=${HADOOP_HOME}/build/native/${JAVA_PLATFORM}/lib
fi if [ -d "${HADOOP_HOME}/hadoop/lib/native" ]; then
if [ "x$JAVA_LIBRARY_PATH" != "x" ]; then
JAVA_LIBRARY_PATH=${JAVA_LIBRARY_PATH}:${HADOOP_HOME}/hadoop/lib/native/
else
JAVA_LIBRARY_PATH=${HADOOP_HOME}/hadoop/lib/native/
fi
fi
fi

Note: the change needs to be applied to all nodes of GPDB cluster.

Additional Information

For further information, please refer to the following resource: 

Understanding GPHDFS Configurations

Internal Comments

JIRA MPP-26384

 

Comments

  • Avatar
    Craig Sylvester

    Scott,

    Good information. I have one suggestion: It is not stated here that this file needs to be copied to all GPDB nodes.

    Regards,
    Craig

  • Avatar
    Scott Gai

    Hi Craig,

    Thanks for the suggestion. Just added one note to Resolution section.

Powered by Zendesk