Pivotal Knowledge Base

Follow

gpload fails in Mac OS 10.11.4 El Capitan

Environment

Product Version
Pivotal Greenplum (GPDB) 4.3.x
OS Mac OSX 10.11.4 "El Capitan" or greater
Environment System Integrated Protection (SIP) enabled (enabled by default in El Capitan)

Symptom

When attempting to run gpload on Mac OSX El Capitan it will fail with the error 

"gpload was unable to import The PyGreSQL Python module"

Error Message:

Pivotals-Mac:lib pivotal$ sw_vers
ProductName:    Mac OS X
ProductVersion: 10.11.4
BuildVersion:   15E65

Pivotals-Mac:lib pivotal$ gpload -v gpload was unable to import The PyGreSQL Python module (pg.py) -
dlopen(/Users/pivotal/greenplum-loaders-4.3.8.0-build-1/bin/ext/pygresql/_pg.so, 2):
Library not loaded:
/Volumes/data/pulse2-agent/agents/agent1/work/GPDB-4_3_x-rcbuilds-1repo/osx106_x86/4.3.8.0-build-1_output/greenplum-db-4.3.8.0-build-1/lib/libpq.5.dylib Referenced from: /Users/pivotal/greenplum-loaders-4.3.8.0-build-1/bin/ext/pygresql/_pg.so Reason: no suitable image found.
Did find: /usr/lib/libpq.5.dylib: mach-o, but wrong architecture

Cause 

El Capitan introduced a new feature, System Integrated Protection (SIP), that locks down many areas including relative paths to shared libraries to all users including root. Since the gpload program is using relative paths it is unable to load the shared libraries required.

More details of SIP can be reviewed on Apple's developer site:  https://developer.apple.com/library/mac/documentation/Security/Conceptual/System_Integrity_Protection_Guide/Introduction/Introduction.html

This is being tracked by Pivotal Engineering under MPP-26292.

RCA 

gpload (from the GP loaders package) breaks in Mac OSX El Capitan due to System Integrated Protection (SIP) that was introduced in El Capitan and locks down many areas including relative paths to shared libraries.

Whether you had gpload working in a prior release of Mac OSX (eg. Yosemite) and upgrade to EL Capitan or start with a fresh install of El Capitan, gpload will fail with:

gpload failure
Pivotals-Mac:lib pivotal$ gpload -v
gpload was unable to import The PyGreSQL Python module (pg.py) - dlopen(/Users/pivotal/greenplum-loaders-4.3.8.0-build-1/bin/ext/pygresql/_pg.so, 2): Library not loaded: /Volumes/data/pulse2-agent/agents/agent1/work/GPDB-4_3_x-rcbuilds-1repo/osx106_x86/4.3.8.0-build-1_output/greenplum-db-4.3.8.0-build-1/lib/libpq.5.dylib
  Referenced from: /Users/pivotal/greenplum-loaders-4.3.8.0-build-1/bin/ext/pygresql/_pg.so
  Reason: no suitable image found.  Did find:
    /usr/lib/libpq.5.dylib: mach-o, but wrong architecture

You can see with otool -L that the shared library is pointing to a non-existent path which is the same path in the error that is not loaded. 

otool -L output
Pivotals-Mac:bin pivotal$ otool -L /Users/pivotal/greenplum-loaders-4.3.8.0-build-1/bin/ext/pygresql/_pg.so
/Users/pivotal/greenplum-loaders-4.3.8.0-build-1/bin/ext/pygresql/_pg.so:
	/Volumes/data/pulse2-agent/agents/agent1/work/GPDB-4_3_x-rcbuilds-1repo/osx106_x86/4.3.8.0-build-1_output/greenplum-db-4.3.8.0-build-1/lib/libpq.5.dylib (compatibility version 5.0.0, current version 5.3.0)
	/opt/tools_build/tools/gnu/gcc/4.4.2/dist/osx106_x86_32/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 169.3.0)
SIP is looking for shared libraries only in /usr/bin and /usr/local bin but only finds 64 bit ones when it wants 32bit ones so these don't work either:
 
lipo -info output
Pivotals-Mac:bin pivotal$ lipo -info /usr/lib/libpq.5.dylib 
Non-fat file: /usr/lib/libpq.5.dylib is architecture: x86_64

Resolution

Workaround

Two workarounds to this issue are detailed below.

Workaround #1

<loaderdir>/bin/gpload is a simple shell script that sets some environment variables and launches the <loaderdir>/bin/gpload.py Python script with exec gpload.py

Pivotals-Mac:greenplum-loaders-4.3.8.0-build-1 pivotal$ cat /Users/pivotal/greenplum-loaders-4.3.8.0-build-1/bin/gpload
#!/bin/sh
if [ ! -z "$GPHOME" ]; then
    . $GPHOME/greenplum_path.sh
fi
if [ ! -z "$GPHOME_LOADERS" ]; then
    . $GPHOME_LOADERS/greenplum_loaders_path.sh
fi
if [ `uname -s` = "AIX" -a ! -z "$GP_LIBPATH_FOR_PYTHON" ]; then
    LIBPATH=$GP_LIBPATH_FOR_PYTHON:$LIBPATH
    export LIBPATH
fi
exec gpload.py

Executing gpload.py this way appears to break in El Capitan as its trying to find all the required shared libraries in /usr/lib and /usr/local/lib and not the realtive path of them.

However, if we just run python gpload.py <options> we can get around this:

Pivotals-Mac:greenplum-loaders-4.3.8.0-build-1 pivotal$ python ./bin/gpload.py -f ./test.yaml
2016-04-05 14:54:03|INFO|gpload session started 2016-04-05 14:54:03
2016-04-05 14:54:03|INFO|setting schema 'public' for table 'gpload_test'
2016-04-05 14:54:03|INFO|started gpfdist -p 8081 -P 8082 -f "/Users/pivotal/greenplum-loaders-4.3.8.0-build-1/load/data/*" -t 30

 

Workaround #2

Since El Capitan is looking for shared libraries in /usr/lib (which we cannot write to or create even a new symlink in) and /usr/local/lib (which we can modify as root) we could copy the required libraries into /usr/local/lib directly.

In the case of the new El Capitan install, I did not have /usr/local/lib so I had to manually create it.

Pivotals-Mac:lib pivotal$ sudo mkdir /usr/local/lib
Pivotals-Mac:lib pivotal$ sudo cp ./libpq.5.dylib /usr/local/lib/libpq.5.dylib

Copying over just libpq.5.dylib failed with additional dependencies so I copied the full directory:

Pivotals-Mac:lib pivotal$ gpload
gpload was unable to import The PyGreSQL Python module (pg.py) - dlopen(/Users/pivotal/greenplum-loaders-4.3.8.0-build-1/bin/ext/pygresql/_pg.so, 2): Library not loaded: /opt/tools_build/tools/krb5/1.11.3/dist/osx106_x86/lib/libkrb5.3.3.dylib
  Referenced from: /usr/local/lib/libpq.5.dylib
  Reason: image not found
Pivotals-Mac:lib pivotal$ ls
libcom_err.3.0.dylib        libkrb5.3.3.dylib       liblber-2.3.0.dylib     libssl.0.9.8.dylib
libcom_err.3.dylib      libkrb5.3.dylib         libldap_r-2.3.0.2.26.dylib  libyaml-0.1.0.0.dylib
libcrypto.0.9.8.dylib       libkrb5support.1.1.dylib    libldap_r-2.3.0.dylib       libyaml-0.1.dylib
libk5crypto.3.1.dylib       libkrb5support.1.dylib      libpq.5.3.dylib         libyaml.dylib
libk5crypto.3.dylib     liblber-2.3.0.2.26.dylib    libpq.5.dylib
Pivotals-Mac:lib pivotal$ sudo cp ./* /usr/local/lib/

Now gpload works successfully.

Pivotals-Mac:lib pivotal$ gpload
gpload [options] -f configuration file

Options:
    -h hostname: host to connect to
    -p port: port to connect to
    -U username: user to connect as
    -d database: database to connect to
    -W: force password authentication
    -q: quiet mode
    -D: do not actually load data
    -v: verbose
    -V: very verbose
    -l logfile: log output to logfile
    --no_auto_trans: do not wrap gpload in transaction
    --gpfdist_timeout timeout: gpfdist timeout value
    --version: print version number and exit
    -?: help

 

Comments

Powered by Zendesk