Pivotal Knowledge Base

Follow

Create table fails with HAWQ Kerberized: "WARNING: failed to login to Kerberos, command line: kinit -k -t keytab_file -c /tmp/postgres.ccname postgres"

Environment

Pivotal HDB (HAWQ): 2.x

OS: RHEL 6.x

Kerberos

Microsoft Active Directory

Symptom

After setting up HAWQ to use Kerberos for secure authentication with Windows Active Directory, upon connecting to Pivotal HDB using the kerberized user and trying to create a database object, you get the "failed to login to Kerberos, command line:" warning. 

Scenario 1

HAWQ Service configuration parameter "krb_server_keyfile" is set to /etc/security/keytabs/hawq.service.keytab

You also have generated a postgres principal keytab file /path/to/gpadmin/keytab_file.keytab

You can get a Kerberos ticket using postgres principal from above:

[gpadmin@host ~]$ export KRB5CCNAME=/tmp/postgres.ccname
[gpadmin@host ~]$ kinit -kt /path/to/gpadmin/keytab_file.keytab
postgres/fully.qualified.domain.name@YOUR.REALM.COM 

The klist output shows your Ticket cache with principal similar to below:

[gpadmin@host ~]$ klist
Ticket cache: FILE:/tmp/krb5cc_30863
Default principal: postgres/fully.qualified.domain.name@YOUR.REALM.COM

Valid starting       Expires              Service principal
06/09/2017 01:55:46  06/09/2017 06:25:46  krbtgt/YOUR.REALM.COM@YOUR.REALM.COM
        renew until 06/16/2017 01:55:46
[gpadmin@host ~]$ 

And you can connect to the Master host using psql. For example:

[gpadmin@host ~]$ psql -U gpadmin -h fully.qualified.domain.name template1
psql (8.2.15)
Type "help" for help.

template1=#
template1=#
template1=# \c gpadmin
You are now connected to database "gpadmin" as user "gpadmin".
gpadmin=# 

But you get a WARNING when trying to create a table object. And it fails:

gpadmin=#
gpadmin=# create table test (col int);
WARNING:  failed to login to Kerberos, command line: kinit -k -t /etc/security/keytabs/hawq.service.keytab -c /tmp/postgres.ccname postgres
WARNING:  failed to login to Kerberos, command line: kinit -k -t /etc/security/keytabs/hawq.service.keytab -c /tmp/postgres.ccname postgres
CONTEXT:  Dropping file-system object -- Relation Directory: '16385/16508/16577'
WARNING:  could not remove relation directory 16385/16508/16577: Permission denied
CONTEXT:  Dropping file-system object -- Relation Directory: '16385/16508/16577'
ERROR:  could not create relation directory hdfs://namenode/hawq_data/16385/16508/16577: Permission denied
gpadmin=# 

If you run the kinit command manually, you get:

gpadmin=#
[gpadmin@host ~]$ kinit -k -t /etc/security/keytabs/hawq.service.keytab -c /tmp/postgres.ccname postgres
kinit: Client 'postgres@YOUR.REALM.COM' not found in Kerberos database while getting initial credentials
[gpadmin@host ~]$

Cause

This is because of principal "postgres@YOUR.REALM.COM," which is included in this hawq.service.keytab  keytab file is not defined in the AD server.

Also note:  The HAWQ params "krb_server_keyfile" and "krb_srvname" must reflect the correct principal.  And that principal must exist in the Active Directory. 

Resolution

Add principal "postgres/@YOUR.REALM.COM" on AD server and still use the value of "hawq.service.keytab" parameter for "krb_server_keyfile"

Then restart the HAWQ cluster. 

Scenario 2

HAWQ Service configuration parameter "krb_server_keyfile" is set to /path/to/gpadmin/keytab_file.keytab

Upon connecting to HAWQ and attempting to create table object, it too fails but with WARNING:

WARNING:  failed to login to Kerberos, command line: kinit -k -t /path/to/gpadmin/keytab_file.keytab -c /tmp/postgres.ccname postgres 

You run that kinit command manually and see a reason is that "Keytab contains no suitable keys." For example:

[gpadmin@zooqa002 master]$ kinit -k -t  /path/to/gpadmin/keytab_file.keytab  -c /tmp/postgres.ccname postgres
kinit: Keytab contains no suitable keys for postgres@YOUR.REALM.COM while getting initial credentials

Cause

The principal "postgres@YOUR.REALM.COM" is not included in the keytab file and does not reflect what you have currently set with the HAWQ "krb_srvname" parameter. 

For example:

The keytab_file.keytab might have the postgres principal entry like:

postgres/fully.qualified.server.name@YOUR.REALM.COM 

But you do not have "krb_srvname" showing the same value of:

postgres/fully.qualified.server.name

Resolution

Change the HAWQ "krb_srvname" to be consistent with the server name you have for your postgres principal.

Then restart the HAWQ cluster. 

Comments

Powered by Zendesk