Pivotal Knowledge Base

Follow

Fails to access HDFS secured with Kerberos

Environment

Product Version
Pivotal HD 3.0.x
OS RHEL 6.x

Symptom

The attempt to access kerberized Hadoop Distributed File System (HDFS) on any cluster host fails with error message "SIMPLE authentication is not enabled" though all Kerberos parameters are configured correctly. 

Error Message:

# hdfs dfs -ls /
ls: Failed on local exception: java.io.IOException: Couldn't setup connection for hdfs@EXAMPLE.COM to node01.example.com/172.28.21.2:8020; Host Details : local host is: "node01.example.com/172.28.21.2"; destination host is: "node01.example.com":8020; 

Namenode logs:

2015-11-25 16:15:34,792 INFO ipc.Server (Server.java:doRead(771)) - Socket Reader #1 for port 8020: readAndProcess from client 172.28.21.2 threw exception [org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled. Available:[TOKEN, KERBEROS]]
org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled. Available:[TOKEN, KERBEROS]
at org.apache.hadoop.ipc.Server$Connection.initializeAuthContext(Server.java:1554)
at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:1510)
at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:762)
at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:636)
at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:607)

Cause

It's due to large clock skew between KDC server and servers in Hadoop cluster.

RCA

Enable HADOOP DEBUG and run Hadoop client command again to get more debugging logs which show error "Clock skew too great".

# HADOOP_ROOT_LOGGER=DEBUG,console hdfs dfs -ls /
......
15/11/26 13:15:54 DEBUG security.UserGroupInformation: hadoop login
15/11/26 13:15:54 DEBUG security.UserGroupInformation: hadoop login commit
15/11/26 13:15:54 DEBUG security.UserGroupInformation: using kerberos user:hdfs@EXAMPLE.COM
15/11/26 13:15:54 DEBUG security.UserGroupInformation: Using user: "hdfs@EXAMPLE.COM" with name hdfs@EXAMPLE.COM
15/11/26 13:15:54 DEBUG security.UserGroupInformation: User entry: "hdfs@EXAMPLE.COM"
15/11/26 13:15:54 DEBUG security.UserGroupInformation: UGI loginUser:hdfs@EXAMPLE.COM (auth:KERBEROS)
......
15/11/26 13:16:11 DEBUG security.UserGroupInformation: PrivilegedActionException as:hdfs@EXAMPLE.COM (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Clock skew too great (37) - PROCESS_TGS)]
15/11/26 13:16:11 DEBUG security.UserGroupInformation: PrivilegedAction as:hdfs@EXAMPLE.COM (auth:KERBEROS) from:org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:644)
15/11/26 13:16:11 WARN ipc.Client: Couldn't setup connection for hdfs@EXAMPLE.COM to node01.example.com/172.28.21.2:8020
15/11/26 13:16:11 DEBUG security.UserGroupInformation: PrivilegedActionException as:hdfs@EXAMPLE.COM (auth:KERBEROS) cause:java.io.IOException: Couldn't setup connection for hdfs@EXAMPLE.COM to node01.example.com/172.28.21.2:8020
15/11/26 13:16:11 DEBUG ipc.Client: closing ipc connection to node01.example.com/172.28.21.2:8020: Couldn't setup connection for hdfs@EXAMPLE.COM to node01.example.com/172.28.21.2:8020
java.io.IOException: Couldn't setup connection for hdfs@EXAMPLE.COM to node01.example.com/172.28.21.2:8020
......
at org.apache.hadoop.fs.FsShell.main(FsShell.java:340)
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Clock skew too great (37) - PROCESS_TGS)]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)

Then it's found that the system time is not synchronized among KDC server and servers of the Hadoop cluster.

System time on KDC server:

# date
Thu Nov 26 05:20:54 GMT+8 2015

System time on Hadoop cluster server:

# date
Thu Nov 26 13:20:41 SGT 2015

Run kinit to get a Kerberos credential successfully, but the valid period of kerberos credential is far ahead of current system time. So the credential cannot be used immediately to access HDFS.

# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: hdfs@SMARTHUB.COM

Valid starting Expires Service principal
11/27/15 06:28:31 11/27/15 14:28:31 krbtgt/SMARTHUB.COM@SMARTHUB.COM

# date
Thu Nov 26 14:29:19 SGT 2015

Resolution

1. Use ntp to synchronize system time of KDC server with servers in Hadoop cluster.

2. Make sure the UTC times on all servers (KDC and Hadoop cluster servers) are synchronized, because this will determine a valid period of Kerberos credentials. In fact, KDC will set a valid period of Kerberos credential based on UTC time and Kerberos client will then adjust it according to its timezone.  

Comments

Powered by Zendesk