Pivotal Knowledge Base

Follow

The Secure HDFS Error "No valid credentials provided" Displays when Running HDFS, DFS or Hadoop FS

Environment

Product Version
PHD 1.x and 2.x

Symptom

ERROR security.UserGroupInformation: PriviledgedActionException as:gpadmin (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
14/05/15 11:41:55 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

Cause

Typically users this error occurs after successfully runnning the kinit command and have a valid TGT issued by the KDC in their respective Kerberos realm.  The error returned by the HDFS client is generic and could be a result of the following issues

  • The user set the  $KRB5_CONFIG environment variable to something other than the default value of /etc/krb5.conf.  The HDFS client will not source the $KRB5_CONFIG from the user's shell.
  • /etc/gphd/hadoop/conf/hdfs-site.xml does not have the correct Kerberos configuration for the namenode HDFS principal.
  • The DNS does not resolve the correct Fully Qualified Domain Name.
  • The Kerberos client libs version do not match the server.
  • Kerberos encryption defaults differ between the client and the KDC.

Refer to the following troubleshooting techniques

The error "No valid credentials provided" is the default error string returned by Hadoop fs command when Kerberos authentication fails.  To better understand which step in the Kerberos handshake failed, it is best to enable debugging. 

  1. vi /etc/gphd/hadoop/conf/hadoop-env.sh
  2. add param sun.security.krb5.debug=true to HADOOP_OPTS variable
    export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true -Dsun.security.krb5.debug=true ${HADOOP_OPTS}”

After enabling debugging, targeting the issue will become easier.

ERROR code table of contents

 1. extra data given to DerValue constructor 
 2. Cross Realm TGS Request no tgt
 3. No valid credentials provided TGT: 18 

1. Extra data is given to DerValue constructor.

[gpadmin@hdm1-prod ~]$ hdfs dfs -ls /
Config name: /etc/krb5.conf
>>>KinitOptions cache name is /tmp/krb5cc_500
>>>DEBUG   client principal is gpadmin@PROD.PIVOTAL.HADOOP
>>>DEBUG  server principal is krbtgt/PROD.PIVOTAL.HADOOP@PROD.PIVOTAL.HADOOP
>>>DEBUG  key type: 16
>>>DEBUG  auth time: Thu May 15 11:47:45 PDT 2014
>>>DEBUG  start time: Thu May 15 11:47:45 PDT 2014
>>>DEBUG  end time: Fri May 16 11:47:45 PDT 2014
>>>DEBUG  renew_till time: Thu May 22 11:47:45 PDT 2014
>>> CCacheInputStream: readFlags()  FORWARDABLE; RENEWABLE; INITIAL;
>>>DEBUG 
>>>DEBUG   client principal is gpadmin@PROD.PIVOTAL.HADOOP
>>>DEBUG  server principal is X-CACHECONF:/krb5_ccache_conf_data/fast_avail/krbtgt/PROD.PIVOTAL.HADOOP@PROD.PIVOTAL.HADOOP
>>>DEBUG  key type: 0
>>>DEBUG  auth time: Wed Dec 31 16:00:00 PST 1969
>>>DEBUG  start time: Wed Dec 31 16:00:00 PST 1969
>>>DEBUG  end time: Wed Dec 31 16:00:00 PST 1969
>>>DEBUG  renew_till time: Wed Dec 31 16:00:00 PST 1969
>>> CCacheInputStream: readFlags()
java.io.IOException: extra data given to DerValue constructor
	at sun.security.util.DerValue.init(DerValue.java:368)
	at sun.security.util.DerValue.(DerValue.java:277)
	at sun.security.krb5.internal.Ticket.(Ticket.java:81)
	at sun.security.krb5.internal.ccache.CCacheInputStream.readData(CCacheInputStream.java:250)
	at sun.security.krb5.internal.ccache.CCacheInputStream.readCred(CCacheInputStream.java:357)
	at sun.security.krb5.internal.ccache.FileCredentialsCache.load(FileCredentialsCache.java:225)
	at sun.security.krb5.internal.ccache.FileCredentialsCache.acquireInstance(FileCredentialsCache.java:104)
	at sun.security.krb5.internal.ccache.CredentialsCache.getInstance(CredentialsCache.java:75)
	at sun.security.krb5.Credentials.acquireTGTFromCache(Credentials.java:304)
	at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:589)
	at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:542)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at javax.security.auth.login.LoginContext.invoke(LoginContext.java:769)
	at javax.security.auth.login.LoginContext.access$000(LoginContext.java:186)
	at javax.security.auth.login.LoginContext$5.run(LoginContext.java:706)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.login.LoginContext.invokeCreatorPriv(LoginContext.java:703)
	at javax.security.auth.login.LoginContext.login(LoginContext.java:575)
	at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:669)
	at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:571)
	at org.apache.hadoop.fs.FileSystem$Cache$Key.(FileSystem.java:2433)
	at org.apache.hadoop.fs.FileSystem$Cache$Key.(FileSystem.java:2425)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2292)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:317)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:163)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:301)
	at org.apache.hadoop.fs.Path.getFileSystem(Path.java:194)
	at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:270)
	at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:224)
	at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:207)
	at org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:190)
	at org.apache.hadoop.fs.shell.Command.run(Command.java:154)
	at org.apache.hadoop.fs.FsShell.run(FsShell.java:255)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
	at org.apache.hadoop.fs.FsShell.main(FsShell.java:305)

Resolution

A similar issue was reported in apache the Jira HTTPCLIENT-934 and is related to a JDK 6 compatibility with KRB 1.8+.  Running "kinit -R" to rewrite the users credential cache should fix this problem.   After implementing the workaround and verifying the fix then you should upgrade to JDK 7 to prevent future occurrences. 

2. Cross Realm TGS Request no TGT.

>>> Credentials acquireServiceCreds: main loop: [0] tempService=krbtgt/PROD.PIVOTAL.HADOOP@DEV.PIVOTAL.HADOOP
default etypes for default_tgs_enctypes: 16 23 1 3.
>>> CksumType: sun.security.krb5.internal.crypto.RsaMd5CksumType
>>> EType: sun.security.krb5.internal.crypto.Des3CbcHmacSha1KdEType
>>> KrbKdcReq send: kdc=pccadmin-dev.phd.local TCP:88, timeout=30000, number of retries =3, #bytes=727
>>>DEBUG: TCPClient reading 621 bytes
>>> KrbKdcReq send: #bytes read=621
>>> KrbKdcReq send: #bytes read=621
>>> KdcAccessibility: remove pccadmin-dev.phd.local:88
>>> EType: sun.security.krb5.internal.crypto.Des3CbcHmacSha1KdEType
>>> Credentials acquireServiceCreds: no tgt; searching backwards
>>> Credentials acquireServiceCreds: inner loop: [1] tempService=krbtgt/PIVOTAL.HADOOP@DEV.PIVOTAL.HADOOP
default etypes for default_tgs_enctypes: 16 23 1 3.
>>> CksumType: sun.security.krb5.internal.crypto.RsaMd5CksumType
>>> EType: sun.security.krb5.internal.crypto.Des3CbcHmacSha1KdEType
>>> KrbKdcReq send: kdc=pccadmin-dev.phd.local TCP:88, timeout=30000, number of retries =3, #bytes=722
>>>DEBUG: TCPClient reading 187 bytes
>>> KrbKdcReq send: #bytes read=187
>>> KrbKdcReq send: #bytes read=187
>>> KdcAccessibility: remove pccadmin-dev.phd.local:88
>>> KDCRep: init() encoding tag is 126 req type is 13
>>>KRBError:
	 cTime is Thu May 15 11:59:58 PDT 2014 1400180398000
	 sTime is Thu May 15 11:59:53 PDT 2014 1400180393000
	 suSec is 860526
	 error code is 7
	 error Message is Server not found in Kerberos database
	 crealm is DEV.PIVOTAL.HADOOP
	 cname is gpadmin
	 realm is DEV.PIVOTAL.HADOOP
	 sname is krbtgt/PIVOTAL.HADOOP
	 msgType is 30
>>> Credentials acquireServiceCreds: no tgt; cannot get creds

Resolution

This is related to a cross-realm trust between PROD.PIVOTAL.HADOOP and DEV.PIVOTAL.HADOOP.  The user is attempting to run Hadoop commands fs -ls from DEV cluster to PROD cluster and the TGS request is failing to find a valid TGT for krbtgt/PROD.PIVOTAL.HADOOP@DEV.PIVOTAL.HADOOP.  The important error here is occurs after the first attempt to get a TGT and it fails with error "no tgt; searching backwards".  After this, it attempts to find TGT. krbtgt/PIVOTAL.HADOOP@DEV.PIVOTAL.HADOOP which does not exist.  

The reason the initial TGS request fails is because JDK 6 has a known issue referenced by bug id 7061379.  Basically, in a later, version of krb5 server they change the response data type for a TGS request and JDK 6 is expecting a different data type in the response.  The solution is to upgrade to JDK 7 on all PHD nodes. 

3. No valid credentials provided TGT: 18

[gpadmin@etl1 ~]$ hdfs dfs -ls /
Java config name: null
Native config name: /etc/krb5.conf
Loaded from native config
>>>KinitOptions cache name is /tmp/krb5cc_500
>>>DEBUG   client principal is gpadmin/etl1.phd.local@PHD.LOCAL
>>>DEBUG  server principal is krbtgt/PHD.LOCAL@PHD.LOCAL
>>>DEBUG  key type: 18
>>>DEBUG  auth time: Mon Aug 18 11:09:40 PDT 2014
>>>DEBUG  start time: Mon Aug 18 11:09:40 PDT 2014
>>>DEBUG  end time: Tue Aug 19 11:09:40 PDT 2014
>>>DEBUG  renew_till time: Mon Aug 18 11:09:40 PDT 2014
>>> CCacheInputStream: readFlags()  FORWARDABLE; RENEWABLE; INITIAL;
>>>DEBUG   client principal is gpadmin/etl1.phd.local@PHD.LOCAL
>>>DEBUG  server principal is X-CACHECONF:/krb5_ccache_conf_data/fast_avail/krbtgt/PHD.LOCAL@PHD.LOCAL
>>>DEBUG  key type: 0
>>>DEBUG  auth time: Wed Dec 31 16:00:00 PST 1969
>>>DEBUG  start time: null
>>>DEBUG  end time: Wed Dec 31 16:00:00 PST 1969
>>>DEBUG  renew_till time: null
>>> CCacheInputStream: readFlags()
>>>DEBUG   client principal is gpadmin/etl1.phd.local@PHD.LOCAL
>>>DEBUG  server principal is /HTTP/hdw1.phd.local
>>>DEBUG  key type: 18
>>>DEBUG  auth time: Mon Aug 18 11:09:40 PDT 2014
>>>DEBUG  start time: Mon Aug 18 11:23:14 PDT 2014
>>>DEBUG  end time: Tue Aug 19 11:09:40 PDT 2014
>>>DEBUG  renew_till time: Mon Aug 18 11:09:40 PDT 2014
>>> CCacheInputStream: readFlags()  FORWARDABLE; RENEWABLE;
>>>DEBUG   client principal is gpadmin/etl1.phd.local@PHD.LOCAL
>>>DEBUG  server principal is HTTP/hdw1.phd.local@PHD.LOCAL
>>>DEBUG  key type: 18
>>>DEBUG  auth time: Mon Aug 18 11:09:40 PDT 2014
>>>DEBUG  start time: Mon Aug 18 11:23:14 PDT 2014
>>>DEBUG  end time: Tue Aug 19 11:09:40 PDT 2014
>>>DEBUG  renew_till time: Mon Aug 18 11:09:40 PDT 2014
>>> CCacheInputStream: readFlags()  FORWARDABLE; RENEWABLE;
>>>DEBUG   client principal is gpadmin/etl1.phd.local@PHD.LOCAL
>>>DEBUG  server principal is /HTTP/hdm1.phd.local
>>>DEBUG  key type: 18
>>>DEBUG  auth time: Mon Aug 18 11:09:40 PDT 2014
>>>DEBUG  start time: Mon Aug 18 11:23:28 PDT 2014
>>>DEBUG  end time: Tue Aug 19 11:09:40 PDT 2014
>>>DEBUG  renew_till time: Mon Aug 18 11:09:40 PDT 2014
>>> CCacheInputStream: readFlags()  FORWARDABLE; RENEWABLE;
>>>DEBUG   client principal is gpadmin/etl1.phd.local@PHD.LOCAL
>>>DEBUG  server principal is HTTP/hdm1.phd.local@PHD.LOCAL
>>>DEBUG  key type: 18
>>>DEBUG  auth time: Mon Aug 18 11:09:40 PDT 2014
>>>DEBUG  start time: Mon Aug 18 11:23:28 PDT 2014
>>>DEBUG  end time: Tue Aug 19 11:09:40 PDT 2014
>>>DEBUG  renew_till time: Mon Aug 18 11:09:40 PDT 2014
>>> CCacheInputStream: readFlags()  FORWARDABLE; RENEWABLE;
>>> unsupported key type found the default TGT: 18 

Resolution

This error occurs when you have AES256 encryption enabled and you recently upgraded Java.  Upgrading Java will overwrite the JCE policy files which include support for AES256 encryption.  To fix this simply re-install your JCE policy jars back into "/usr/java/default/jre/lib/security/".

Comments

  • Avatar
    Sangdon Shin

    Thank you so much for the great article, this allowed me to solve one customer issue in five mins.

Powered by Zendesk