Pivotal Knowledge Base

Follow

Secondary namenode fails to get fsimage from namenode in kerberized enviroment

Environment

  • PHD 2.x

Symptom

ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Exception in doCheckpoint
org.apache.hadoop.hdfs.server.namenode.TransferFsImage$HttpGetFailedException: Image transfer servlet at http://hdm.xxx.com:50070/getimage?getimage=1&txid=8464&storageInfo=-40:1006297165:0:CID-ff2db014-06e6-4c70-a33f-bfdd69063735 failed with status code 403
Response message:
Only Namenode, Secondary Namenode, and administrators may access this servlet
 at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.doGetUrl(TransferFsImage.java:245)
 at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:222)
 at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.downloadImageToStorage(TransferFsImage.java:86)
 at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:430)
 at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:416)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:415)
 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
 at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.downloadCheckpointFiles(SecondaryNameNode.java:415)
 at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:515)
 at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:367)
 at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$2.run(SecondaryNameNode.java:333)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:356)
 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1458)
 at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:454)
 at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:329)
 at java.lang.Thread.run(Thread.java:745)

Namenode shows error

INFO org.apache.hadoop.hdfs.server.namenode.GetImageServlet: GetImageServlet rejecting: hdfs/hdw3.xxx.com@VIADEA.COM
WARN org.apache.hadoop.hdfs.server.namenode.GetImageServlet: Received non-NN/SNN/administrator request for image or edits from hdfs/hdw3.xxx.com@VIADEA.COM at 192.168.192.104

Cause

Namenode is not aware of the kerberos authentication information for secondary namenode.

By checking hdfs-site.xml on namenode, and found below entries are missing.

<!-- (optional) secondary name node secure configuration info -->
<property>
 <name>dfs.secondary.namenode.keytab.file</name>
 <value>/etc/security/phd/keytab/hdfs.service.keytab</value>
</property>

<property>
 <name>dfs.secondary.namenode.kerberos.principal</name>
 <value>hdfs/_HOST@VIADEA.COM</value>
</property>

<property>
 <name>dfs.secondary.namenode.kerberos.http.principal</name>
 <value>HTTP/_HOST@VIADEA.COM</value>
</property>

<property>
 <name>dfs.secondary.namenode.kerberos.internal.spnego.principal</name>
 <value>HTTP/_HOST@VIADEA.COM</value>
</property>

Fix

After adding above entries back in hdfs-site.xml on namenode, restart HDFS cluster.

Below log entries are shown on secondary namenode:

INFO org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Opening connection to http://hdm.xxx.com:50070/getimage?putimage=1&txid=11104&port=50090&storageInfo=-40:1006297165:0:CID-ff2db014-06e6-4c70-a33f-bfdd69063735
INFO org.apache.hadoop.hdfs.server.namenode.GetImageServlet: GetImageServlet allowing checkpointer: hdfs/hdm.xxx.com@VIADEA.COM
INFO org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Transfer took 0.42s at 0.00 KB/s
INFO org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Uploaded image with txid 11104 to namenode at hdm.xxx.com:50070

Comments

Powered by Zendesk