Pivotal Knowledge Base

Follow

HBase fails to start with an error "NoAuth for /hbase/unassigned/1028785192"

Environment:

  • PHD 1.x
  • HBase 0.94.x

Symptom:

HBase master may fail to start due to ACL issues and can report an error "NoAuth for /hbase/unassigned/nnnnnnn", where n represents an integer.

2014-03-18 10:27:11,678 DEBUG org.apache.hadoop.hbase.catalog.CatalogTracker: Starting catalog tracker org.apache.hadoop.hbase.catalog.
CatalogTracker@629ca8b8
2014-03-18 10:27:11,684 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil: master:60000-0x144d63702df0003 Unable to get data of znode /hbase/unassigned/1028785192
org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /hbase/unassigned/1028785192
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:113)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1151)
        ...
        at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:413)
        at java.lang.Thread.run(Thread.java:744)  ... ... 2014-03-18 10:27:11,747 ERROR org.apache.hadoop.hbase.master.HMasterCommandLine: Failed to start master
java.lang.RuntimeException: HMaster Aborted

Background:

/hbase/unassigned znode contains a sub-znode for each unassigned region e.g. /hbase/unassigned/<region name>, This znode is used by the Assignment Manager to discover the regions to assign. If the auth scheme & user information is not in sync with that of the user accessing the node, zookeeper reports an authentication error. 

[gpadmin@hdm1] hbase zkcli
[zk: localhost:2181(CONNECTED) 3] getAcl /hbase/unassigned
'world,'anyone
: cdrwa
[zk: localhost:2181(CONNECTED) 4] ls /hbase/unassigned
[1028785192, 70236052]
[zk: localhost:2181(CONNECTED) 5] getAcl /hbase/unassigned/1028785192
'sasl,'hbase
: cdrwa

In the above ACL outputs, sub-znodes under /hbase/unassigned has got authorization scheme as sasl for hbase user.

Generally, you may see such errors when the cluster was not unsecured properly. When the cluster is secured, zookeeper uses SASL authentication and SASL will be the authorization scheme to access the znode data. However, if the authorization scheme is not set to "world:anyone:cdrwa" and the cluster is unsecured, the nodes will still have SASL authorization scheme, but since SASL is not used on secured hbase, "No Auth" or "Authentication is not valid" error will be seen.

Workaround:

Case 1: Delete all the sub-znodes under /hbase/unassigned and restart hbase master server. {You can delete unassigned nodes, since they don't contain any data.

[gpadmin@hdm1] hbase zkcli
[zk: localhost:2181(CONNECTED) 3]delete /hbase/unassigned/1028785192
[zk: localhost:2181(CONNECTED) 3]delete /hbase/unassigned/70236052
[zk: localhost:2181(CONNECTED) 3] quit 
[gpadmin@hdm1] sudo service hbase-master start
[OK]

Case 2: In case you are not able to delete or access a node after unsecuring the cluster, you will have to resecure the cluster, perform a kinit and change the authorization scheme for the required znodes or sub nodes to "world:anyone:cdrwa". After changing the authorization scheme you can unsecure the cluster.

a. Secure the cluster

b. kinit

c. hbase zkcli

d. getAcl /hbase/<znode>

e. setAcl /hbase/<znode> world:anyone:cdrwa

f. Unsecure the cluster

Miscelleneous:

Apache JIRA: HBASE-7558

Comments

Powered by Zendesk