Pivotal Knowledge Base

Follow

PHD upgrade to 2.0.1 namenode upgrade fails with InconsistentFSStateException

Environment

PHD Upgrade from 1.1.1 to 2.0.1

Symptom

STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = hdm1.phd.local/172.28.16.122
STARTUP_MSG:   args = [-upgrade]
STARTUP_MSG:   version = 2.2.0-gphd-3.0.1.0
.
.
2014-05-20 10:33:49,292 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /data/nn/dfs/name is in an inconsistent state: previous fs state should not exist during upgrade. Finalize or rollback first.
        at org.apache.hadoop.hdfs.server.namenode.FSImage.doUpgrade(FSImage.java:326)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:251)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:787)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:568)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:443)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:491)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:684)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:669)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1254)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1320)
2014-05-20 10:33:49,294 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2014-05-20 10:33:49,302 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at hdm1.phd.local/172.28.16.122
************************************************************/

Cause

In this scenario the user initially runs the namenode upgrade and the upgrade was successful but starting the namenode failed.

[gpadmin@pccadmin CONFIGS]$ /usr/bin/python /usr/lib/gphd/gphdmgr/lib/client/HdfsUpgrader.py -l phd -o nnupgrade -s 2.0.5_alpha_gphd_2_1_1_0 -t 2.2.0_gphd_3_0_0_0
Running command massh /tmp/tmp.4bKwDRVCg4 verbose /usr/bin/python /usr/lib/gphd/gphdmgr/util/hdfs_upgrade_cli.py -o nnupgrade -s 2.0.5_alpha_gphd_2_1_1_0 -t 2.2.0_gphd_3_0_0_0
['hdm1.phd.local : Upgrading namenode meta data...\n',
 'hdm1.phd.local : Error\t(code:6001): Failed to start the namenode data upgrade. The upgrade might have completed, but the namenode failed to start due to other reasons. Check the hadoop namenode log for details.\n']

The user then proceeds to correct start issue and re-run the namenode upgrade.   All subsequent execution of HdfsUpgrader.py will fail because the namenode was already successfully upgrade. 

Fix

In this case simply start the namenode manually and then continue with the HDFS verification steps as per the PHD upgrade documentation. 

Comments

Powered by Zendesk