Pivotal Knowledge Base

Follow

Cluster Id is incompatible error reported when starting datanode service

Environment

PHD 1.1.1

Symptom

When starting hadoop-hdfs-datanode service, it may fail with the following error:

org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /data/data1/dfs/data is in an inconsistent state: cluster Id is incompatible with others.
 
Verify 
Check the content of the cluster ID in /data/data1/dfs/data/current/VERSION and compare it with Cluster ID in a different data directory. For example /data/data2/dfs/data/current/VERSION, you may find those the ID are different.
 
Cause
The datanode service failed to start cause it finds the reported data mounting dir that has a different cluster ID than others. The following events may have caused this type of issue:
  • The hadoop cluster was reformatted with new cluster ID and datanode data dir directory was removed or offline
  • A disk was replaced on the datanode but this replacement disk was from a different Hadoop cluster.
Fix
Backup then delete the ${dfs.datanode.data.dir}/current directory in the path then restart the datanode service, the service will be restarted successfully and a new current directory will be created.
 
 

Comments

Powered by Zendesk