Pivotal Knowledge Base

Follow

Transaction ID errors when bootstraping standby namenode

PHD all version


SYMPTOM

Fatal error when running command "hdfs namenode -bootstrapStandby"

13/12/18 20:29:04 FATAL ha.BootstrapStandby: Unable to read transaction ids
12345-12346 from the configured shared edits storage
qjournal://hdw1:8485;hdw2:8485;hdw3:8485/phdcluster1.
Please copy these logs into the shared edits storage or call saveNamespace
on the active node.
Error: Gap in transactions. Expected to be able to read up until at least
txid 12346 but unable to find any edit logs containing txid 12346
13/12/18 20:29:04 INFO util.ExitUtil: Exiting with status 6



CAUSE
This is a known issue reported in apache jira HDFS-3752.  


FIX
1.  Put the active namenode into safemode
sudo -u hdfs hdfs dfsadmin -safemode enter

2. rsync the data from primary to the standby. Example command below is run from standby.  Becarefull not to rsyc the inverse overwriting data on the primary.  
rsync –r hdfs@primary_NN:/data/nn/current /data/nn/current

3.  start the stanby namenode
service hadoop-hdfs-namenode start

4.  check the current status of both namenodes
sudo -u hdfs hdfs haadmin -getServiceState << primary namnode id>>
sudo -u hdfs hdfs haadmin -getServiceState << standby namnode id>>





Comments

Powered by Zendesk