Pivotal Knowledge Base

Follow

PHD Upgrade or reconfigure in 1.1.0 or 1.1.1 may hang indefinitely when upgrading HDFS block permissions

Environment

  • PHD 1.1.1 / PHD 1.1.0

Symptom

If hdfs contains more then 400GB of data when performing icm_client self-upgrade ( page 57 ) or reconfigure ( page 60 ) puppet catalog sync may hang indefinately.  

Cause

Pivotal Bug number: HD-8381

During puppet catalog sync the upgrade or reconfigure will attempt to modify all file permission on HDFS blocks if it identifies that the permission is not set to 770. In case, when there is high volume of data resulting in large number of dataset on any given datanode (around 400GB or more) then the ruby processes would start changing the block permission and it could take long. However, the reconfigure / upgrade process monitor from admin node would consider the sync to take too long and will assume that the process is hung and will abort the process, resulting in reconfigure / upgrade failure.

FIX

The fix for this will in the Feburary 2014 release, but engineering has released a patched version of PCC in PHD 1.1.1 without changing the actual version numbers of the underlying files.  

Upgrading to PHD 1.1.1 :  

Here are the steps to ensure you have installed the patch and are safe to perform the upgrade or reconfigure operations. 

File: /etc/puppet/modules/hadoop_2_0/manifests/mapred.pp 
Dec PHD 1.1.1 release MD5: 1819f0729883aa9169c9b7a521361c00
PATCH for PHD 1.1.1   MD5: de497a541180098e24c8a9d252f5421f

PLR
Dec PHD 1.1.1 release: plr-1.1.4.0-5152.x86_64.tgz
PATCH for PHD 1.1.1  : plr-1.1.4.0-5664.x86_64.tgz

Upgrading to PHD 1.1.0 :  

In case the customer wants to upgrade to PHD 1.1.0, a patch is not available as of now. In order to mitigate this, a change has to be made in puppet manifest files manually after installing PCC supplied with PHD 1.1.0. There are 2 location where these changes has to be made, just make change at 1 location & then copy them to the other.

- /etc/puppet/modules/
- /usr/lib/gphd/gphdmgr/lib/server/puppet/modules/

Here is the change:

Append the below value after recurse => true, 

 recurselimit => 0,

Note: A comma at the end is ok.

Just for reference the list of files to be modified files.

/usr/lib/gphd/gphdmgr/lib/server/puppet/modules/hadoop_2_0/manifests/journalnode.pp
/usr/lib/gphd/gphdmgr/lib/server/puppet/modules/hadoop_2_0/manifests/secondarynamenode.pp
/usr/lib/gphd/gphdmgr/lib/server/puppet/modules/hadoop_2_0/manifests/base.pp
/usr/lib/gphd/gphdmgr/lib/server/puppet/modules/hadoop_2_0/manifests/mapred.pp
/usr/lib/gphd/gphdmgr/lib/server/puppet/modules/hadoop_2_0/manifests/datanode.pp
/usr/lib/gphd/gphdmgr/lib/server/puppet/modules/hadoop_2_0/manifests/namenode.pp
/usr/lib/gphd/gphdmgr/lib/server/puppet/modules/hadoop_2_0/manifests/yarn_nodemanager.pp
/usr/lib/gphd/gphdmgr/lib/server/puppet/modules/zookeeper_2_0/manifests/base.pp
/usr/lib/gphd/gphdmgr/lib/server/puppet/modules/gpxf_2_0/manifests/base.pp
/usr/lib/gphd/gphdmgr/lib/server/puppet/modules/hawq_2_0/manifests/hawq_standbymaster.pp
/usr/lib/gphd/gphdmgr/lib/server/puppet/modules/hawq_2_0/manifests/base.pp
/usr/lib/gphd/gphdmgr/lib/server/puppet/modules/hawq_2_0/manifests/hawq_segment.pp
/usr/lib/gphd/gphdmgr/lib/server/puppet/modules/hawq_2_0/manifests/hawq_master.pp

Modify the above and replace the below with them.

/etc/puppet/modules/gpxf_2_0/manifests/base.pp
/etc/puppet/modules/hadoop_2_0/manifests/journalnode.pp
/etc/puppet/modules/hadoop_2_0/manifests/secondarynamenode.pp
/etc/puppet/modules/hadoop_2_0/manifests/base.pp
/etc/puppet/modules/hadoop_2_0/manifests/mapred.pp
/etc/puppet/modules/hadoop_2_0/manifests/datanode.pp
/etc/puppet/modules/hadoop_2_0/manifests/namenode.pp
/etc/puppet/modules/hadoop_2_0/manifests/yarn_nodemanager.pp
/etc/puppet/modules/hawq_2_0/manifests/hawq_standbymaster.pp
/etc/puppet/modules/hawq_2_0/manifests/base.pp
/etc/puppet/modules/hawq_2_0/manifests/hawq_segment.pp
/etc/puppet/modules/hawq_2_0/manifests/hawq_master.pp
/etc/puppet/modules/zookeeper_2_0/manifests/base.pp 

Comments

Powered by Zendesk