Pivotal Knowledge Base

Follow

How to change heap size of Pivotal Hadoop cluster components?

Overview

Most Hadoop components run as Linux processes. These processes create, manage and monitor jobs running in Hadoop cluster. These processes may run out of memory if the default heap size is not sufficient to handle the heavy workloads.

Solution

Please refer to the following table of how to change the default heap size of these components.

Note that you need to restart the corresponding service after changing the heap size of a Hadoop component.

Component       

Configuration file

Changing Heap size(java Xmx) example

Verification

HDFS  NameNode

/etc/gphd/hadoop /conf/hadooenv.sh           

export HADOOP_HEAPSIZE=2048

ps –ef|grep namenode

HDFS DataNode

/etc/gphd/hadoop/conf/hadoop-env.sh

export HADOOP_HEAPSIZE=2048

ps –ef|grep datanode

Yarn Resource Manger

/etc/gphd/hadoop/conf/yarn-env.sh

export YARN_RESOURCEMANAGER_HEAPSIZE=1024

ps –ef|grep resourcemanager

Yarn Node Manager

/etc/gphd/hadoop/conf/yarn-env.sh

export YARN_NODEMANAGER_HEAPSIZE=1024

ps –ef|grep nodemanager

Hive Server ( Hive Server2)

/etc/gphd/hive/conf/hive-env.sh

export HADOOP_CLIENT_OPS="${HADOOP_CLIENT_OPS} -Xmx2048"

ps -ef|grep HiveServer

Hive Meta Store

/etc/gphd/hive/conf/hive-env.sh

export HADOOP_CLIENT_OPS="${HADOOP_CLIENT_OPS} -Xmx2048"

ps -ef|grep HiveMetaStore

HBase Master Server

/etc/gphd/hbase/conf/hbase-env.sh

export HBASE_HEAPSIZE=1000

ps -ef |grep hbase-master

HBase Region Server

/etc/gphd/hbase/conf/hbase-env.sh

export HBASE_HEAPSIZE=1000

ps –ef|grep hbase-regionserver

Zookeeper Server

/etc/gphd/zookeeper/conf/java.env

 export JVMFLAGS="-Xmx2048m"

ps -ef|grep zookeeper

Secondary NameNode

/etc/gphd/hadoop/conf/hadoop-env.sh

 export HADOOP_HEAPSIZE=1000

ps –ef |grep  secondarynamenode

mapreduce-historyserver

/etc/gphd/hadoop/conf/mapred-env.sh

export HADOOP_JOB_HISTORYSERVER_HEAPSIZE=1000          

ps –ef|grep historyserver

Note:

Some of the Hadoop services use the same parameter in the same configuration files to set the heap size. If two daemons are collocated on the same host, changing the parameter will affect both services.

 

Using Hive server2 and hive meta store as an example. Both services uses the parameter export HADOOP_CLIENT_OPS in file /etc/gphd/hive/conf/hive-env.sh. If you just want to  change hive server 2 heap size, you may follow the work around as below:

1. mkdir -p /etc/gphd/hive2/conf
2. cp -rp /etc/gphd/hive/conf/* /etc/gphd/hive2/conf/
3. vi /etc/gphd/hive2/conf/hive-env.sh
export HADOOP_CLIENT_OPS="${HADOOP_CLIENT_OPS} <Your_Heap_Size>"
4. vi /etc/default/hive-server2
export HIVE_CONF_DIR=/etc/gphd/hive2/conf
5. service hive-server2 restart

 

 

 

 
 

Comments

  • Avatar
    Sanjay Kumar Bonthapally

    I think there is a Typo, as OPS instead of OPTS.

Powered by Zendesk