Pivotal Knowledge Base

Follow

How to enable Garbage Collection logging for Hadoop namenode and datanode service

Environment

Product Version
PHD 2.x
JAVA 1.7

How to enable GC logging for Namenode service

  1. Open /etc/gphd/hadoop/conf/hadoop-env.sh and look for the following line
    export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote -Xms2048m -Xmx2048m -Dhadoop.security.logger=INFO,DRFAS -Dhdfs.audit.logger=INFO,RFAAUDIT -XX:ParallelGCThreads=8 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+HeapDumpOnOutOfMemoryError -XX:ErrorFile=${HADOOP_LOG_DIR}/hs_err_pid%p.log $HADOOP_NAMENODE_OPTS"
  2. Insert the following into HADOOP_NAMENODE_OPTS param
    -verbose:gc
    -XX:+PrintGCDetails
    -Xloggc:${HADOOP_LOG_DIR}/hadoop-hdfs-namenode-`date +'%Y%m%d%H%M'`.gclog
    -XX:+UseGCLogFileRotation
    -XX:NumberOfGCLogFiles=20
  3. After adding the GC log params then HADOOP_NAMENODE_OPTS should look like this
    export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote -Xms2048m -Xmx2048m -Dhadoop.security.logger=INFO,DRFAS -Dhdfs.audit.logger=INFO,RFAAUDIT -XX:ParallelGCThreads=8 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+HeapDumpOnOutOfMemoryError -XX:ErrorFile=${HADOOP_LOG_DIR}/hs_err_pid%p.log -verbose:gc -XX:+PrintGCDetails -Xloggc:${HADOOP_LOG_DIR}/hadoop-hdfs-namenode-`date +'%Y%m%d%H%M'`.gclog -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=20 $HADOOP_NAMENODE_OPTS"
  4. service hadoop-hdfs-namenode restart

How to enable GC logging for Datanode service

  1. Open /etc/gphd/hadoop/conf/hadoop-env.sh and look for the following line
    export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote -Xms2048m -Xmx2048m -Dhadoop.security.logger=ERROR,DRFAS $HADOOP_DATANODE_OPTS"
  2. Insert the following into HADOOP_DATANODE_OPTS param
    -verbose:gc
    -XX:+PrintGCDetails
    -Xloggc:${HADOOP_LOG_DIR}/hadoop-hdfs-datanode-`date +'%Y%m%d%H%M'`.gclog
    -XX:+UseGCLogFileRotation
    -XX:NumberOfGCLogFiles=20
  3. After adding the GC log pram the HADOOP_DATANODE_OPTS should look like this
    export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote -Xms2048m -Xmx2048m -Dhadoop.security.logger=ERROR,DRFAS -verbose:gc -XX:+PrintGCDetails -Xloggc:${HADOOP_LOG_DIR}/hadoop-hdfs-datanode-`date +'%Y%m%d%H%M'`.gclog -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=20 $HADOOP_DATANODE_OPTS"

 

Comments

Powered by Zendesk