Pivotal Knowledge Base

Follow

How to set heap size of hive clients different from other hive services

Environment

Product Version
Pivotal HD, Pivotal HDP  2.x, 3.x
OS  RHEL 6.x
Ambari 1.7
Hive Server  

Purpose

This article describes how to set heap size of Hive client different from other hive services like metastore and Hive server .

Cause

Heap size for all the Hive services (metastore, hiveserver and beeline clients) are same as per the default configuration and which may cause severe performance issues in the cluster. A user should reduce the heap sizes for all the Hive clients and thereby allow more concurrent connections to the cluster.

Procedure

Let's consider a scenario of a cluster having hive heap size of 12 GB. Then all the individual connected hive clients will have 12 GB allocated memory and would limit the number of client connections and that would result into the cluster to run out of memory.

Here is the procedure to set heap size of hive clients different from metastore and Hive server. 

Add the following changes to hive-env.sh template in Ambari and restart hive services. You will see the heap size for all the hive clients got reduced to 1 GB and thereby can have more concurrent Hive client sessions to the cluster .

 if [ "$SERVICE" = "cli" ]; then

if [ -z "$DEBUG" ]; then
export HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=12 -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:+UseNUMA -XX:+UseParallelGC -XX:-UseGCOverheadLimit"
else
export HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=12 -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:-UseGCOverheadLimit"
fi
fi

if [ "$SERVICE" = "metastore" ] || [ "$SERVICE" = "hiveserver2" ]; then
export HADOOP_HEAPSIZE={{hive_heapsize}} # Setting for HiveMetastore & HiveServer2
else
export HADOOP_HEAPSIZE=1024 #Client
fi

export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS -Xmx${HADOOP_HEAPSIZE}m"

 

Comments

Powered by Zendesk