Pivotal Knowledge Base

Follow

YARN Alerts: NodeManager is unhealthy, local-dirs are bad

Environment

Product Version
 Pivotal HD /HDP  3.x / 2.3,2.4
 OS  RHEL 6.x

Symptom

Getting the following alerts in Ambari for a YARN NodeManager:

Error Message:

NodeManager Health Summary
1 NodeManager is unhealthy.
 
1/1 local-dirs are bad: /data/hadoop/yarn/local; 1/1 log-dirs are bad: /data/hadoop/yarn/log
 
Cause

The most likely cause of "local-dirs are bad" errors is the available disk space on the node exceeding YARN's yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage (default value is 90%).

Resolution

Check space utilization on the node and the disk where we are getting the unhealthy node errors and compare it to the value for yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage on Ambari. For example:

[root@nodeN ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda5 9.8G 7.1G 2.3G 76% /
tmpfs 3.8G 0 3.8G 0% /dev/shm
/dev/sda1 477M 77M 375M 17% /boot
/dev/sda7 55G 52G 3G 95% /data
/dev/sda2 20G 45M 19G 1% /home
/dev/sda3 9.8G 24M 9.2G 1% /tmp

If the utilization in the yarn disk (in this case /data) is above the limit set by yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage, try these options:

  • Free up some disk space
  • Increase the value for yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage through Ambari.

Restart NodeManager services to pick up the changes.

 

 

 

 

Comments

Powered by Zendesk