Pivotal Knowledge Base

Follow

How to deal with 100% filesystem full situations

Environment

Product Version
 Pivotal Greenplum (GPDB)  4.2.x and 4.3.x
 Pivotal HD (PHD)  1.1.1 and 2.0.1
 DCA  v1 and v2 

Symptom

The system is reporting 100% full on certain filesystem.
If this happens on the root filesystem, it can cause a real DU situation.

Cause

There are various big files taking up most of filesystem space.

  • PHD Log files under /var/log/gphd/*/*.log
  • User generated dump or temporary files scattered around the filesystem

Resolution

Use the command to determine which files are consuming the most space. Based on the output of this command you can decide which files should be deleted or retained.
If this reports too many files you narrow the scope by increasing the size from the +50000 to even higher values.

# find <mount_point> -mount -size +50000

# find / -mount -size +50000 <<< this will list all files of the root FS bigger than 25MB in size
/usr/java/jdk1.6.0_34/jre/lib/rt.jar
/usr/lib/locale/locale-archive
/usr/local/GP-4.2.6.0/bin/postgres
/usr/local/greenplum-db-4.2.3.2/bin/postgres
/var/lib/rpm/Packages
/var/lib/zookeeper/version-2/log.1400000001
/var/lib/zookeeper/version-2/log.140000c3a5
/var/lib/zookeeper/version-2/log.d00000001
/var/lib/zookeeper/version-2/log.f00000001
/var/lib/zookeeper/version-2/log.1300000001
/var/lib/zookeeper/version-2/log.c00000001
/var/lib/zookeeper/version-2/log.200000001
/var/lib/zookeeper/version-2/log.700000001
/var/lib/zookeeper/version-2/log.b00000001
/var/lib/zookeeper/version-2/log.100000001
/var/lib/zookeeper/version-2/log.1200000001
/var/log/gphd/hadoop-yarn-2.0.2_alpha_gphd_2_0_1_0/yarn-yarn-resourcemanager-hdm3.gphd.local.log.2 <<
/var/log/gphd/hadoop-yarn-2.0.2_alpha_gphd_2_0_1_0/yarn-yarn-resourcemanager-hdm3.gphd.local.log.4 <<
/var/log/gphd/hadoop-yarn-2.0.2_alpha_gphd_2_0_1_0/yarn-yarn-resourcemanager-hdm3.gphd.local.log.3 <<
/var/log/gphd/hadoop-yarn-2.0.2_alpha_gphd_2_0_1_0/yarn-yarn-resourcemanager-hdm3.gphd.local.log.4 <<
:
:

Comments

Powered by Zendesk