Pivotal Knowledge Base

Follow

How to deal with "100% Filesystem Full" Situations

Environment

  • Pivotal Greenplum Database (GPDB) 4.2.x and 4.3.x
  • Pivotal HD (PHD) 1.1.1, 2.0.1
  • Data Computing Appliance (DCA) 1 and 2

Symptom

The system is reporting 100% full on a certain filesystem.
If this happens on the root filesystem, it can cause a real DU situation.

Cause

There are various big files taking up most of the filesystem space.

  • PHD Log files under /var/log/gphd/*/*.log
  • User-generated dump or temporary files scattered around the filesystem

Resolution

Use the command to determine which files are consuming the most space. Based on the output of this command you can decide which files should be deleted or retained.
If this reports too many files you narrow the scope by increasing the size from the +50000 to even higher values.

# find <mount_point> -mount -size +50000

# find / -mount -size +50000 <<< this will list all files of the root FS bigger than 25MB in size/usr/java/jdk1.6.0_34/jre/lib/rt.jar
/usr/lib/locale/locale-archive
/usr/local/GP-4.2.6.0/bin/postgres
/usr/local/greenplum-db-4.2.3.2/bin/postgres
/var/lib/rpm/Packages
/var/lib/zookeeper/version-2/log.1400000001
/var/lib/zookeeper/version-2/log.140000c3a5
/var/lib/zookeeper/version-2/log.d00000001
/var/lib/zookeeper/version-2/log.f00000001
/var/lib/zookeeper/version-2/log.1300000001
/var/lib/zookeeper/version-2/log.c00000001
/var/lib/zookeeper/version-2/log.200000001
/var/lib/zookeeper/version-2/log.700000001
/var/lib/zookeeper/version-2/log.b00000001
/var/lib/zookeeper/version-2/log.100000001
/var/lib/zookeeper/version-2/log.1200000001
/var/log/gphd/hadoop-yarn-2.0.2_alpha_gphd_2_0_1_0/yarn-yarn-resourcemanager-hdm3.gphd.local.log.2 <<
/var/log/gphd/hadoop-yarn-2.0.2_alpha_gphd_2_0_1_0/yarn-yarn-resourcemanager-hdm3.gphd.local.log.4 <<
/var/log/gphd/hadoop-yarn-2.0.2_alpha_gphd_2_0_1_0/yarn-yarn-resourcemanager-hdm3.gphd.local.log.3 <<
/var/log/gphd/hadoop-yarn-2.0.2_alpha_gphd_2_0_1_0/yarn-yarn-resourcemanager-hdm3.gphd.local.log.4 <<

Comments

Powered by Zendesk