Pivotal Knowledge Base

Follow

Space Full Caused by log_alert_stage Table

Environment

  • Pivotal Greenplum Database (GPDB) 4.3.x
  • Operating System- Red Hat Enterprise Linux 6.x

Symptom

Data directory space usage at Master & certain segment host reach to near 100%.

Cause

GP command center will check pg log at master for any error messages and save error message to $MASTER_DATA_DIRECTORY/gpperfmon/log/gpdb-alert-yyyymmdd.csv file. If some error messages included the special character or unformatted messages, these message in gpdb-alert file will not be inserted into the log_alert_history table, they will stay in log_alert_stage table forever.

The current design is once messages have been inserted into log_alert_history table, then gpcc would truncate log_alert_stage table. In this situation, more and more error message will be saved in log_alert_stage table and use more and more space, until it's 100% full. 

Resolution

1. Truncate log_alert_stage table.
2. Review gpdb-alert*.csv files at  $MASTER_DATA_DIRECTORY/gpperfmon/log/ directory. Check if there are any special message saved.Then follow up with the customer to check.

Comments

Powered by Zendesk