Pivotal Knowledge Base

Follow

Space full caused by log_alert_stage table

Environment

Product Version
Pivotal Greenplum (GPDB) 4.3.x
OS RHEL 6.x
Others  

Symptom

Data directory space usage at Master & certain segment host reach to near 100%.

Cause 
GP command center will check pg log at master for any error messages and save error message to  $MASTER_DATA_DIRECTORY/gpperfmon/log/gpdb-alert-yyyymmdd.csv file. If some error messages included the special character or unformatted messages, these message in gpdb-alert file will not be inserted into the log_alert_history table, they will stay in log_alert_stage table forever. 

The current design is once messages have been inserted into log_alert_history table, then gpcc would truncate log_alert_stage table. In this situation, more and more error message will be saved in log_alert_stage table and use more and more space, until it's 100% full. 

Resolution
1. Truncate log_alert_stage table.
2. Review gpdb-alert*.csv files at  $MASTER_DATA_DIRECTORY/gpperfmon/log/ directory. Check if there are any special message saved.Then follow up with the customer to check.

Comments

Powered by Zendesk