Pivotal Knowledge Base

Follow

Missing Data for Column "Loguser" from GPPERFMON

Environment

 Product  Version
 Pivotal Greenplum  4.3.4.2
 OS  RHEL 6.x

Symptom

gpperfmon is logging errors processing log alerts:

2016-11-16 08:59:04.215512 CET|gpmon|gpperfmon|p2050|th-663976160|[local]||2016-11-16 08:59:04 CET|26658479|con133196|cmd2|seg
-1|slice1|dx20101022|x26658479|sx1|ERROR: |22P04|missing data for column "loguser"|||||External table log_alert_tail, line 1469
of execute:gpperfmoncat.sh gpperfmon/logs/alert_log_stage 2> /dev/null || true: " pkt_loss_term"|insert into log_alert_history select
* from log_alert_tail;|0||copy.c|5380|

Cause

Upon further analysis, we see the actual full error code:

2016-11-14 08:51:27.992558 CET,"perfmon","network",p19216,th-663976160,"192.168.200.33","34232",2016-11-14 02:44:57 CET,26343796,con77853,cmd185,seg-1,,dx19853465,x26343796,sx1,"ERROR","XX000","could not write 3148 bytes to temporary file: No space left on device (buffile.c:405)",,,,,,"SELECT * FROM cea.object_lock WHERE id='eea_aggregators' FOR UPDATE",0,,"buffile.c",405,"Stack trace:
1    0xace6da postgres <symbol not found> + 0xace6da
2    0xad0868 postgres elog_finish + 0xb8
3    0x91ab10 postgres BufFileFlush + 0xa0
4    0xb4a8f0 postgres dumpSharedLocalSnapshot_forCursor + 0x310
5    0xc90979 postgres verify_shared_snapshot_ready + 0x29
6    0x7255a5 postgres Executor2016-11-14 10:53:39.836789 CET,"perfmon","network",p893,th-663976160,"192.168.220.200","43963",2016-11-14 10:52:26 CET,26356962,con86898,cmd5,seg-1,,dx19864015,x26356962,sx1,"ERROR","42601","syntax error at or near ""G""",,,,,,"select * from kpi_meta_info where rat = 4G;",42,,"scan.l",979,

The host ran out of disk space and crashed. This was because of an unmatched quote here. 

Resolution 

To solve this issue, please follow these steps:

  1. Access the data located in: /data/master/gpseg-1/gpperfmon/logs
  2. Take the alert_log_stage generated by the gpdb-alert-<date>.csv file and parse it with iconv for correct formatting. 
  3. Find the offending file and remove it to allow processing to continue.
  4. It may take some trial and error to find any log that is not correctly terminated with matching quotes.

Additional Information

An error similar to this could be found in this article

For more information about iconv, refer here.  

Comments

Powered by Zendesk