How to change log retention policy in DCA for GPDB master and segments.
Greenplum DCA has feature to configure your log management which controls GPDB, CommandCenter and Hadoop Logs management and this controlled from the configuration settings in the dca_log_cleanup configuration file. This file is located in /opt/dca/etc/log_cleanup/dca_log_cleanup.conf
This file has 4 sections for each individual components like kernelCores, UserCores, GPDB master, GPDB GpAdminLogs, GPDB Primary segments, GPDB mirror segments, gpsmonLogsHadoop and HadoopLogs, the four sections are mainly
- filesetRegex: a regular expression used as a "glob" to match the set of files
- maxFiles: The maximum number of items (files or directories) matched by the glob allowed. Any over this limit will be deleted.
- maxSize: The maximum size of all the items allowed. This must be specified in megabytes or gigabytes using an "M" or "G".
- continuousCheck: Specify "False" if only a single check after startup is needed. This is optional and defaults to "True" which enables checking of the fileset on every polling cycle.
For Example GPDB master log rotation settings.
[master_gpseg] filesetRegex: /data/master/gpseg-1/pg_log/gpdb-*.csv maxFiles:25 maxSize:10G continuousCheck:True
Based on the above settings
- Maximum logs that can be stored in $MASTER_DATA_DIRECTORY/pg_log/ with file names like gpdb-*.csv is 25, If logs are > 25 then older logs will be purged.
- Sum of all the file matching "gpdb-*.csv" should not be > 10GB, if its greater than 10GB, older logs will be deleted to bring down the value of file matching with the name gpdb*csv < 10 GB , but if its still > 10GB or if current gpdb logfile is > 10GB then current log will be truncated, thats when you may see gpdb logs for the current date/time are missing. To avoid missing logs you need to make sure you maxSize is configured properly.
Whenever logs are removed by dca_log_cleanup an entry is added to /var/log/messages on what it had removed.
Modification to configuration
Any changes/modfication to this file which require a restart of the service as mentioned below instructions.
Following steps need to be followed to configure or make any changes to dca_log_cleanup.conf file.
-- Step1: Stop service from master.
[root@mdw etc]# service dca_log_cleanup stop Stopping dca_log_cleanup.py: [ OK ]
-- Step2: Modify settings per your requirement to "/opt/dca/etc/log_cleanup/dca_log_cleanup.conf".
[root@mdw etc]# vi /opt/dca/etc/log_cleanup/dca_log_cleanup.conf
-- Step3: Start service.
[root@mdw etc]# chkconfig dca_log_cleanup on [root@mdw etc]# service dca_log_cleanup start Starting dca_log_cleanup.py: [ OK ]
-- Step4: Make sure service is running without any errors.
[root@mdw etc]# service dca_log_cleanup status dca_log_cleanup.py (pid 19953) is running... [root@mdw etc]# ps -ef |grep -v grep |grep dca_log_cleanup root 19953 1 0 10:41 ? 00:00:00 python /opt/dca/bin/dca_log_cleanup.py