|GPDB or HAWQ|
|OS: Centos or Linux|
|DCA or software-only|
In this article, we will learn how to deploy, start and stop the mem_watcher daemon on all the cluster nodes.
The mem_watcher utility is a daemon that runs on all the servers of GPDB/HAWQ cluster and tracks the memory usage of each process on GPDB/HAWQ every minute (by default) i.e. it collects PS output every minute. It is a low impact process that consumes only 4 MB of memory. You should expect that it will generate approximately 30 MB of data over a 24-hour period.
Download the attached script and move it to the GPDB/HAWQ master server under:
Here are the instructions to start the mem_watcher daemon:
- Log on to the host where mem_watcher was installed (usually a 'master' host) as user gpadmin.
- Move to the mem_watcher directory:
- Create a hostmap.txt file with all cluster hosts, along with the absolute path you want to use to maintain the mem_watcher output.
Example hostmap.txt file like:
master1.phd.local:/home/gpadmin/mem_watcher/working slave1.phd.local:/home/gpadmin/mem_watcher/working slave2.phd.local:/home/gpadmin/mem_watcher/working slave3.phd.local:/home/gpadmin/mem_watcher/working
- In case you have many segment host a simple script like below should help in creating a hostmap , provide the "location" parameter value.
export location=<location-to-collect-the-memory-stats> psql -d template1 -Atc "select distinct hostname||':$location' from gp_segment_configuration" > hostmap.txt
- On each of the cluster hosts above, create the working directory (above) as user 'gpadmin'. for example:
gpssh -f all_hosts "mkdir -p /home/gpadmin/mem_watcher/working"
- Start mem_watcher using
./mem_watcher -f ./hostmap.txt
- Verify the mem_watcher demons are running using the below command
gpssh -f <hostfile> "ps -ef |grep mem_watcher|grep -v grep"
This section includes instructions to stop all the mem_watcher processes, retrieve all data files to the current directory and remove the work files on all the hosts.
Note: Ensure that there is enough space in the current directory to retrieve the data files.
- Log on to the host where mem_watcher was deployed from (usually a 'master' host) as user gpadmin.
- Navigate to the mem_watcher directory:
- Stop mem_watcher:
./mem_watcher -f ./hostmap.txt --stop
- The mem_watcher logs should be collected to the current directory.
Reference: Follow the guide to understand the mem_watcher output.
- If you have executed the mem_watcher once, reexecuting the mem_watcher with the same location in the hostmap would indicate the error.
< cannot create directory: ..../gpsupport_reswatch ' file exists >
Change the directory location in the hostmap to avoid the issue.
- If you execute the mem_watcher from any different location other than location where the mem_watcher executables are present (for example, /usr/local/pivotal-support/bin) you will encounter the following error:
< mem_watcher: No such file or directory .... Error when trying to copy script to .... >
Ensure you are on the bin directory, where the mem_watcher is present.