Pivotal Knowledge Base

Follow

Pivotal Command Center does not disaply any data in Mapreduce and Yarn app monitor tabs

Environment

  • PHD 2.0.1.0

Symptom

Cause

In this case the yarn resourcemanager is up and running and mapreduce jobs are executing successfully.

Pivotal Command Center uses postgres database to store metrics and state of the hadoop cluster. When the status field in deploymentstatus table is set to something other then one of the following states PCC will skip polling for the given role.

  • START_FAILED
  • START_PROGRESS
  • STOP_PROGRESS
  • STOP_FAILED
  • STARTED
[gpadmin@pccadmin ~]$ psql -U postgres -d gphdmgr  -p 10432 -c "select role,status from deploymentstatus where role = 'yarn-resourcemanager';"
         role         |    status
----------------------+--------------
 yarn-resourcemanager | UNINST_PROGRESS
(1 row)

In some cases we saw situations where there was no entryin this table for role yarn-resourcemanager. This will have the same impact to PCC. 

Workaround

  1. After you have verified the issue stems from a bad entry in the deploymentstatus table then we can workaround this by issuing "icm_client start". This will not impact the services because it simply will attempt to start a service that is allready started
    [gpadmin@pccadmin ~]$ icm_client start -l phd
    Starting services
    Starting cluster
    [====================================================================================================] 100%
    Results:
    hdw3.phd.local... [Success]
    hdw1.phd.local... [Success]
    hdw2.phd.local... [Success]
    hdm1.phd.local... [Success]
    Details at /var/log/gphd/gphdmgr/
  • Make sure the start updated the deploymentstatus table correctly
    [gpadmin@pccadmin ~]$ psql -U postgres -d gphdmgr  -p 10432 -c "select role,status from deploymentstatus where role = 'yarn-resourcemanager';"
             role         | status
    ----------------------+---------
     yarn-resourcemanager | STARTED
    (1 row)

Pivotal internal JIRA reference CC-4005

Comments

Powered by Zendesk