Pivotal Knowledge Base

Follow

Ambari - Alerts show Ambari is unable to contact HiveServer2 on port 10000

Environment

Product Version
PHD 3.x
Ambari 1.7.1
Hive 0.14.0

Overview

Alerts and Health check in the HiveServer show HiveServer2 process: "Connection to <hiveserver hostname> on port 10000 failed: 

Symptom

HiveServer2 works correctly however the above alert shows up in Ambari which is misleading.

Cause

HiveServer2 has two transport modes: 

  • binary - this is the default mode and by default will listen on port 10000
  • HTTP - This is the non-default mode and by default will listen on port 10001

The ports used may change depending on what has been configured in Ambari. This is what the configuration looks like by default:

If hive.server2.transport.mode is set to "binary" then hiveserver2 will listen on the port set in Hive Server Port in this case 10000 (this is the default setting).

If hive.server2.transport.mode is set to "http" then hiveserver2 will listen on the port set in hive.server2.thrift.http, in this case 10001. 

Ambari is not aware of the possible port changes and whatever the transport mode Ambari will only check for hiveserver2 on the port 10000 which causes this reporting issue if the HTTP mode is used. 

Resolution

This is a reporting issue, if hiveserver2 is working correctly the alert can be ignored, however if hiveserver2 goes down the Hive administrator may not be correctly notified. 

To fully resolve this reporting issue: 

  1. Upgrade to Ambari 2.1.2
  2. Make sure Ambari Metrics is installed
  3. Restart HIVE 

 

Comments

  • Avatar
    Dino Bukvic

    I encountered this issue on the same environment (Ambari 1.7.1 + PHD 3.0.1 ). After investigation, I noticed that the ambari-agent hive-site.xml file, on the server where hiveserver2 is hosted, does not get updated. Resolution is to update the /var/lib/ambari-agent/cache/stacks/PHD/3.0/services/HIVE/configuration/hive-site.xml and change hive.server2.transport.mode to "http". Restart ambari-server, ambari-agent and nagios. After doing this the Alert went away.
    Hope it helps.

Powered by Zendesk