Environment

Symptom

This article discusses how to fix a failed Hive query. Hive query fails with the following error:

"Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient".

The problem results in Out Of Memory (OOM).

Error Message:

hive> show tables;
FAILED: Error in metadata: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask

Cause

The Out Of Memory (OOM) error, in this case, is a result of the server not having enough memory to start HiveMetaStore service.

RCA

HiveMetaStore service is configured in hive-site.xml on the client side, but the service is not started.

For example:

/etc/gphd/hive-0.11.0_gphd_2_1_0_0/conf/hive-site.xml

<property>
<name>hive.metastore.uris
<value>thrift://hdw1.viadea.com:9083
</property>

However on hdw1:

-bash-4.1$ service hive-metastore status
hive-metastore dead but pid file exists

After observing the hive-metastore.log, you can see that reason why the service failed to start was due to an Out of Memory error, as shown in the example below.

/var/log/gphd/hive/hive-metastore.log:

14/04/07 16:43:13 WARN conf.HiveConf: DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore.
WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
library initialization failed - unable to allocate file descriptor table - out of memory 

Resolution

1. Firstly resolve the root cause why HiveMetaStore service is not started. In this case, increase the physical memory of the server.
2. Then start the HiveMetaStore service manually using root user:

service hive-metastore start