Environment
Product | Version |
PHD | 1.x |
Symptom
This article discusses how to fix a failed Hive query. Hive query fails with the following error:
"Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient".
The problem results in Out Of Memory (OOM).
Error Message:
hive> show tables;
FAILED: Error in metadata: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
Cause
The Out Of Memory (OOM) error, in this case, is a result of the server not having enough memory to start HiveMetaStore service.
RCA
HiveMetaStore service is configured in hive-site.xml on the client side, but the service is not started.
For example:
/etc/gphd/hive-0.11.0_gphd_2_1_0_0/conf/hive-site.xml
<property>
<name>hive.metastore.uris
<value>thrift://hdw1.viadea.com:9083
</property>
However on hdw1:
-bash-4.1$ service hive-metastore status
hive-metastore dead but pid file exists
After observing the hive-metastore.log, you can see that reason why the service failed to start was due to an Out of Memory error, as shown in the example below.
/var/log/gphd/hive/hive-metastore.log:
14/04/07 16:43:13 WARN conf.HiveConf: DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore.
WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
library initialization failed - unable to allocate file descriptor table - out of memory
Resolution
- Firstly resolve the root cause why HiveMetaStore service is not started. In this case, increase the physical memory of the server.
- Then start the HiveMetaStore service manually using root user:
service hive-metastore start
Comments