Pivotal Knowledge Base

Follow

Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient

Environment

Pivotal HD: All Supported Versions

Symptom

A Hive job, probably launched from Oozie, fails with error

Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient

The complete stack trace is below:

Error Message

java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:677)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
        at org.apache.oozie.action.hadoop.HiveMain.runHive(HiveMain.java:306)
        at org.apache.oozie.action.hadoop.HiveMain.run(HiveMain.java:290)
        at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:47)
        at org.apache.oozie.action.hadoop.HiveMain.main(HiveMain.java:68)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:236)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
        at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1533)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:86)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:132)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104)
        at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3043)
        at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3062)
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:503)
        ... 19 more
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1531)
        ... 25 more
Caused by: java.lang.NoSuchMethodError: org.apache.thrift.protocol.TProtocol.getScheme()Ljava/lang/Class;
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$set_ugi_args.write(ThriftHiveMetastore.java)
        at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:63)
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.send_set_ugi(ThriftHiveMetastore.java:3682)
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.set_ugi(ThriftHiveMetastore.java:3673)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:437)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:236)
        at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.(SessionHiveMetaStoreClient.java:74)
        ... 30 more

This article is only applicable if the following error is observed as a part of the stack trace:

java.lang.NoSuchMethodError: org.apache.thrift.protocol.TProtocol.getScheme

Cause

There is a JAR in the Java classpath that includes the class TProtocol but doesn't have the method "getScheme." This JAR is probably from a custom application.

Resolution

Option A: If your team knows what JAR is loading this class, change the "scope" in the "pom.xml" file to "provided" and re-deploy the JAR

Option B: If your team doesn't know what JAR is causing this issue, add the following property to your Job

<property>
   <name>oozie.launcher.mapreduce.map.java.opts</name>
   <value>-verbose</value>
</property>
  1. Re-run the job and reproduce the error
  2. Fetch the logs into a file
  3. Grep the logs for "TProtocol"
  4. Search for an output similar to this
container_e16_1483434141462_118653_01_000002.stdout:[Loaded org.apache.thrift.protocol.TProtocol from file:/grid/hadoop/yarn/local/filecache/6013/forBlogPosts-1.0-SNAPSHOT.jar] 
container_e16_1483434141462_118653_01_000002.stdout:[Loaded org.apache.thrift.protocol.TProtocolException from file:/grid/hadoop/yarn/local/filecache/6013/forBlogPosts-1.0-SNAPSHOT.jar] 
container_e16_1483434141462_118653_01_000002.stdout:Caused by: java.lang.NoSuchMethodError: org.apache.thrift.protocol.TProtocol.getScheme()Ljava/lang/Class;

Now that you know what JAR files are causing trouble, apply the change in the pom.xml mentioned in Option A, or remove this JAR from your classpath.

Additional Information

How to collect YARN logs 

 

Comments

Powered by Zendesk