Pivotal Knowledge Base

Follow

Oozie hive action fails with "java.lang.NoClassDefFoundError: org/apache/hadoop/hive/conf/HiveConf"

Problem:

0:24:52,497 ERROR ActionStartXCommand:536 - USER[nweissi] GROUP[-] TOKEN[] APP[hive-wf] JOB[0000000-140619202307001-oozie-oozi-W] ACTION[0000000-140619202307001-oozie-oozi-W@hive-node] Error,
java.lang.NoClassDefFoundError: org/apache/hadoop/hive/conf/HiveConf
at org.apache.oozie.action.hadoop.HCatCredentialHelper.getHCatClient(HCatCredentialHelper.java:79)
at org.apache.oozie.action.hadoop.HCatCredentialHelper.set(HCatCredentialHelper.java:52)
at
..
..
at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:731)
at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:945)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:215)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:60)
at org.apache.oozie.command.XCommand.call(XCommand.java:280)
at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:326)
at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:255)
at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Exception in thread "pool-2-thread-9" java.lang.NoClassDefFoundError: org/apache/hadoop/hive/conf/HiveConf
at org.apache.oozie.action.hadoop.HCatCredentialHelper.getHCatClient(HCatCredentialHelper.java:79)
at org.apache.oozie.action.hadoop.HCatCredentialHelper.set(HCatCredentialHelper.java:52)
at org.apache.oozie.action.hadoop.HCatCredentials.addtoJobConf(HCatCredentials.java:58)
at
..
..
org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:326)
at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:255)
at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
[gpadmin@pccadmin ~]$ oozie job -oozie http://pccadmin.phd.local:11000/oozie -info 0000000-140619225354896-oozie-oozi-W
Job ID : 0000000-140619225354896-oozie-oozi-W
------------------------------------------------------------------------------------------------------------------------------------
Workflow Name : hive-wf
App Path : hdfs://test/user/gpadmin/examples/apps/hive
Status : RUNNING
Run : 0
User : gpadmin
Group : -
Created : 2014-06-20 05:54 GMT
Started : 2014-06-20 05:54 GMT
Last Modified : 2014-06-20 05:54 GMT
Ended : -
CoordAction ID: -

Actions
------------------------------------------------------------------------------------------------------------------------------------
ID Status Ext ID Ext Status Err Code
------------------------------------------------------------------------------------------------------------------------------------
0000001-140619235104359-oozie-oozi-W@:start: OK - OK -
------------------------------------------------------------------------------------------------------------------------------------
0000001-140619235104359-oozie-oozi-W@hive-node PREP - - -
------------------------------------------------------------------------------------------------------------------------------------

Fix:

common.loader variable in catalog.properties sets the classpath for the Oozie launcher jobs. It is located under $CATALINA_BASE/conf directory, use /etc/default/oozie to identify the CATALINA_BASE path

Ensure that catalina.properties file has specified the required libraries. In this case, it was HiveConf.class which is packaged in hive-common.jar and hive-exec.jar, but hive library directory was missing from common.loader variable, due to which the launcher job was failing to retrieve the hcat credentials used in workflow.xml on secured cluster.

Another way to verify is to open Oozie Web Console UI at http://<oozie-server-host>:11000/oozie/. Browser over to the "System Info" -> "Java System Props" tab and look for the parameter common.loaders. It must have the required jars.

Comments

Powered by Zendesk