Pivotal Knowledge Base

Follow

Oozie jobs fails with "Protocol message end-group tag did not match expected tag"

Environment:

  • PHD 1.1.1
  • Oozie 3.3.2

Symptom:

Oozie jobs may fail indicating an exception UndeclaredThrowableException and Protocol message end-group tag did not match expected tag in the logs

Snippet:

Check the status of the job:

[hadoop@hdm1 ~]$ oozie job -oozie http://hdm1.phd.local:11000/oozie -info 0000001-140523001749705-oozie-oozi-W
Job ID : 0000001-140523001749705-oozie-oozi-W
------------------------------------------------------------------------------------------------------------------------------------
Workflow Name : map-reduce-wf
App Path : hdfs://hdm1.phd.local:8020/user/hadoop/apps/map-reduce
Status : KILLED
Run : 0
User : hadoop
...
0000001-140523001749705-oozie-oozi-W@mr-node ERROR - ERROR UndeclaredThrowableException

In this example, Oozie server and Resource manager are on the same node : hdm1.phd.local

Review Oozie logs to review further details on the error message.

[hadoop@hdm1 ~]$ oozie job -oozie http://hdm1.phd.local:11000/oozie -log 0000001-140523001749705-oozie-oozi-W log
2014-05-23 00:20:55,354  INFO ActionStartXCommand:539 - USER[hadoop] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000001-140523001749705-oozie-oozi-W] ACTION
...
...
[UndeclaredThrowableException], Message [UndeclaredThrowableException: null]
...
Caused by: com.google.protobuf.ServiceException: java.io.IOException: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is: "hdm1.phd.local/172.28.16.60"; destination host is: "hdm1.phd.local":8088;
     at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:212)

Cause:

Exception class com.google.protobuf.ServiceException is thrown by blocking RPC methods when a failure occurs and there can be multiple reason for this exception, but it indicates that it did not receive an expected response. So, it's essential that to check the configuration of the oozie setup.

In this case, oozie job was run using configuration file job.properties which had jobTracker defined as hdm1.phd.local:8088. However, on a Yarn setup, jobTracker must point to  yarn.resourcemanager.address configured in the /etc/gphd/hadoop/conf/yarn-site.xml and by default the port used is 8032. 8088 is the default port for yarn.resourcemanager.webapp.address and must not be used unless it has been changed.

Fix jobTracker to point to yarn.resourcemanager.address value in job.properties configuration file and rerun the job.

Job executed: oozie job -oozie http://smdw.gphd.local:11000/oozie -config job.properties -run

[hadoop@hdm1 ~]$ oozie job -oozie http://hdm1.phd.local:11000/oozie -info 0000004-140523001749705-oozie-oozi-W Job ID : 0000004-140523001749705-oozie-oozi-W ------------------------------------------------------------------------------------------------------------------------------------ Workflow Name : map-reduce-wf App Path : hdfs://phdha/user/hadoop/examples/apps/map-reduce Status : SUCCEEDED Run : 0 User : hadoop 0000004-140523001749705-oozie-oozi-W@mr-node OK job_1400828609740_0006 SUCCEEDED -

Comments

Powered by Zendesk