Pivotal Knowledge Base

Follow

Hive Cli fails to start with the message "TezSession has already shutdown."

Environment

Product Version
Pivotal HD  3.x
TEZ  0.5.x
Ambari  1.7 / 2.1

Symptom 

Hive CLI fails to start with the following message:

[hdfs@datanode2 hive]$ hive
WARNING: Use "yarn jar" to launch YARN applications.

Logging initialized using configuration in file:/etc/hive/2.3.2.0-2950/0/hive-log4j.properties
Exception in thread "main" java.lang.RuntimeException: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. Application application_1467175461700_0015 failed 2 times due to AM Container for appattempt_1467175461700_0015_000002 exited with exitCode: 1
For more detailed output, check application tracking page:http://datatnode2.local.lab:8088/cluster/app/application_1467175461700_0015Then, click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_e12_1467175461700_0015_02_000001
Exit code: 1
Stack trace: ExitCodeException exitCode=1:

Error Message:

When reviewing the YARN application log by running "yarn log -applicationId application_1467175461700_0015" the following can be seen:

2016-06-29 15:27:05,647 [INFO] [main] |launcher.ContainerLauncherImpl|: Upper limit on the thread pool size is 500
2016-06-29 15:27:05,647 [INFO] [main] |history.HistoryEventHandler|: Initializing HistoryEventHandler withrecoveryEnabled=true, hist
oryServiceClassName=org.apache.tez.dag.history.logging.ats.ATSV15HistoryLoggingService
2016-06-29 15:27:05,649 [INFO] [main] |service.AbstractService|: Service org.apache.tez.dag.history.HistoryEventHandler failed in st
ate INITED; cause: org.apache.tez.dag.api.TezReflectionException: Unable to load class: org.apache.tez.dag.history.logging.ats.ATSV1
5HistoryLoggingService
org.apache.tez.dag.api.TezReflectionException: Unable to load class: org.apache.tez.dag.history.logging.ats.ATSV15HistoryLoggingServ
ice
at org.apache.tez.common.ReflectionUtils.getClazz(ReflectionUtils.java:46)
at org.apache.tez.common.ReflectionUtils.createClazzInstance(ReflectionUtils.java:78)
at org.apache.tez.dag.history.HistoryEventHandler.serviceInit(HistoryEventHandler.java:61)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.tez.dag.app.DAGAppMaster.initServices(DAGAppMaster.java:1671)
at org.apache.tez.dag.app.DAGAppMaster.serviceInit(DAGAppMaster.java:554)

Cause 

This is caused by the Ambari software bug 15041.

Resolution

1. Open Ambari.

2. Go to TEZ / Configs / Advanced tez-site.

3. Locate the configuration " tez.history.logging.service.class".

4. Replace the value "org.apache.tez.dag.history.logging.ats.ATSV15HistoryLoggingService" with the new value: "org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService"

5. Save the configuration changes.

6. Restart all services that Ambari is asking you to restart.

 

Comments

Powered by Zendesk