Pivotal Knowledge Base

Follow

Hive CLI hangs for an extended period of time, then fails with "No Route to Host" in Pivotal HD

Environment

Product Version
Pivotal HD 3.x
Hive 0.14.x

Symptom

When all Hive servers are up, the connection to the Hive CLI may hang for an extended period of time and will eventually fail with a connection error such the error shown below:

Error Message:

[root@dn1 ~]# hive
Logging initialized using configuration in file:/etc/hive/conf/hive-log4j.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/phd/3.0.1.0-1/hadoop/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/phd/3.0.1.0-1/hive/lib/hive-jdbc-0.14.0.3.0.1.0-1-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Exception in thread "main" java.lang.RuntimeException: java.net.NoRouteToHostException: No Route to Host from dn1.lab/192.168.177.149 to phd301.lab:8020 failed on socket timeout exception: java.net.NoRouteToHostException: No route to host; For more details see: http://wiki.apache.org/hadoop/NoRouteToHost
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:444)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:672)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:616)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

Cause 

As well as connecting to the Hive services, the Hive CLI also needs to connect to the NameNode. By default, the NameNode is running on port 8020. If the above error message is seen, the Hive CLI does not have access to the namenode server.

Resolution

1. In Ambari, determine the hostname and port that the NameNode is using by searching for the configuration "dfs.namenode.rpc-address" under HDFS (Hadoop Distributed File System):

2. Confirm that the port is blocked by trying to run telnet from the host that the HIVE CLI is being run on:

[root@dn1 ~]# telnet phd301 8020
Trying 192.168.177.146...
telnet: connect to address 192.168.177.146: No route to host
[root@dn1 ~]#

3. Review firewall rules in place to allow network traffic from the Hive CLI host to the Namenode host on the necessary port. 

4. If the connection still fails, enable debug logging in the Hive CLI to help understand the issue further.

 

Comments

Powered by Zendesk