Pivotal Knowledge Base

Follow

Hadoop components fail to start

Environment

Component Version
Apache Ambari 2.2.1
HDP (Hortonworks Data Platform, Hadoop) 2.3.4 
PHD 3.0.x
OS Linux

Symptom

When deploying a cluster, some services fail to start with the following symptoms in Ambari:

The error message is shown below:

resource_management.core.exceptions.Fail: Execution of 'curl -sS -L -w '%{http_code}' -X GET 'http://clpqn1pdhdbmn01.infosolco.net:50070/webhdfs/v1/ats/done?op=GETFILESTATUS&user.name=hdfs'' returned status_code=503. 
<HTML><HEAD>
<TITLE>Network Error</TITLE>
</HEAD>
<BODY>
<FONT face="Helvetica">
<big><strong></strong></big><BR>
</FONT>
<blockquote>
<TABLE border=0 cellPadding=1 width="80%">
<TR><TD>
<FONT face="Helvetica">
<big>Network Error (tcp_error)</big> 

Cause

An HTTP proxy environment variable is set on one or more cluster hosts. This can be confirmed by checking "env" on the affected node:

[root@NAMENODE ~]# env | grep -i proxy
http_proxy=http://user:password@prox-server:3128
https_proxy=http://user:password@prox-server:3128
[root@NAMENODE ~]# 

Resolution

Remove "HTTP_PROXY" and "HTTPS_PROXY" settings from the root (or the configured Ambari agent user) profile and restart Ambari Agent. The places that this may be set include the following:

/etc/bashrc
/etc/profile
~root/.bashrc
~root/.bash_profile 

Steps

  1. Manually remove HTTP_PROXY and HTTPS_PROXY from the Ambari Server user's environment setup files. For root, (the default Ambari Server user) those files are: /etc/bashrc , /etc/profile, ~root/.bashrc,  and ~root/.bash_profile.
  2. Restart the Ambari Server and Ambari Agent daemons. In CentOS and Red Hat 6.x, execute the following command as root:
    service ambari-agent restart
    service ambari-server restart
    

     

Comments

Powered by Zendesk