Pivotal Knowledge Base

Follow

When connecting to gpload or gpfdist, "Connection Refused" error message is seen

Environment 

Product Version
Pivotal Greenplum (GPDB) 4.3.x

Symptom

When attempting to connect to gpfdist to work with external data, a "connection refused" error is generated. The error may be intermittent.

The error message is shown below:

2016-03-07 15:36:07|ERROR|ERROR: connection with gpfdist failed for gpfdist://hostname:8001//path/file. effective url: http://123.456.789:8001//path/file. error code = 111 (Connection refused) (seg22 slice1 sdw4:40004 pid=13246)

Cause

This issue can occur when IPv4 and IPv6 are both enabled, but the port is only available to one protocol.

In the following logging example, gpfdist first tries to establish a listening socket on port 8000. However, the port is in use for both IPv4 and IPv6. 

2016-03-17 12:28:06|LOG|trying to run gpfdist -p 8000 -P 9000 -f "/path/external.file" -t 30 -V
2016-03-17 12:28:06|LOG|gpfdist says: 2016-03-17 12:28:06 27168 INFO Before opening listening sockets - following listening sockets are available:
2016-03-17 12:28:06|DEBUG|gpfdist: [msg] libevent using: epoll
2016-03-17 12:28:06|LOG|gpfdist says: 2016-03-17 12:28:06 27168 INFO IPV6 socket: [::]:8000
2016-03-17 12:28:06|DEBUG|gpfdist: 
2016-03-17 12:28:06|LOG|gpfdist says: 2016-03-17 12:28:06 27168 INFO IPV4 socket: 0.0.0.0:8000
2016-03-17 12:28:06|LOG|gpfdist says: 2016-03-17 12:28:06 27168 INFO Trying to open listening socket:
2016-03-17 12:28:06|LOG|gpfdist says: 2016-03-17 12:28:06 27168 INFO IPV6 socket: [::]:8000
2016-03-17 12:28:06|LOG|gpfdist says: 2016-03-17 12:28:06 27168 WARN Address already in use (errno = 98), port: 8000
2016-03-17 12:28:06|LOG|gpfdist says: 2016-03-17 12:28:06 27168 INFO Trying to open listening socket:
2016-03-17 12:28:06|LOG|gpfdist says: 2016-03-17 12:28:06 27168 INFO IPV4 socket: 0.0.0.0:8000
2016-03-17 12:28:06|LOG|gpfdist says: 2016-03-17 12:28:06 27168 WARN Address already in use (errno = 98), port: 8000

As per the port range specified (8000-9000), gpfdist now tries to use the next sequential port number (8001). In this instance, the port is available to IPv6 but not available to IPv4.

2016-03-17 12:28:06|LOG|gpfdist says: 
2016-03-17 12:28:06|LOG|gpfdist says: 2016-03-17 12:28:06 27168 INFO Before opening listening sockets - following listening sockets are available:
2016-03-17 12:28:06|LOG|gpfdist says: 2016-03-17 12:28:06 27168 INFO IPV6 socket: [::]:8001
2016-03-17 12:28:06|LOG|gpfdist says: 2016-03-17 12:28:06 27168 INFO IPV4 socket: 0.0.0.0:8001
2016-03-17 12:28:06|LOG|gpfdist says: 2016-03-17 12:28:06 27168 INFO Trying to open listening socket:
2016-03-17 12:28:06|LOG|gpfdist says: 2016-03-17 12:28:06 27168 INFO IPV6 socket: [::]:8001
2016-03-17 12:28:06|LOG|gpfdist says: 2016-03-17 12:28:06 27168 INFO Opening listening socket succeeded
2016-03-17 12:28:06|LOG|gpfdist says: 2016-03-17 12:28:06 27168 INFO Trying to open listening socket:
2016-03-17 12:28:06|LOG|gpfdist says: 2016-03-17 12:28:06 27168 INFO IPV4 socket: 0.0.0.0:8001
2016-03-17 12:28:06|LOG|gpfdist says: 2016-03-17 12:28:06 27168 WARN Address already in use (errno = 98), port: 8001
2016-03-17 12:28:06|LOG|gpfdist says: Serving HTTP on port 8001, directory /path/JobMonitor/uiJobMonitor
2016-03-17 12:28:06|INFO|started gpfdist -p 8000 -P 9000 -f "/path/external.file" -t 30 -V
2016-03-17 12:28:06|LOG|gpfdist is running on port 8001

Since gpfdist was able to bind on one protocol, the binding is considered a success.

However, when GPDB attempts to connect, the logging indicates that it is resolving the hostname to the IPv4 address, and the connection fails.

2016-03-07 15:36:07|ERROR|ERROR: connection with gpfdist failed for gpfdist://hostname:8001//path/file. effective url: http://123.456.789:8001//path/file. error code = 111 (Connection refused) (seg22 slice1 sdw4:40004 pid=13246)

Resolution

As a workaround, when launching from a command line, or in the YAML configuration file, specify the desired IP address (IPv4 or IPv6) to ensure a consistent endpoint resolution. 

Additional Information

This issue has been logged as a defect (MPP-26261). 

 

 

Comments

  • Avatar
    Richard Ji

    How do you specify the desired IP address IP v4 from command line? I don't see a -h (host) option. Only a -p to specify the port.

  • Avatar
    Brendan Stephens

    You can specify a hostname or ip address when you setup the location for the table using the gpfdist:// protocol.
    However, there is no host option for gpfdist.
    All hostnames are valid on the local host and gpfdist will attempt to bond to all IP address on the host.

Powered by Zendesk