Pivotal Knowledge Base

Follow

When Connecting to gpload or gpfdist, "Connection Refused" Error Message is Seen

Environment 

Pivotal Greenplum Database(GPDB) 4.3.x

Symptom

When attempting to connect to gpfdist to work with external data, a "connection refused" error is generated. The error may be intermittent.

The error message is shown below:

2016-03-07 15:36:07|ERROR|ERROR: connection with gpfdist failed for gpfdist://hostname:8001//path/file. effective url: http://123.456.789:8001//path/file. error code = 111 (Connection refused) (seg22 slice1 sdw4:40004 pid=13246)

Cause

This issue can occur when IPv4 and IPv6 are both enabled, but the port is only available to one protocol.

In the following logging example, gpfdist first tries to establish a listening socket on port 8000. However, the port is in use for both IPv4 and IPv6. 

2016-03-17 12:28:06|LOG|trying to run gpfdist -p 8000 -P 9000 -f "/path/external.file" -t 30 -V
2016-03-17 12:28:06|LOG|gpfdist says: 2016-03-17 12:28:06 27168 INFO Before opening listening sockets - following listening sockets are available:
2016-03-17 12:28:06|DEBUG|gpfdist: [msg] libevent using: epoll
2016-03-17 12:28:06|LOG|gpfdist says: 2016-03-17 12:28:06 27168 INFO IPV6 socket: [::]:8000
2016-03-17 12:28:06|DEBUG|gpfdist: 
2016-03-17 12:28:06|LOG|gpfdist says: 2016-03-17 12:28:06 27168 INFO IPV4 socket: 0.0.0.0:8000
2016-03-17 12:28:06|LOG|gpfdist says: 2016-03-17 12:28:06 27168 INFO Trying to open listening socket:
2016-03-17 12:28:06|LOG|gpfdist says: 2016-03-17 12:28:06 27168 INFO IPV6 socket: [::]:8000
2016-03-17 12:28:06|LOG|gpfdist says: 2016-03-17 12:28:06 27168 WARN Address already in use (errno = 98), port: 8000
2016-03-17 12:28:06|LOG|gpfdist says: 2016-03-17 12:28:06 27168 INFO Trying to open listening socket:
2016-03-17 12:28:06|LOG|gpfdist says: 2016-03-17 12:28:06 27168 INFO IPV4 socket: 0.0.0.0:8000
2016-03-17 12:28:06|LOG|gpfdist says: 2016-03-17 12:28:06 27168 WARN Address already in use (errno = 98), port: 8000

As per the port range specified (8000-9000), gpfdist now tries to use the next sequential port number (8001). In this instance, the port is available to IPv6 but not available to IPv4.

2016-03-17 12:28:06|LOG|gpfdist says: 
2016-03-17 12:28:06|LOG|gpfdist says: 2016-03-17 12:28:06 27168 INFO Before opening listening sockets - following listening sockets are available:
2016-03-17 12:28:06|LOG|gpfdist says: 2016-03-17 12:28:06 27168 INFO IPV6 socket: [::]:8001
2016-03-17 12:28:06|LOG|gpfdist says: 2016-03-17 12:28:06 27168 INFO IPV4 socket: 0.0.0.0:8001
2016-03-17 12:28:06|LOG|gpfdist says: 2016-03-17 12:28:06 27168 INFO Trying to open listening socket:
2016-03-17 12:28:06|LOG|gpfdist says: 2016-03-17 12:28:06 27168 INFO IPV6 socket: [::]:8001
2016-03-17 12:28:06|LOG|gpfdist says: 2016-03-17 12:28:06 27168 INFO Opening listening socket succeeded
2016-03-17 12:28:06|LOG|gpfdist says: 2016-03-17 12:28:06 27168 INFO Trying to open listening socket:
2016-03-17 12:28:06|LOG|gpfdist says: 2016-03-17 12:28:06 27168 INFO IPV4 socket: 0.0.0.0:8001
2016-03-17 12:28:06|LOG|gpfdist says: 2016-03-17 12:28:06 27168 WARN Address already in use (errno = 98), port: 8001
2016-03-17 12:28:06|LOG|gpfdist says: Serving HTTP on port 8001, directory /path/JobMonitor/uiJobMonitor
2016-03-17 12:28:06|INFO|started gpfdist -p 8000 -P 9000 -f "/path/external.file" -t 30 -V
2016-03-17 12:28:06|LOG|gpfdist is running on port 8001

Since gpfdist was able to bind on one protocol, the binding is considered a success.

However, when GPDB attempts to connect, the logging indicates that it is resolving the hostname to the IPv4 address, and the connection fails.

2016-03-07 15:36:07|ERROR|ERROR: connection with gpfdist failed for gpfdist://hostname:8001//path/file. effective url: http://123.456.789:8001//path/file. error code = 111 (Connection refused) (seg22 slice1 sdw4:40004 pid=13246)

Resolution

As a workaround, when launching from a command line, or in the YAML configuration file, specify the desired IP address (IPv4 or IPv6) to ensure a consistent endpoint resolution.

Additional Information

This issue has been logged as a defect (MPP-26261). 

 

Comments

  • Avatar
    Richard Ji

    How do you specify the desired IP address IP v4 from command line? I don't see a -h (host) option. Only a -p to specify the port.

  • Avatar
    Brendan Stephens

    You can specify a hostname or ip address when you setup the location for the table using the gpfdist:// protocol.
    However, there is no host option for gpfdist.
    All hostnames are valid on the local host and gpfdist will attempt to bond to all IP address on the host.

  • Avatar
    Darryl Head

    Same Issue as this is related to specifically an IPV6 addressing. My customer is mandated to move to all external NIC interfaces utilizing only IPV6 addressing. Therefore in my efforts to use GPFDIST for now but will need similar solution for GPLOAD (gpload.yaml file) here is what I have tried and not been successful.

    Per above "You can specify a hostname or ip address when you setup the location for the table using the gpfdist:// protocol."

    ********************************************************************************************************************************

    northwind=# CREATE READABLE EXTERNAL TABLE ext_customers_readable_ipv6_nossl(LIKE customers_new)
    northwind-# LOCATION ('gpfdist://[fe80::21e:67ff:fe6a:bcd2]:9002/customers_new_dump.*')
    northwind-# FORMAT 'TEXT' ( delimiter '|' null '#' escape E'\\' );
    CREATE EXTERNAL TABLE
    northwind=#

    northwind=# \d+ ext_customers_readable_ipv6_nossl
    External table "public.ext_customers_readable_ipv6_nossl"
    Column | Type | Modifiers | Storage | Description
    --------------+-----------------------+-----------+----------+-------------
    customerid | bpchar | | extended |
    companyname | character varying(40) | | extended |
    contactname | character varying(30) | | extended |
    contacttitle | character varying(30) | | extended |
    address | character varying(60) | | extended |
    city | character varying(15) | | extended |
    region | character varying(15) | | extended |
    postalcode | character varying(10) | | extended |
    country | character varying(15) | | extended |
    phone | character varying(24) | | extended |
    fax | character varying(24) | | extended |
    Type: readable
    Encoding: UTF8
    Format type: text
    Format options: delimiter '|' null '#' escape '\'
    External location: gpfdist://[fe80::21e:67ff:fe6a:bcd2]:9002/customers_new_dump.*

    northwind=# select * from ext_customers_readable_ipv6_nossl;

    ERROR: connection with gpfdist failed for gpfdist://[fe80::21e:67ff:fe6a:bcd2]:9002/customers_new_dump.*. effective url:
    http://[fe80::21e:67ff:fe6a:bcd2]:9002/customers_new_dump.*. error code = 22 (Invalid argument) (seg25 slice1 sdw4.gphd.local:1026 pid=433730)

    @ the ETL Host which in this test is the MDW host.

    [gpadmin@smdw tmp]$ /usr/local/greenplum-db/bin/gpfdist -p 9002 -d /tmp -l /tmp/gpfdist_new.9002.log -t 30 -V &
    [2] 191764
    [gpadmin@smdw tmp]$ [msg] libevent using: epoll

    2018-04-17 09:02:16 191764 INFO Before opening listening sockets - following listening sockets are available:
    2018-04-17 09:02:16 191764 INFO IPV6 socket: [::]:9002
    2018-04-17 09:02:16 191764 INFO IPV4 socket: 0.0.0.0:9002
    2018-04-17 09:02:16 191764 INFO Trying to open listening socket:
    2018-04-17 09:02:16 191764 INFO IPV6 socket: [::]:9002
    2018-04-17 09:02:16 191764 INFO Opening listening socket succeeded
    2018-04-17 09:02:16 191764 INFO Trying to open listening socket:
    2018-04-17 09:02:16 191764 INFO IPV4 socket: 0.0.0.0:9002
    2018-04-17 09:02:16 191764 WARN Address already in use (errno = 98), port: 9002
    Serving HTTP on port 9002, directory /tmp

    [gpadmin@smdw tmp]$ ps -ef |grep 9002
    gpadmin 191764 185238 0 09:02 pts/0 00:00:00 /usr/local/greenplum-db/bin/gpfdist -p 9002 -d /tmp -l /tmp/gpfdist_new.9002.log -t 30 -V
    gpadmin 192183 185238 0 09:06 pts/0 00:00:00 grep 9002

    (NOTE: attached logfile /tmp/gpfdist_new.9002.log is empty)

    [gpadmin@smdw tmp]$ /sbin/ifconfig -a |more

    eth0 Link encap:Ethernet HWaddr 00:1E:67:6A:BC:D2
    inet addr:172.28.4.251 Bcast:172.28.7.255 Mask:255.255.248.0
    inet6 addr: fd11::16/64 Scope:Global
    inet6 addr: fe80::21e:67ff:fe6a:bcd2/64 Scope:Link
    UP BROADCAST RUNNING ALLMULTI MULTICAST MTU:1500 Metric:1
    RX packets:1876839 errors:0 dropped:0 overruns:0 frame:0
    TX packets:137010 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:1000
    RX bytes:299765179 (285.8 MiB) TX bytes:14071760 (13.4 MiB)
    Memory:d0960000-d097ffff

    bond0 Link encap:Ethernet HWaddr 8C:7C:FF:30:64:4E
    inet addr:172.28.8.251 Bcast:172.28.11.255 Mask:255.255.252.0
    inet6 addr: fd11::14/64 Scope:Global
    inet6 addr: fe80::8e7c:ffff:fe30:644e/64 Scope:Link
    UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
    RX packets:490880 errors:0 dropped:0 overruns:0 frame:0
    TX packets:472278 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:0
    RX bytes:64861106 (61.8 MiB) TX bytes:54627566 (52.0 MiB)

    bond0.125 Link encap:Ethernet HWaddr 8C:7C:FF:30:64:4E
    inet addr:10.1.1.3 Bcast:10.1.1.255 Mask:255.255.255.0
    inet6 addr: fd11::15/64 Scope:Global
    inet6 addr: fe80::8e7c:ffff:fe30:644e/64 Scope:Link
    UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
    RX packets:8 errors:0 dropped:0 overruns:0 frame:0
    TX packets:13 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:0
    RX bytes:336 (336.0 b) TX bytes:954 (954.0 b)

    bond1 Link encap:Ethernet HWaddr 00:1E:67:6A:BC:D3
    inet addr:10.237.192.38 Bcast:10.237.192.255 Mask:255.255.255.0
    inet6 addr: fe80::21e:67ff:fe6a:bcd3/64 Scope:Link
    UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
    RX packets:1092583 errors:0 dropped:0 overruns:0 frame:0
    TX packets:673 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:0
    RX bytes:128104121 (122.1 MiB) TX bytes:46747 (45.6 KiB)

Powered by Zendesk