Pivotal Knowledge Base

Follow

"Interconnect error writing an outgoing packet: Network is unreachable" seen during gpcheckcat execution

Environment 

Product Version
Pivotal Greenplum (GPDB) 4.3.x
OS RHEL 6.x

Symptom

During a gpcheckcat execution - queries are failing with "ERROR: Interconnect error writing an outgoing packet: Network is unreachable".

[ERROR] executing: Missing or extraneous entries check for pg_type_encoding
  Execution error: ERROR:  Interconnect error writing an outgoing packet: Network is unreachable  (seg0 slice4 sdw1:46000 pid=21056)
DETAIL:  
error during sendto() call (error:101).
For Remote Connection: contentId=-1 at 10.110.121.160:32399

The segment (Seg0 sdw1:46000) is trying to connect to the Master (contentId=-1) using IP 10.110.121.160.

Cause

As you can notice from the Error and description above, the master host was unreachable when the segment was trying to connect through the IP: 10.110.121.160.

The reason behind this error is that /etc/hosts on the master is pointing towards external IP address that cannot be accessed from the segment. Example of the bad entries are mentioned below from /etc/hosts:

10.110.121.160  mdw --> Only this IP is from ETH0, rest below are from ETH1 and in this case other segments did not have a valid route to it.
172.28.19.228   sdw1
172.28.19.230   sdw2
172.28.19.229   sdw3
172.28.19.227   smdw

Resolution

In this scenario, the entries in /etc/hosts on the master have an IP that is unreachable.

Ensure that the entries in /etc/hosts (see fix below) file on both Master and Segments are correct. On the segment, the master host entry (used in a cluster) should be pointing to internal IP. Run a quick test of "telnet <hostname-on-/etc/hosts> <port-number> and check the ability to have a successful connection.

Rerun gpcheckcat or the failing query:

10.110.121.160  mdw-esx
172.28.19.220   mdw - reflects IP from ETH1 like below
172.28.19.228   sdw1
172.28.19.230   sdw2
172.28.19.229   sdw3
172.28.19.227   smdw

Note: /etc/hosts should be consistent across the clusters. If an external IP needs to be used then make sure the hostname used in the database for master should not point towards it.

Related Articles:

Comments

Powered by Zendesk