Pivotal Knowledge Base

Follow

gpfdist: "HTTP/1.0 408 time out ..." error reported from segment

Environment

  • Pivotal HDB 
  • Pivotal Greenplum

Symptom

Sometimes loading data into HDB/GPDB cluster with gpfdist will fail with following error message reported from segment instance.

ERROR: http response code 408 from gpfdist (gpfdist://avw7hdw3d1.ae.ge.com:8090/hawq_mig_pipe): HTTP/1.0 408 time out (url.c:290) (seg26 hdw3.gphd.local:40006 pid=167513) (cdbdisp.c:1571)

Similar timeout message could also be seen in gpfdist.log

2015-07-02 15:48:37 85869 WARN [0:2:0:8] HTTP ERROR: ::ffff:10.81.103.91 - 408 time out

Cause

In some scenario like gpfdist-based writable external table the gpfdist server process need to read data from segment instances of HDB/GPDB cluster. A timer is set at gpfdist side to control allowed time for segment instance to establish a connection to a gpfdist process. The default timeout value is 5 seconds. And allowed values are 2 to 600 seconds for Greenplum Database (2~30 for HDB).

However when there is a lot of network traffic on the system or it's very busy at gpfdist/segment instance side, it's possible that the connection fails to be established before timeout occurs.

Fix

If the timeout is due to heavy traffic on network or gpfdist/segment instance busy with processing tasks, it may be solved by increase timeout value with "-t" option when start gpfdist. 

However if the maximum allowed value is still not enough, then "-t 0" could be tried to disable timeout.

"-t 0" is not mentioned in manual, but it does exist, which could be shown in the following way.

Greenplum Database

[gpadmin@mdw ~]$ gpfdist -t 10000
Error: -t timeout must be between 2 and 600, or 0 for no timeout

HDB

[gpadmin@hdm1 ~]$ gpfdist -t 1000
Error: -t timeout must be between 2 and 30, or 0 for no timeout

Refer to internal JIRA GPSQL-3191. 

Comments

Powered by Zendesk