Pivotal Knowledge Base

Follow

Loading Data into Pivotal HDB using PXF external tables returns error "failed sending to remote component"

Environment

  • Pivotal HDB 1.2.1

Symptom

1. Create a writable external table in Pivotal HDB based on one existing table using PXF service.

CREATE WRITABLE EXTERNAL TABLE backups.dwh_agg_fact_usage_2014_12_16_020547 (LIKE dwh.agg_fact_usage) LOCATION ('pxf://qrioustempdev/home/skinny//backups/hdb_backups/2014/12/16/020547/skinny_live/dwh/agg_fact_usage/table_rows?profile=HdfsTextSimple') FORMAT 'TEXT';

2. Insert data into backups.dwh_agg_fact_usage_2014_12_16_020547 from table dwh.agg_fact_usage. Note that dwh.agg_fact_usage is very large and has over 180 million records.

INSERT INTO backups.dwh_agg_fact_usage_2014_12_16_020547 SELECT * FROM dwh.agg_fact_usage;

3. After a while INSERT query fails with following errors

psql:./12/16/020547/skinny_live/dwh/agg_fact_usage/dump_table.sql:3: ERROR: failed sending to remote component '10.100.160.84:51200' (libchurl.c:551) (seg23 qt-phdnode-3.corp.qrious.co.nz:40003 pid=48095) (cdbdisp.c:1571)

 4. The following error messages is found in segment 23 logs

2014-12-18 12:58:56.207567 NZDT,"gpadmin","skinny_live",p48095,th1872422944,"10.
100.160.73","29210",2014-12-18 12:58:13 NZDT,723077,con1684,cmd5,seg23,,,x723077
,sx1,"ERROR","XX000","failed sending to remote component '10.100.160.84:51200' (
libchurl.c:551)",,,,,,"INSERT INTO backups.dwh_agg_fact_usage_2014_12_16_020547
SELECT * FROM dwh.agg_fact_usage;",0,,"libchurl.c",551,"Stack trace:
1 0xaf4dca postgres <symbol not found> + 0xaf4dca
2 0xaf6ef8 postgres elog_finish + 0xb8
3 0x5668f1 postgres churl_write + 0x1d1
4 0x7f2354b3fa61 pxf.so gpbridge_write + 0x31
5 0x7f2354b4034a pxf.so gpbridge_export + 0x3a
6 0x55c48c postgres <symbol not found> + 0x55c48c
7 0x55f1c4 postgres url_fwrite + 0x84
8 0x5572a3 postgres external_insert + 0x193
9 0x7b3796 postgres ExecInsert + 0xd86
10 0x7b1d94 postgres ExecDML + 0x1e4
11 0x759dfe postgres ExecProcNode + 0x4ae
12 0x74e59c postgres <symbol not found> + 0x74e59c
13 0x74ee38 postgres ExecutorRun + 0x3a8
14 0x9939ba postgres <symbol not found> + 0x9939ba
15 0x996560 postgres PortalRun + 0xb40
16 0x991eeb postgres PostgresMain + 0x375b
17 0x8e9621 postgres <symbol not found> + 0x8e9621
18 0x8ef5d9 postgres PostmasterMain + 0x1209

5. The PXF catalina log in direcotry /var/gphd/pxf/pxf-service/logs on server 10.100.160.84 reports "Read timed out" error message.

18-Dec-2014 12:58:48.312 SEVERE [tomcat-http--19] com.sun.jersey.spi.container.ContainerResponse.mapMappableContainerException The exception contained within MappableContainerException could not be mapped to a response, re-throwing to the HTTP container
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:535)
at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:504)
at org.apache.coyote.http11.InternalInputBuffer$InputStreamInputBuffer.doRead(InternalInputBuffer.java:566)
at org.apache.coyote.http11.filters.ChunkedInputFilter.readBytes(ChunkedInputFilter.java:309)
at org.apache.coyote.http11.filters.ChunkedInputFilter.parseChunkHeader(ChunkedInputFilter.java:339)
at org.apache.coyote.http11.filters.ChunkedInputFilter.doRead(ChunkedInputFilter.java:185)
at org.apache.coyote.http11.AbstractInputBuffer.doRead(AbstractInputBuffer.java:348)
at org.apache.coyote.Request.doRead(Request.java:422)
at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:290)
at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:390)
at org.apache.catalina.connector.InputBuffer.readByte(InputBuffer.java:304)
at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:106)
at java.io.FilterInputStream.read(FilterInputStream.java:83)
at com.pivotal.pxf.service.io.Text.readFields(Text.java:301)
at com.pivotal.pxf.service.BridgeInputBuilder.makeInput(BridgeInputBuilder.java:29)
at com.pivotal.pxf.service.WriteBridge.setNext(WriteBridge.java:52)
at com.pivotal.pxf.service.rest.WritableResource.writeResponse(WritableResource.java:135)
at com.pivotal.pxf.service.rest.WritableResource.stream(WritableResource.java:109)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1070)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)

6. Try INSERT for several times and it fails with same error but from different segment

7. If try the same operation with a child table of dwh.agg_fact_usage, which has about 800,000 records, INSERT completes successfully

Cause

"failed sending to remote component" message in Pivotal HDB segment logs indicates it there is either network communication issue a remote daemon service has crashed. 

"Read timed out" error message in PXF catalina log file means this PXF service daemon fails to read data within specified time and gets timed out. This possibly can be caused by network congestion when there are huge amounts of data to transmit, which is what we saw when inserting from a large table with 180 million records.

Solution

Based on the above analysis, try to increase the value of connectionTimeout parameter in the pxf service configuration for each node running PXF. The default is 20 seconds.  try with a bigger value such as 60 seconds.

The configuration file is /var/gphd/pxf/pxf-service/conf/server.xml. 

<Connector acceptCount="100"
connectionTimeout="60000"
executor="tomcatThreadPool"
maxKeepAliveRequests="15"
maxHeaderCount="30000"
maxHttpHeaderSize="1048576"
port="${bio.http.port}"
protocol="org.apache.coyote.http11.Http11Protocol"
redirectPort="${bio.https.port}"/>

PXF service will then need to be restarted

[root@hdw1 conf]# service pxf-service restart
/var/gphd/pxf /
Instance is running as PID=32224, shutting down...
Instance is running PID=32224, sleeping for up to 5 seconds waiting for shutdown
Instance shut down gracefully
/
/var/gphd/pxf /
Using CATALINA_BASE: /var/gphd/pxf/pxf-service
Using CATALINA_HOME: /opt/vmware/vfabric-tc-server-standard/tomcat-7.0.55.A.RELEASE
Using CATALINA_TMPDIR: /var/gphd/pxf/pxf-service/temp
Using JRE_HOME: /usr/java/default
Using CLASSPATH: /opt/vmware/vfabric-tc-server-standard/tomcat-7.0.55.A.RELEASE/bin/bootstrap.jar:/opt/vmware/vfabric-tc-server-standard/tomcat-7.0.55.A.RELEASE/bin/tomcat-juli.jar
Using CATALINA_PID: /var/gphd/pxf/pxf-service/logs/tcserver.pid
Tomcat started.
Status: RUNNING as PID=662535
/
Waiting for tcServer to start...
tcServer not responding, re-trying after 1 second (attempt number 1)
Checking if PXF webapp is up and running...
PXF webapp is up

Verify After increasing connectionTimeout to 60000 the INSERT operation which failed previously now completes.

Internal JIRA GPSQL-3085.

Comments

Powered by Zendesk