Pivotal Knowledge Base

Follow

Gpfdist errors out http response code 500 no complete data row found for writing

Symptom

Try to insert into one writable table , but sometimes it hung there and get this kind of error message in master logs  "ERROR","XX000","http response code 500 from gpfdist"

 

2013-11-14 08:45:54.166234 EST,"gpadmin","comscore",p20439,th-1202217568,"172.16.103.50","55544",2013-11-14 08:45:47 EST,5709150,con27,cmd28,seg487,,dx28,x5709150,sx1,"ERROR","XX000","http response code 500 from gpfdist (gpfdist://smdw-1:8908/kpathak/720w-sdk-analysis.txt): HTTP/1.0 200 ok (url.c:366)",,,,,,"INSERT INTO wext_gpfdist_write_test
select
'30E724' || substring(md5(random()::text), 0, 11) as ns_vid
, '1382270400000' as ns_utc
, 'SDK_Analysis' as source_file
, *
from analytics.jjoyner_sdk_agg_720w;",0,,"url.c",366,"Stack trace:
1 0xa6f7e9 postgres <symbol not found> + 0xa6f7e9
2 0xa718a8 postgres elog_finish + 0xb8
3 0x540adb postgres url_fwrite + 0x49b
4 0x537c01 postgres external_insert + 0x1a1
5 0x6efb7c postgres <symbol not found> + 0x6efb7c
6 0x6f01b0 postgres ExecutorRun + 0x340
7 0x91451a postgres <symbol not found> + 0x91451a
8 0x916ef6 postgres PortalRun + 0xa86
9 0x90f839 postgres PostgresMain + 0x2049
10 0x875ac1 postgres <symbol not found> + 0x875ac1
11 0x87bc00 postgres PostmasterMain + 0x12f0
12 0x780ada postgres main + 0x4da
13 0x3c95a1d994 libc.so.6 __libc_start_main + 0xf4
14 0x47cc19 postgres <symbol not found> + 0x47cc19
"
[...]

 

gpfdist logs like below:

 

[2013-11-14 07:00:11] ::ffff:172.16.102.246 POST /jjoyner/720w-sdk-analysis.txt - Continue
[2013-11-14 07:00:11] available data to consume 65469, starting at offset 304
[2013-11-14 07:00:11] wrote 32683 bytes to file
[2013-11-14 07:00:11] wrote 32679 bytes to file
[2013-11-14 07:00:11] wrote 107 bytes to file
[2013-11-14 07:00:11] ::ffff:172.16.102.246 POST /jjoyner/720w-sdk-analysis.txt - OK
[2013-11-14 07:00:11] ::ffff:172.16.102.156 requests /jjoyner/720w-sdk-analysis.txt
[2013-11-14 07:00:11] joined session (/data/data_dump/jjoyner/720w-sdk-analysis.txt, 1384424584-0000013980.3.0.0)
[2013-11-14 07:00:11] active segids in session: 1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70 73 76 79 82 85 88 91 94 97 100 103 106 109 112 115 118 121 124 127 130 133 136 139 142 145 148 151 154 157 160 163 166 169 172 175 178 181 184 187 190 193 196 199 202 205 208 211 214 217 220 223 226 229 232 235 238 241 244 247 250 253 256 259 262 265 268 271 274 277 280 283 286 289 292 295 298 301 304 307 310 313 316 319 322 325 328 331 334 337 340 343 346 349 352 355 358 361 364 367 370 373 376 379 382 385 388 391 394 397 400 403 406 409 412 415 418 421 424 427 430 433 436 439 442 445 448 451 454 457 460 463 466 469 472 475 478 481 484 487 490 493 496 499 502 505 508 511 514 517 520 523 526 529 532 535 538 541 544 547 550 553 556 559 562 565 568 571 574 577 580 583 586 589 592 595 598 601 604 607 610 613 616 619 622 625 628 631 634 637 640 643 646 649 652 655 658 661 664 667 670 673 676 679 682 685 688 691 694 697 700 703 706 709 712 715 718 721 724 727 730 733 736 739 742 745 748 751 754 757 760 763 766 769 772 775 778 781 784 787 790 793
[2013-11-14 07:00:11] ::ffff:172.16.102.156 POST /jjoyner/720w-sdk-analysis.txt - Continue
[2013-11-14 07:00:11] available data to consume 65491, starting at offset 304
[2013-11-14 07:00:11] wrote -1 bytes to file
[2013-11-14 07:00:11] ::ffff:172.16.102.156 - 500 no complete data row found for writing
[2013-11-14 07:00:11] ::ffff:172.16.102.99 requests /jjoyner/720w-sdk-analysis.txt
[2013-11-14 07:00:11] joined session (/data/data_dump/jjoyner/720w-sdk-analysis.txt, 1384424584-0000013980.3.0.0)
[2013-11-14 07:00:11] active segids in session: 1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70 73 76 79 82 85 88 91 94 97 100 103 106 109 112 115 118 121 124 127 130 133 136 139 142 145 148 151 154 157 160 163 166 169 172 175 178 181 184 187 190 193 196 199 202 205 208 211 214 217 220 223 226 229 232 235 238 241 244 247 250 253 256 259 262 265 268 271 274 277 280 283 286 289 292 295 298 301 304 307 310 313 316 319 322 325 328 331 334 337 340 343 346 349 352 355 358 361 364 367 370 373 376 379 382 385 388 391 394 397 400 403 406 409 412 415 418 421 424 427 430 433 436 439 442 445 448 451 454 457 460 463 466 469 472 475 478 481 484 487 490 493 496 499 502 505 508 511 514 517 520 523 526 529 532 535 538 541 544 547 550 553 556 559 562 565 568 571 574 577 580 583 586 589 592 595 598 601 604 607 610 613 616 619 622 625 628 631 634 637 640 643 646 649 652 655 658 661 664 667 670 673 676 679 682 685 688 691 694 697 700 703 706 709 712 715 718 721 724 727 730 733 736 739 742 745 748 751 754 757 760 763 766 769 772 775 778 781 784 787 790 793
[2013-11-14 07:00:11] ::ffff:172.16.102.99 POST /jjoyner/720w-sdk-analysis.txt - Continue
[2013-11-14 07:00:11] available data to consume 65518, starting at offset 304
[2013-11-14 07:00:11] wrote -1 bytes to file
[2013-11-14 07:00:11] ::ffff:172.16.102.99 - 500 no complete data row found for writing

Cause

Root cause gpfdist default row limit is 32k as documented, if one row is longer than 32k , it will errors out like above.

Resolution
Use -m option with valid range 32K to 256MB when stratup gpfdist like below and this issue has been fixed in 4.2.7.0 also.

 

gpfdist -t 600 -d /data/data_dump/ -l /data/data_dump/logs/gpfdist_Verbose.8911.m.log -p 8911 -V -m 655350 &

 

Comments

Powered by Zendesk