Pivotal Knowledge Base

Follow

Connection to HAWQ Database Error: "could not count blocks of relation 1663/16508/1249: Not a directory"

Environment

 Product  Version
 Pivotal HDB  2.x

Symptom

Connection to a HAWQ database fails with:

[gpadmin@hdm1 16508]$ psql 
psql: FATAL: could not count blocks of relation 1663/16508/1249: Not a directory.

Error Message

Master log may show these errors messages:

2017-02-17 02:12:50.119207 PST,,,p22482,th-1681897184,,,,0,,,seg-10000,,,,,"DEBUG1","00000","opening ""pg_xlog/00000001000000050000001D"" for readin
g (log 5, seg 29)",,,,,,,0,,"xlog.c",3162,
2017-02-17 02:12:50.176450 PST,,,p22482,th-1681897184,,,,0,,,seg-10000,,,,,"FATAL","42809","could not count blocks of relation 1663/16508/1249: Not
a directory",,,,,"xlog redo insert: rel 1663/16508/1249; tid 32682/85
REDO PASS 3 @ 5/7669B838; LSN 5/7669E480: prev 5/76694C98; xid 825193; bkpb1: Heap - insert: rel 1663/16508/1249; tid 32682/85",,0,,"smgr.c",1146,"S
tack trace:
1    0x8c5628 postgres errstart + 0x288
2    0x7ddfbc postgres smgrnblocks + 0x3c
3    0x4fbdf8 postgres XLogReadBuffer + 0x18
4    0x4ea2c9 postgres <symbol not found> + 0x4ea2c9
5    0x4eaf47 postgres <symbol not found> + 0x4eaf47
6    0x4f8af3 postgres StartupXLOG_Pass3 + 0x153
7    0x4fb277 postgres StartupProcessMain + 0x187
8    0x557cd8 postgres AuxiliaryProcessMain + 0x478
9    0x793c40 postgres <symbol not found> + 0x793c40
10   0x798901 postgres <symbol not found> + 0x798901
11   0x79a8c9 postgres PostmasterMain + 0x759
12   0x4a4039 postgres main + 0x519
13   0x7f3b979e1d5d libc.so.6 __libc_start_main + 0xfd
14   0x4a40b9 postgres <symbol not found> + 0x4a40b9
" 

Cause 

This is caused by software defect GPSQL-3350, fixed in HDB 2.1.2.0. This can happen if one of the internal catalog files grows above 1GB, for example, above, the file /data/hawq/master/base/1663/16508/1249 had grown to 1GB exactly because maintenance (vacuum) had not been run periodically on the system tables.

Resolution

To avoid running into this issue, the best practices should be followed to run maintenance on HAWQ system tables.

To resolve this issue, the HAWQ database should be upgraded to HDB 2.1.2.0 or higher. 

Comments

Powered by Zendesk