Pivotal Knowledge Base

Follow

Segment Allocation Process Error: "Too many Connections for Database <database_name>"

Environment

 Product  Version
 Pivotal Greenplum  4.2.x
 OS  All Supported OS

Symptom

Running a query on Pivotal Greenplum fails and returns the following error message:

Error: Unexpected internal error (cdbgang.c:1447) 

The master log for the database during that period of time indicates that the error was generated during the allocation of the gang (as per the stack trace below).

2014-09-15 10:24:05.192697 IST,"xxxx_user","xxxdatabase",p14391,th1003468544,"10.135.9.17","29382",2014-09-15 10:24:0
3 IST,12646033,con82704,cmd4,seg-1,,dx205280,x12646033,sx1,"ERROR","XX000","Unexpected internal error (cdbgang.c:1447)
",,,,,,"--SO1_Queries
.........
..........
        select  XXXXXXXXXXXXXXXXX from XXXXXXXXXXX
 ......
 ......
         )
*/",0,,"cdbgang.c",1447,"Stack trace:
1    0xa6fdf9 postgres  (elog.c:468)
2    0xa74202 postgres elog_internalerror (elog.c:279)
3    0xb9bb4a postgres allocateGang (cdbgang.c:1519)
4    0x705c6d postgres AssignGangs (execUtils.c:1691)
5    0x6ebceb postgres ExecutorStart (execMain.c:549)
6    0x915ff9 postgres PortalStart (pquery.c:873)
7    0x90c704 postgres  (postgres.c:2451)
8    0x91067d postgres PostgresMain (postgres.c:4928)
9    0x876181 postgres  (postmaster.c:6801)
10   0x87c2c0 postgres PostmasterMain (postmaster.c:2346)
11   0x7811ba postgres main (main.c:212)
12   0x343ee1ecdd libc.so.6 __libc_start_main (??:0)
13   0x47cae9 postgres  (??:0)
"

Picking/Checking any one of the segment logs indicates the below message during that period of time:

2014-09-15 10:24:04.858893 IST,"xxxx_user","xxxdatabase",p22043,th1196578560,"176.16.1.10","5919",2014-09-15 10:24:04 IST,8751573,con82704,,seg5,,,x8751573,sx1,"FATAL","53300","too many connections for database ""xxxdatabase""",,,,,,,0,,"postinit.c",238,
2014-09-15 10:24:04.964784 IST,"xxxx_user","xxxdatabase",p22067,th1196578560,"176.16.1.10","5973",2014-09-15 10:24:04 IST,8751575,con82704,,seg5,,,x8751575,sx1,"FATAL","53300","too many connections for database ""xxxdatabase""",,,,,,,0,,"postinit.c",238,

Cause

The issue is caused due to the limit of the connection that can be connected to the database. Querying the pg_database reveals that the database was limited to 50, so when there are more than 50 connections querying the database, it was unable to allocate gangs to address the new queries.

test=# select * from pg_database where datname='xxxdatabase';
   datname   | datdba | encoding | datistemplate | datallowconn | datconnlimit | datlastsysoid | datfrozenxid | dattablespace | datconfig |
        datacl
-------------+--------+----------+---------------+--------------+--------------+---------------+--------------+---------------+-----------+-----------------------------
----------------------------------------------------
 xxxdatabase     |     10 |        6 | f             | t            |           50 |         10899 |          803 |          1663 |           | {=Tc/gpadmin,gpadmin=CTc/gpa
dmin,xxxxdba=CTc/gpadmin,xxxx_user=CTc/gpadmin}
Time: 2.371 ms

Resolution

Increase the limit of the database connection:

alter database <database_name> with connection limit <value>;

OR

Set the value to unlimited using:

alter database <database_name> with connection limit -1;

Comments

Powered by Zendesk