Pivotal Knowledge Base

Follow

HDB query errors with failed to acquire resources on segment 0

Environment

Product Version
Pivotal HDB 1.2.1.2
OS RHEL 6.x

Symptom

When executing HDB query we see the following error haver a few seconds of query execution

gpadmin=# SELECT * from schema.table;
ERROR:  failed to acquire resources on segment 0

Cause

Generally when we see an error like this it has to do with HDB establishing a TCP connection with all the segments so it can begin query execution. This is called the dispatch phase. 

EDT,434811,con1456,cmd7,seg-1,,,x434811,sx1,"LOG","00000","Master unable to connect to entry db hdm2:10432 with options gpqeid=1456;496378700739355 options=' -c gp_segment=-1 -c gp_qd_hostname=3.14.145.7 -c gp_qd_port=10432 -c gp_qd_callback_info=port=5432 -c gp_qd_proc_offset=208577280 -c client_min_messages=notice -c commit_delay=0 -c commit_siblings=5 -c coredump_on_memerror=false -c DateStyle=ISO,MDY -c debug_database_command_error_level=log -c debug_persistent_print_level=debug1 -c debug_persistent_recovery_print_level=debug1 -c debug_persistent_store_print_level=debug1 -c enable_secure_filesystem=true -c explain_memory_verbosity=suppress -c filesystem_support_truncate=true -c force_bitmap_table_scan=false -c gp_blockdirectory_entry_min_range=0 -c gp_blockdirectory_minipage_size=161 -c gp_cancel_query_delay_time=0 -c gp_cancel_query_print_log=false -c gp_connections_per_thread=64 -c gp_crash_handler_async=true -c gp_dbg_flags=0 -c gp_debug_linger=0 -c gp_disable_catalog_access_on_segment=false -c gp_disable_tuple_hints=true -c gp_dump_memory_usage=false -c gp_eager_hashtable_release=true -c gp_enable_mk_sort=true -c gp_enable_motion_mk_sort=true -c gp_force_use_default_temporary_directory=false -c gp_gpperfmon_send_interval=1 -c gp_hashagg_compress_spill_files=none -c gp_hashagg_default_nbatches=32 -c gp_hashagg_groups_per_bucket=5 -c gp_hashjoin_bloomfilter=1 -c gp_hashjoin_metadata_memory_percent=20 -c gp_hashjoin_tuples_per_bucket=5 -c gp_interconnect_default_rtt=20 -c gp_interconnect_elide_setup=true -c gp_interconnect_fc_method=loss -c gp_interconnect_full_crc=false -c gp_interconnect_log_stats=false -c gp_interconnect_min_retries_before_timeout=100 -c gp_interconnect_min_rto=20 -c gp_interconnect_queue_depth=4 -c gp_interconnect_setup_timeout=7200 -c gp_interconnect_snd_queue_depth=2 -c gp_interconnect_timer_checking_period=20 -c gp_interconnect_timer_period=5 -c gp_interconnect_transmit_timeout=3600 -c gp_interconnect_type=udp -c gp_is_callback=false -c gp_log_interconnect=terse -c gp_log_resqueue_memory=false -c gp_log_stack_trace_lines=true",,,,,,,0,,,,
2015-09-24 07:31:40.464663 EDT,"gpadmin","anhffp01",p572824,th-717891360,"[local]",,2015-09-24 07:31:18 EDT,434811,con1456,cmd7,seg-1,,,x434811,sx1,"LOG","00000","Failed connection to entry db hdm2:10432",,,,,,

As you can see in this case this error message has too much information which is not always the case. But at least the message "Failed connection to entry db hdm2:10432" tells us that we were able to make a TCP connection to the master segment which means there are no loopback network issues.  Also we see the connection string options has this param to segment "gp_qd_port=10432"

It looks like master is listing on 5432 and not 10432

[gpadmin@hdm2 ~]$ ps -ef | grep silent
gpadmin  476950      1  0 Sep23 ?        00:00:02 /usr/local/hawq-1.2.1.2/bin/postgres -D /data/hawq/master_new/gpseg-1 -p 5432 -b 1 -z 176 --silent-mode=true -i -M master -C -1 -x 178 -E

[gpadmin@hdm2 ~]$ stat /tmp/.s.PGSQL.10432
stat: cannot stat `/tmp/.s.PGSQL.10432': No such file or directory

Database has 10432 as the port for master in the gp_segment configuration table

gpadmin=# select hostname,port from gp_segment_configuration where content = -1;
hostname | port
---------------------+------
hdm2.gphd.local | 10432
hdm1.gphd.local | 10432

GDB master $MASTER_DATA_DIRECTORY/postgresql.conf is configured for 5432

[gpadmin@hdm3 ~]$ cat $MASTER_DATA_DIRECTORY/postgresql.conf | egrep ^port
port=5432 ##port = 5432				# sets the database listener port for

Fix

Ensure that both $MASTER_DATA_DIRECTORY/postgresql.conf and gp_segment_configuration table have the same port defined for master. This will ensure the dispatcher is able to communicate with the HDB master during query execution

Internal References

See Internal JIRA GPSQL-3219

Comments

Powered by Zendesk