Several reasons contribute to a GPDB process that generates an out of memory (OOM) error. GPDB attempts to provide meaningful error codes and messages that detail why an action resulted in the OOM. These errors fall into two categories and are identified by the associated error codes. Within the two categories a number of reasons for the generated error.
The following are examples of possible OOM errors. Note that the error codes are
"ERROR","53400","Out of memory (seg13 slice13 sdw1-1:40001 pid=10183)","VM Protect failed to allocate 8388608 bytes, 6 MB available"
"ERROR","53200","Out of memory. Failed on request of size 156 bytes. (context 'CacheMemoryContext') (aset.c:840)"
Memory Error Code 53400
These errors correspond to the query being terminated due to an issue allocating memory under the
"VM Protect failed to allocate %d bytes,%d MB available": This happens when the request results in exceeding the
"Per-query VM protect limit reached: current limit is %d kB, requested %d bytes, available %d MB": As shown above the following limit exceeded:
"VM protect failed to allocate %d bytes from system, VM Protect %d MB available": This is not a result of reaching the
gp_vmem_protect_limitrather, the error is due to the OS not having enough user memory available.
"Failed to allocate memory under virtual memory protection" "Error %d, errno %d, %s": This error is due to a memory allocation problem that is not covered by the three cases above.
Possible solutions to Error Code 53400
- The most common cause for this error is due to:
a) Misconfigured resource queues leading to concurrency being too high or memory oversubscribed.
b) A Query to too large of a plan is consuming too much memory. For the first case please review the resource queue settings. For the large plan size issue please consider implementing the GUC
Refer to the following documentation: http://gpdb.docs.pivotal.io/4360/guc_config-gp_max_plan_size.html
NOTE: Letter B only occurs if you have set the GUC
gp_vmem_limit_per_query. Again, likely causes for reaching this value are due to overly large query plans.
- If the OS is reporting OOM then either:
a) A misconfigured our servers to oversubscribe to memory (i.e.
gp_vmem_protect_limitis too high for the number of segments)
b) Another process is consuming the system memory. Consult with Pivotal Support if you have trouble finding the cause.
NOTE: If you encounter #2, please open a support ticket.
Error Code 53200
This error covers a large number of possible error messages. They all relate to the postmaster process not being able to allocate memory directly from the OS. The exact cause will depend on the situation and the error message provided.
If you need further assistance, contact Pivotal Support.