Pivotal Knowledge Base

Follow

How to Identify Out of Memory (OOM) Errors

Environment

Product Version
GPDB All Versions

Overview

Several reasons contribute to a GPDB process that generates an out of memory (OOM) error. GPDB attempts to provide meaningful error codes and messages that detail why an action resulted in the OOM. These errors fall into two categories and are identified by the associated error codes. Within the two categories a number of reasons for the generated error.

The following are examples of possible OOM errors. Note that the error codes are 53400 and 53200.

  • "ERROR","53400","Out of memory (seg13 slice13 sdw1-1:40001 pid=10183)","VM Protect failed to allocate 8388608 bytes, 6 MB available"

  • "ERROR","53200","Out of memory. Failed on request of size 156 bytes. (context 'CacheMemoryContext') (aset.c:840)"

Memory Error Code 53400

These errors correspond to the query being terminated due to an issue allocating memory under the gp_vmem_protect_limit.

  1. "VM Protect failed to allocate %d bytes,%d MB available": This happens when the request results in exceeding the gp_vmem_protect_limit.

  2. "Per-query VM protect limit reached: current limit is %d kB, requested %d bytes, available %d MB": As shown above the following limit exceeded: gp_vmem_protect_limit_per_query.

  3. "VM protect failed to allocate %d bytes from system, VM Protect %d MB available": This is not a result of reaching the gp_vmem_protect_limit rather, the error is due to the OS not having enough user memory available.

  4. "Failed to allocate memory under virtual memory protection" "Error %d, errno %d, %s": This error is due to a memory allocation problem that is not covered by the three cases above.

Resolution

Possible solutions to Error Code 53400

  1. The most common cause for this error is due to:
    a) Misconfigured resource queues leading to concurrency being too high or memory oversubscribed.
    OR
    b) A Query to too large of a plan is consuming too much memory. For the first case please review the resource queue settings. For the large plan size issue please consider implementing the GUC gp_max_plan_size.

    Refer to the following documentation: http://gpdb.docs.pivotal.io/4360/guc_config-gp_max_plan_size.html

    NOTE: Letter B only occurs if you have set the GUC gp_vmem_limit_per_query. Again, likely causes for reaching this value are due to overly large query plans.

  2. If the OS is reporting OOM then either: 
    a) A misconfigured our servers to oversubscribe to memory (i.e. gp_vmem_protect_limit is too high for the number of segments)
    OR
    b) Another process is consuming the system memory. Consult with Pivotal Support if you have trouble finding the cause.

    NOTE: If you encounter #2, please open a support ticket.

Error Code 53200

This error covers a large number of possible error messages. They all relate to the postmaster process not being able to allocate memory directly from the OS. The exact cause will depend on the situation and the error message provided.

Additional Information

If you need further assistance, contact Pivotal Support.

Comments

  • Avatar
    Deepesh Chaudhary

    Archana, Could you update this article with other error we have identified and are in later versions.

  • Avatar
    Archana Sundararam

    Article updated as per Deepesh's recommendations.

Powered by Zendesk