Pivotal Knowledge Base

Follow

Hive Runtime Error: Map local work exhausted memory

Environment

Product Version
PHD 2.x
PHD 3.0
Others  

Problem

When executing hive query one of the following Out of Memory errors erros is observed

ERROR mr.MapredLocalTask (MapredLocalTask.java:executeFromChildJVM(323)) – Hive Runtime Error: Map local work exhausted memory org.apache.hadoop.hive.ql.exec.mapjoin.MapJoinMemoryExhaustionException: 2013-11-20 07:30:48 Processing rows: 64000000 Hashtable size: 1589999 Memory usage: 9892387468 percentage: 0.92
at org.apache.hadoop.hive.ql.exec.mapjoin.MapJoinMemoryExhaustionHandler.checkMemoryStatus(MapJoinMemoryExhaustionHandler.java:91)
759962 [main] ERROR org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask  - Hive Runtime Error: Map local work exhausted memory
java.lang.OutOfMemoryError: GC overhead limit exceeded
        at org.apache.hadoop.hive.serde2.lazy.objectinspector.primitive.LazyStringObjectInspector.copyObject(LazyStringObjectInspector.java:46)
        at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.copyToStandardObject(ObjectInspectorUtils.java:305)
        at org.apache.hadoop.hive.ql.exec.JoinUtil.computeMapJoinValues(JoinUtil.java:171)
        at org.apache.hadoop.hive.ql.exec.HashTableSinkOperator.processOp(HashTableSinkOperator.java:236)
        at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
        at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
        at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
        at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95)
        at org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.startForward(MapredLocalTask.java:404)
        at org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.startForward(MapredLocalTask.java:375)
        at org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.executeInProcess(MapredLocalTask.java:341)
        at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:744)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

 

Workaround

Mapred will exhaust if there are millions of rows to join. Set the below property to "false" to disable local inmemory joins. This will force the join to be distributed Map-Reduce phase. Once the job completes you need to set back to the original value "true". 

set hive.auto.convert.join=false;
set hive.auto.convert.join=true;

Also please ensure hive partition purning bug HIVE-2677 does not apply to your case. If it does then the following workaround can be used. Note:hive.auto.convert.join.noconditionaltask.size may need to be increased or decreased depending on use case

SET hive.auto.convert.join.noconditionaltask=true;
SET hive.auto.convert.join.noconditionaltask.size=67108864;

 

Comments

Powered by Zendesk