Pivotal Knowledge Base

Follow

MapReduce job failed with "Unable to initialize any output collector"

Environment

Product Version
 Pivotal HD  

Symptom

MapReduce / Hive query failed with below error:

2015-04-24 11:41:41,861 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.io.IOException: Unable to initialize any output collector
	at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:412)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:439)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)

Cause

The error "Unable to initialize any output collector" indicates that the job failed to start the container's, there can be multiple reasons for the same. However, one must review the container logs at HDFS to identify the cause the error. 

In this specific instance, the value of mapreduce.task.io.sort.mb value was greater than 2047 MB, though the maximum allowed value is 2047 MB. Thus anything above the maximum value causes the jobs to fail marking the value provided as Invalid.

The size of Container logs revealed the below error:

2015-04-24 11:41:41,858 WARN [main] org.apache.hadoop.mapred.MapTask: Unable to initialize MapOutputCollector org.apache.hadoop.mapred.MapTask$MapOutputBuffer
java.io.IOException: Invalid "mapreduce.task.io.sort.mb": 2048
	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.init(MapTask.java:975)
	at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:401)

Resolution

Set the value of mapreduce.task.io.sort.mb less than 2048MB.

Comments

Powered by Zendesk