Pivotal Knowledge Base


MapReduce job failed with "Unable to initialize any output collector"


Product Version
 Pivotal HD  


MapReduce / Hive query failed with below error:

2015-04-24 11:41:41,861 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.io.IOException: Unable to initialize any output collector
	at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:412)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:439)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)


The error "Unable to initialize any output collector" indicates that the job failed to start the container's, there can be multiple reasons for the same. However, one must review the container logs at HDFS to identify the cause the error. 

In this specific instance, the value of mapreduce.task.io.sort.mb value was greater than 2047 MB, though the maximum allowed value is 2047 MB. Thus anything above the maximum value causes the jobs to fail marking the value provided as Invalid.

The size of Container logs revealed the below error:

2015-04-24 11:41:41,858 WARN [main] org.apache.hadoop.mapred.MapTask: Unable to initialize MapOutputCollector org.apache.hadoop.mapred.MapTask$MapOutputBuffer
java.io.IOException: Invalid "mapreduce.task.io.sort.mb": 2048
	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.init(MapTask.java:975)
	at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:401)


Set the value of mapreduce.task.io.sort.mb less than 2048MB.


Powered by Zendesk