Pivotal Knowledge Base

Follow

Hive query fails with error message "ArrayIndexOutOfBoundsException"

Environment

Product Version
Pivotal HD/(Hortonworks Data Platform) HDP 3.x / 2.2 / 2.3 / 2.4
OS All supported

Symptom 

Hive queries with Tez fail with the following error message:

Error Message:

Vertex failed, vertexName=Map 1, vertexId=vertex_1465366512401_87702_1_01, diagnostics=[Task failed, taskId=task_1465366512401_87702_1_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"itemnum":"11038949","location":"4000","conditioncode":"B","condrate":75.0,"itemsetid":"WMS_IT05","siteid":"FVN-NU62","stdcost":0,"avgcost":0,"lastcost":0,"glaccount":null,"controlacc":"GBGN.0000000000.CTR.0000000000","shrinkageacc":"GBGN.0000000000.CTR.0000000000","invcostadjacc":"GBGN.0000000000.CTR.0000000000","orgid":"UKS","invcostid":1.06719823E8,"rowstamp":"623292900","change_date":623292900,"load_ts":"2016-02-03 11:04:01","row__id":{"transactionid":640,"bucketid":2,"rowid":13770426}}
        at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:186)
        at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:138)
        at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:324)
        at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:176)
        at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
        at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168)
        at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:163)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"itemnum":"11038949","location":"4000","conditioncode":"B","condrate":75.0,"itemsetid":"WMS_IT05","siteid":"FVN-NU62","stdcost":0,"avgcost":0,"lastcost":0,"glaccount":null,"controlacc":"GBGN.0000000000.CTR.0000000000","shrinkageacc":"GBGN.0000000000.CTR.0000000000","invcostadjacc":"GBGN.0000000000.CTR.0000000000","orgid":"UKS","invcostid":1.06719823E8,"rowstamp":"623292900","change_date":623292900,"load_ts":"2016-02-03 11:04:01","row__id":{"transactionid":640,"bucketid":2,"rowid":13770426}}
        at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:91)
        at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68)
        at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:294)
        at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:163)
        ... 13 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"itemnum":"11038949","location":"4000","conditioncode":"B","condrate":75.0,"itemsetid":"WMS_IT05","siteid":"FVN-NU62","stdcost":0,"avgcost":0,"lastcost":0,"glaccount":null,"controlacc":"GBGN.0000000000.CTR.0000000000","shrinkageacc":"GBGN.0000000000.CTR.0000000000","invcostadjacc":"GBGN.0000000000.CTR.0000000000","orgid":"UKS","invcostid":1.06719823E8,"rowstamp":"623292900","change_date":623292900,"load_ts":"2016-02-03 11:04:01","row__id":{"transactionid":640,"bucketid":2,"rowid":13770426}}
        at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:503)
        at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:83)
        ... 16 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ArrayIndexOutOfBoundsException
        at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:397)
        at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
        at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95)
        at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:157)
        at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:493)
        ... 17 more
Caused by: java.lang.ArrayIndexOutOfBoundsException
        at org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter$Buffer.write(DefaultSorter.java:582)
        at java.io.DataOutputStream.write(DataOutputStream.java:107)
        at org.apache.tez.runtime.library.common.serializer.TezBytesWritableSerialization$TezBytesWritableSerializer.serialize(TezBytesWritableSerialization.java:123)
        at org.apache.tez.runtime.library.common.serializer.TezBytesWritableSerialization$TezBytesWritableSerializer.serialize(TezBytesWritableSerialization.java:110)
        at org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter.collect(DefaultSorter.java:275)
        at org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter.write(DefaultSorter.java:185)
        at org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput$1.write(OrderedPartitionedKVOutput.java:127)
        at org.apache.hadoop.hive.ql.exec.tez.TezProcessor$TezKVOutputCollector.collect(TezProcessor.java:211)
        at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.collect(ReduceSinkOperator.java:534)
        at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:380)
        ... 21 more

Cause 

This error is caused by a known bug reported in Apache Tez: TEZ-2732. This happens when "tez.runtime.io.sort.mb" is set to 2047 (max value allowed).

Resolution

The workaround is to set tez.runtime.io.sort.mb to something lower. The recommendation is 1800.

This change should be done through Ambari in Tez service.

Additional Information

For further information, please access the following resources: 

 

Comments

Powered by Zendesk