Pivotal Knowledge Base

Follow

"DataXceiverServer: java.io.IOException: Xceiver count 10001 exceeds the limit of concurrent xcievers: 10000" seen in Pivotal HD

Environment

Product Version
Pivotal HD  2.x / 3.0.x  
Pivotal Hortonworks Data Platform (HDP)  2.4

Symptom

The datanode's log shows the following warning message:

Error Message:

2016-08-08 10:04:19,509 WARN datanode.DataNode (DataXceiverServer.java:run(158)) - dn00.localdomain.com:50010:DataXceiverServer: 
java.io.IOException: Xceiver count 10001 exceeds the limit of concurrent xcievers: 10000

Cause 

The current value of the dfs.datanode.max.transfer.threads is not enough for the number of simultaneous I/Os that the datanode is doing. The param dfs.datanode.max.transfer.threads specifies the maximum number of threads to use for transferring data in and out of the datanode.

Resolution

Increase the value of dfs.datanode.max.transfer.threads to a higher value. In this specific case, the value was increased from 10,000 to 16,384 and resolved the issue. For clusters with high concurrent I/Os, you may try an even higher value, like 40,960. Do this using the Ambari interface for consistent changes across the cluster (Ambari WebUI > HDFS > Configs > Advancedhdfs-site >dfs.datanode.max.transfer.threads). Once the change is done, restart the Hadoop Distributed File System (HDFS) service or complete a rolling restart of all datanodes.

Additional Information 

Higher values of  dfs.datanode.max.transfer.threads require a higher value of maximum open file limits on the OS side. Please make sure you also have enough of an open file limit using the "ulimit -" command. The open file limit should be greater than the value of dfs.datanode.max.transfer.threads.

 

 

Comments

Powered by Zendesk