|Pivotal HD||2.x / 3.0.x|
|Pivotal Hortonworks Data Platform (HDP)||2.4|
The datanode's log shows the following warning message:
2016-08-08 10:04:19,509 WARN datanode.DataNode (DataXceiverServer.java:run(158)) - dn00.localdomain.com:50010:DataXceiverServer:
java.io.IOException: Xceiver count 10001 exceeds the limit of concurrent xcievers: 10000
The current value of the dfs.datanode.max.transfer.threads is not enough for the number of simultaneous I/Os that the datanode is doing. The param dfs.datanode.max.transfer.threads specifies the maximum number of threads to use for transferring data in and out of the datanode.
Increase the value of dfs.datanode.max.transfer.threads to a higher value. In this specific case, the value was increased from 10,000 to 16,384 and resolved the issue. For clusters with high concurrent I/Os, you may try an even higher value, like 40,960. Do this using the Ambari interface for consistent changes across the cluster (Ambari WebUI > HDFS > Configs > Advancedhdfs-site >dfs.datanode.max.transfer.threads). Once the change is done, restart the Hadoop Distributed File System (HDFS) service or complete a rolling restart of all datanodes.
Higher values of dfs.datanode.max.transfer.threads require a higher value of maximum open file limits on the OS side. Please make sure you also have enough of an open file limit using the "ulimit -" command. The open file limit should be greater than the value of dfs.datanode.max.transfer.threads.