Pivotal Knowledge Base

Follow

Nodemanager fails to start with container-executor java.io.IOException: error=13, Permission denied

Symptom:

Yarn nodemanager may fail to start in a secure cluster with the an error message: container-executor: java.io.IOException: error=13, Permission denied

Snippet for the error message from yarn-nodemanager logs.

2014-03-28 20:52:22,266 INFO org.apache.hadoop.yarn.server.nodemanager.NodeManager: Security is enabled on NodeManager. Creating ContainerTokenSecretManager
2014-03-28 20:52:22,732 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: setsid exited with exit code 0
2014-03-28 20:52:22,736 FATAL org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting NodeManager
org.apache.hadoop.yarn.YarnException: Failed to initialize container executor
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.init(NodeManager.java:144)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:321)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:359)
Caused by: java.io.IOException: Cannot run program "/usr/lib/gphd/hadoop-yarn/bin/container-executor": java.io.IOException: error=13, Permission denied
at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
... 2 more
Caused by: java.io.IOException: java.io.IOException: error=13, Permission denied
at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
... 7 more
2014-03-28 21:14:27,520 INFO org.apache.hadoop.yarn.server.nodemanager.NodeManager: SHUTDOWN_MSG:

Cause:

container-executor.cfg is configured with the below permissions, where yarn is the group name to which this file is associated. Thus, yarn user must be associated with yarn group so as to execute the container-executor program, else it will bail out due to permission issues.

[root@phd11-nn ~]# ls -tlr /usr/lib/gphd/hadoop-yarn/bin/container-executor
---Sr-s--- 1 root yarn 34200 Nov 12 18:53 /usr/lib/gphd/hadoop-yarn/bin/container-executor

In this case, yarn user was not associated with yarn group, but both were available.

root@tstr400059:$ id yarn 
uid=99523(yarn) gid=5910(hadoop) groups=5910(hadoop) 

Fix:

Associate yarn user with the yarn group available in the system and try starting node manager again.

root@tstr400059:$ usermod -G yarn yarn
root@tstr400059:$ id yarn 
uid=99523(yarn) gid=5910(hadoop) groups=5910(hadoop),497(yarn) root@tstr400059:$ service hadoop-yarn-nodemanager start starting nodemanager, logging to /var/log/gphd/hadoop-yarn/yarn-yarn-nodemanager-tstr400059.out                                        [ OK ]

Comments

  • Avatar
    Mary Rogers
    I am getting this response from service hadoop-yarn-nodemanager start. I was able to get id yarn to change using the user mod -G yarn yarn Redirecting to /bin/systemctl start hadoop-yarn-nodemanager.service Failed to start hadoop-yarn-nodemanager.service: Unit hadoop-yarn-nodemanager.service failed to load: No such file or directory.
Powered by Zendesk