Pivotal Knowledge Base

Follow

Map Reduce jobs on a secured cluster fails to create app directories.

Symptom:

After securing a cluster, if the permission on the cluster directories are not setup properly, Map Reduce jobs can fail with an error like below:

main : user is gpadmin
Can't open / in /data1/yarn/nm-local-dir/usercache/gpadmin/appcache/application_1397684926758_0002 - Permission denied
Can't open / in /data2/yarn/nm-local-dir/usercache/gpadmin/appcache/application_1397684926758_0002 - Permission denied
Can't open / in /data3/yarn/nm-local-dir/usercache/gpadmin/appcache/application_1397684926758_0002 - Permission denied
Can't open / in /data4/yarn/nm-local-dir/usercache/gpadmin/appcache/application_1397684926758_0002 - Permission denied
Can't open / in /data5/yarn/nm-local-dir/usercache/gpadmin/appcache/application_1397684926758_0002 - Permission denied
Did not create any app directories 

Cause:

The above error indicates that the application failed to open "/" directory while trying to write application logs on local filesystem. It is a symptom, so depending on the directory on which MapReduce jobs failed to read, you can verify the permission and ensure that all the users have read & write permission appropriately.

In this particular case, "/" directory did not had permission like below, after changing it to 755 MapReduce jobs were able to create the directories successfully.

gpadmin:Fullrack@mdw $ ls -tlrd /
drwx--x--x 41 root root 4096 Apr 10 19:19 /

Container executor program ensures that the path specified by yarn.nodemanager.local-dirs in yarn-site.xml must have at-least readonly permission on all of the parent directories starting from "/". 

Snippet from container-executor.c indicating check for / directory. This snippet further extends to check for all the child directories as well.

int mkdirs(const char* path, mode_t perm) {
  char *buffer = strdup(path);
  char *token;
  int cwd = open("/", O_RDONLY);
  if (cwd == -1) {
    fprintf(LOGFILE, "Can't open / in %s - %s\n", path, strerror(errno));
    free(buffer);
    return -1;
  }

Fix:

Provide read / write privileges to the directory displayed in the error message

gpadmin:Fullrack@mdw $ ls -tlrd /
drwxr-xr-x 41 root root 4096 Apr 10 21:19 /

 

Comments

Powered by Zendesk