Pivotal Knowledge Base

Follow

When running hadoop list command, files in local filesystem are returned

Problem:

While running hadoop fs commands, for example: hadoop dfs -ls, output shows the list of files on the local unix / linux filesystem and not on hdfs filesystem.

[root@phd11-dn-1-ha ~]# sudo -u hdfs hdfs dfs -ls /
Found 29 items
-rw-r--r-- 1 root root 0 2013-11-26 23:14 /.autofsck
-rw-r--r-- 1 root root 0 2013-07-18 22:43 /.autorelabel
dr-xr-xr-x - root root 4096 2013-07-19 22:01 /bin
drwxrwxrwx - root root 4096 2013-11-27 00:51 /binary
dr-xr-xr-x - root root 1024 2013-07-18 22:35 /boot
drwxrwxrwx - root root 4096 2014-01-11 00:55 /data
drwxrwxrwx - root root 4096 2014-01-09 23:53 /data1
drwxr-xr-x - root root 3640 2013-11-26 23:15 /dev
-rw-r--r-- 1 root root 72700 2013-11-03 22:18 /dmidecode-2.11-2.el6.x86_64.rpm

Solution:

In majority of the case, it could happen when core-site.xml does not have the fs.defaultFS parameter specificied correctly.

Identify the address on which hdfs is running and update it in core-site.xml

Ex: In case of PHD HA configuration, it would look like below where phdha is the name of nameservice

<property>
<name>fs.defaultFS</name>
<value>hdfs://phdha</value>
</property>

After adding the required value, test the command again, it should show expected results.

[root@phd11-dn-1-ha ~]# sudo -u hdfs hdfs dfs -ls /
Found 7 items
drwxr-xr-x - hdfs hadoop 0 2014-01-11 00:19 /apps
drwxr-xr-x - gpadmin hadoop 0 2014-01-11 16:30 /hawq_data
drwxr-xr-x - hdfs hadoop 0 2014-01-11 00:20 /hive
drwxr-xr-x - mapred hadoop 0 2014-01-11 00:18 /mapred

Comments

Powered by Zendesk