Problem running tasktracker in hadoop under windows

Question

I am trying to use hadoop under windows and I have a problem when I want to run tasktracker. For example:

$bin/start-all.sh 

then the logs write:

 2011-06-08 16:32:18,157 ERROR org.apache.hadoop.mapred.TaskTracker: Can not start task tracker because java.io.IOException: Failed to set permissions of path: /tmp/hadoop-Administrator/mapred/local/taskTracker to 0755 at org.apache.hadoop.fs.RawLocalFileSystem.checkReturnValue(RawLocalFileSystem.java:525) at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:507) at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:318) at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:183) at org.apache.hadoop.mapred.TaskTracker.initialize(TaskTracker.java:630) at org.apache.hadoop.mapred.TaskTracker.<init>(TaskTracker.java:1328) at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3430) 

What is the problem? How can i solve this? Thanks!

Answer 1

I encountered this problem when installing 1.0.3 on a Windows server. I changed the default directory in hdfs-site.xml so that the directory that hasoop creates for dfs is a cygwin directory subdirectory like this ...

...

  <property> <name>dfs.name.dir</name> <value>c:/cygwin/usr/mydir/dfs/logs</value> </property> <property> <name>dfs.data.dir</name> <value>c:/cygwin/usr/mydir/dfs/data</value> </property> </configuration> 

This seemed to solve the problem.

The apache documentation for configuration files is here

Answer 2

This issue is tracked at https://issues.apache.org/jira/browse/HADOOP-7682

Answer 3

Use this owner to change the hadoop-adminminrator folder. You can use the chown command to do this.

Answer 4

This issue was raised on the Apache Hadoop user mailing list. This seems to be a problem in some versions of Hadoop and not in others.

A simple solution is to download another version of Hadoop (assuming you do not need a specific version of Hadoop for some other reason).

I ran into this exact issue with version 1.0.0 (beta).

Then I tried 0.23.0, but got a fatal ClassNotFoundException:

 log4j:ERROR Could not find value for key log4j.appender.NullAppender log4j:ERROR Could not instantiate appender named "NullAppender". Exception in thread "main" java.lang.ClassNotFoundException: hadoop-mapreduce-examples-0.23.0.jar at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:423) at java.lang.ClassLoader.loadClass(ClassLoader.java:356) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.util.RunJar.main(RunJar.java:182) 

Finally, I tried version 0.22.0 and worked without errors. Therefore, I recommend trying to download and install version 0.22.0: http://hadoop.apache.org/common/releases.html#10+December%2C+2011%3A+release+0.22.0+available

Answer 5

The issue seems to be related to permissions on the path /tmp/hadoop-Administrator/mapred/local/taskTracker
as indicated by the error message

 ERROR org.apache.hadoop.mapred.TaskTracker: Can not start task tracker because java.io.IOException: Failed to set permissions of path: /tmp/hadoop-Administrator/mapred/local/taskTracker 

The taskTracker account is started as required by the chmod of the specified folder. This may require more control, such as being the owner, for other aspects. I do not remember the specific permissions required for the components in the hadoop configuration.

I did not understand the aspect of setting the Hadoop resolution, especially in windows (in general), so I say that this is mainly due to the error message. I also have no shortage of permission for the cygwin folder, so I don’t know how to fix it, but hopefully it will force you to point in the right direction.