How to suppress the Hadoop protocol message on the console

Question

This is the Hadoop protocol message I was trying to compress

11/10/17 19:42:23 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584) 11/10/17 19:42:23 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100 11/10/17 19:42:23 INFO mapred.MapTask: soft limit at 83886080 11/10/17 19:42:23 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600 11/10/17 19:42:23 INFO mapred.MapTask: kvstart = 26214396; length = 6553600 

I suppose they are configured using logg 4j.properties in the conf directory in the hadoop installation folder. No matter how I delete the logger, comment exit the logger and even rename log4j.properties. They are still being printed. Please advise.

I also suspect that these bulk messages will affect overall performance if the volume is large. Thanks,

Answer 1

You should increase the log level to WARN, I assume that it is currently on INFO. You may need to recompile, because the properties may also be in the bank.

Answer 2

I found a solution for this. All he needs to do is change the mapreduce configuration file.

• mapreduce.map.log.level can take values ​​like OFF, FATAL, ERROR, WARN, INFO, DEBUG, TRACE and ALL. Installation may be canceled if "mapreduce.job.log4j-properties-file" is installed.

• mapreduce.reduce.log.level can also take values ​​like OFF, FATAL, ERROR, WARN, INFO, DEBUG, TRACE, and ALL. The installation can be overridden if "mapreduce.job.log4j-properties-file" is set. So it’s better to make sure that "mapreduce.job.log4j-properties-file" is not set.

We must set the following properties in mapred-site.xml.

 <property> <name>mapreduce.map.log.level</name> <value>OFF</value> </property> <property> <name>mapreduce.reduce.log.level</name> <value>OFF</value> </property> 

Now I do not see log messages on the console. But this also has a drawback, because we cannot find any error if this happens when the mapreduce code is executed, because the log messages are not visible.