In your Hadoop MapReduce job if you are wondering how to put logs or where to check MapReduce logs or even System.out statements then this post shows the same. Note that here accessing logs is shown for MapReuduce 2.
Location of logs in Hadoop MapReduce
An application ID is created for every MapReduce job. You can get that application ID from the console itself after starting your MapReduce job. It will be similar to as shown below.
18/07/11 14:39:23 INFO impl.YarnClientImpl: Submitted application application_1531299441901_0001
A folder with the same application ID will be created in the logs/userlogs of your Hadoop installation directory. For example I can see following directory for the application ID mentioned above.
HADOOP_INSTALLATION_DIR/logs/userlogs/application_1531299441901_0001
With in this directory you will find separate folders created for mappers and reducers and there you will have following files for logs and sysouts.
syslog- Contains the log messages.
sysout- Contains the System.out messages.
MapReduce example with logs
Here is a simple word count MapReduce program with logs and sysouts added.
import java.io.IOException;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;
public class WordCount extends Configured implements Tool {
public static final Log log = LogFactory.getLog(WordCount.class);
// Map function
public static class MyMapper extends Mapper<LongWritable, Text, Text, IntWritable>{
private Text word = new Text();
public void map(LongWritable key, Text value, Context context)
throws IOException, InterruptedException {
log.info("In map method");
// Splitting the line on spaces
String[] stringArr = value.toString().split("\\s+");
System.out.println("Array length- " + stringArr.length);
for (String str : stringArr) {
word.set(str);
context.write(word, new IntWritable(1));
}
}
}
// Reduce function
public static class MyReducer extends Reducer<Text, IntWritable, Text, IntWritable>{
private IntWritable result = new IntWritable();
public void reduce(Text key, Iterable<IntWritable> values, Context context)
throws IOException, InterruptedException {
log.info("In reduce method with key " + key);
int sum = 0;
for (IntWritable val : values) {
sum += val.get();
}
System.out.println("Key - " + key + " sum - " + sum);
result.set(sum);
context.write(key, result);
}
}
public static void main(String[] args) throws Exception {
int exitFlag = ToolRunner.run(new WordCount(), args);
System.exit(exitFlag);
}
@Override
public int run(String[] args) throws Exception {
Configuration conf = new Configuration();
Job job = Job.getInstance(conf, "WordCount");
job.setJarByClass(getClass());
job.setMapperClass(MyMapper.class);
job.setReducerClass(MyReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
return job.waitForCompletion(true) ? 0 : 1;
}
}
Once you run this MapReduce job, using the application ID you can go to the
location as already explained above and check the log and sysout messages.
That's all for this topic How to Check Hadoop MapReduce Logs. If you have any doubt or any suggestions to make please drop a comment. Thanks!
>>>Return to Hadoop Framework Tutorial Page
Related Topics
You may also like-