Speculative execution in Hadoop MapReduce is an option to run a duplicate map or reduce task for the same input data on an alternative node. This is done so that any slow running task doesn’t slow down the whole job.
Why is speculative execution in Hadoop needed
In Hadoop an input file is partitioned into several blocks and those blocks are stored on different nodes on a Hadoop cluster, there is also a replication of those blocks for redundancy.
When a Map-Reduce job runs it calculates the number of input splits (size of split is equal to HDFS block) and run as many map tasks as the number of splits. These map tasks run in parallel on the nodes where the data referred by the split resides.
What if few nodes in the cluster are not performing as fast as other nodes because of hardware or network problems. Map tasks running on those nodes will be slower compared to the map tasks running on other nodes. Reduce tasks can only start their execution once intermediate outputs of all the map tasks are available. So few slow moving map tasks can delay the execution of reduce tasks.
Also reduce tasks running on a slower node may take more time to finish thus delaying the over all job final output.
To guard against such slow tasks Hadoop starts the same task (working on the same input) on another node. Note that every block is replicated thrice by default. Hadoop will get the location of another node where the same input data resides and launch the task on that node with the assumption that on that node task will finish faster. This optimization by Hadoop is called the speculative execution of the task.
When is speculative task started
Once the map tasks or reduce tasks are started and monitored for some time Hadoop framework can determine which map task or reduce task is not making as much progress as the other running tasks of the same type. Only after this monitoring for some time and determining which tasks are slower Hadoop starts speculative execution of the tasks.
Since the speculative task in MapReduce and the original task both are working on the same set of data, output of which ever task finishes first successfully is used and the other one is killed.
How to configure speculative execution in Hadoop
Speculative execution is enabled by default for both map and reduce tasks. Properties for speculative execution are set in mapred-site.xml file.
- mapreduce.map.speculative- If set to true then speculative execution of map task is enabled. Default is true.
- mapreduce.reduce.speculative- If set to true then speculative execution of reduce task is enabled. Default is true.
- mapreduce.job.speculative.speculative-cap-running-tasks- The max percent (0-1) of running tasks that can be speculatively re-executed at any time. Default value is 0.1.
- mapreduce.job.speculative.speculative-cap-total-tasks- The max percent (0-1) of all tasks that can be speculatively re-executed at any time. Default value is 0.01.
Consideration for turning off speculative execution
Since speculative execution of task means running duplicate tasks, it increases the cluster load. If you have a very busy cluster or a cluster with limited resources then you may consider turning off the speculative execution.
Another thing to consider is that reduce task gets its input from more than one map tasks running on different nodes so there is data transfer in case of reduce tasks. Running a duplicate reduce task means same data transfer happens more than once thus increasing load on network.
That's all for this topic Speculative Execution in Hadoop. If you have any doubt or any suggestions to make please drop a comment. Thanks!
>>>Return to Hadoop Framework Tutorial Page
Related Topics
You may also like-