org.apache.hadoop.tools.mapred.lib

Class DynamicInputFormat<K,V>

java.lang.Object

org.apache.hadoop.mapreduce.InputFormat<K,V>

org.apache.hadoop.tools.mapred.lib.DynamicInputFormat<K,V>

public class DynamicInputFormat<K,V>
extends InputFormat<K,V>

DynamicInputFormat implements the "Worker pattern" for DistCp. Rather than to split up the copy-list into a set of static splits, the DynamicInputFormat does the following: 1. Splits the copy-list into small chunks on the DFS. 2. Creates a set of empty "dynamic" splits, that each consume as many chunks as it can. This arrangement ensures that a single slow mapper won't slow down the entire job (since the slack will be picked up by other mappers, who consume more chunks.) By varying the split-ratio, one can vary chunk sizes to achieve different performance characteristics.

Constructor Summary

Constructors

Constructor and Description
DynamicInputFormat()

Method Summary

Modifier and Type	Method and Description
RecordReader<K,V>	createRecordReader(InputSplit inputSplit, TaskAttemptContext taskAttemptContext) Implementation of Inputformat::createRecordReader().
org.apache.hadoop.tools.mapred.lib.DynamicInputChunkContext<K,V>	getChunkContext(Configuration configuration)
List<InputSplit>	getSplits(JobContext jobContext) Implementation of InputFormat::getSplits().

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DynamicInputFormat

public DynamicInputFormat()

Method Detail

getSplits

public List<InputSplit> getSplits(JobContext jobContext)
                           throws IOException,
                                  InterruptedException

Implementation of InputFormat::getSplits(). This method splits up the copy-listing file into chunks, and assigns the first batch to different tasks.

Specified by:

getSplits in class InputFormat<K,V>

Parameters:

jobContext - JobContext for the map job.

Returns:

The list of (empty) dynamic input-splits.

Throws:

IOException

InterruptedException

getChunkContext

public org.apache.hadoop.tools.mapred.lib.DynamicInputChunkContext<K,V> getChunkContext(Configuration configuration)
                                                                                 throws IOException

Throws:

IOException

createRecordReader

public RecordReader<K,V> createRecordReader(InputSplit inputSplit,
                                            TaskAttemptContext taskAttemptContext)
                                     throws IOException,
                                            InterruptedException

Implementation of Inputformat::createRecordReader().

Specified by:

createRecordReader in class InputFormat<K,V>

Parameters:

inputSplit - The split for which the RecordReader is required.

taskAttemptContext - TaskAttemptContext for the current attempt.

Returns:

DynamicRecordReader instance.

Throws:

IOException

InterruptedException