org.apache.hadoop.tools.mapred

Class UniformSizeInputFormat

java.lang.Object

org.apache.hadoop.mapreduce.InputFormat<Text,CopyListingFileStatus>

org.apache.hadoop.tools.mapred.UniformSizeInputFormat

public class UniformSizeInputFormat
extends InputFormat<Text,CopyListingFileStatus>

UniformSizeInputFormat extends the InputFormat class, to produce input-splits for DistCp. It looks at the copy-listing and groups the contents into input-splits such that the total-number of bytes to be copied for each input split is uniform.

Constructor Summary

Constructors

Constructor and Description
UniformSizeInputFormat()

Method Summary

Modifier and Type	Method and Description
RecordReader<Text,CopyListingFileStatus>	createRecordReader(InputSplit split, TaskAttemptContext context) Implementation of InputFormat::createRecordReader().
List<InputSplit>	getSplits(JobContext context) Implementation of InputFormat::getSplits().

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

UniformSizeInputFormat

public UniformSizeInputFormat()

Method Detail

getSplits

public List<InputSplit> getSplits(JobContext context)
                           throws IOException,
                                  InterruptedException

Implementation of InputFormat::getSplits(). Returns a list of InputSplits, such that the number of bytes to be copied for all the splits are approximately equal.

Specified by:

getSplits in class InputFormat<Text,CopyListingFileStatus>

Parameters:

context - JobContext for the job.

Returns:

The list of uniformly-distributed input-splits.

Throws:

IOException

InterruptedException

createRecordReader

public RecordReader<Text,CopyListingFileStatus> createRecordReader(InputSplit split,
                                                                   TaskAttemptContext context)
                                                            throws IOException,
                                                                   InterruptedException

Implementation of InputFormat::createRecordReader().

Specified by:

createRecordReader in class InputFormat<Text,CopyListingFileStatus>

Parameters:

split - The split for which the RecordReader is sought.

context - The context of the current task-attempt.

Returns:

A SequenceFileRecordReader instance, (since the copy-listing is a simple sequence-file.)

Throws:

IOException

InterruptedException