Package com.google.cloud.bigtable.mapreduce

Class ShuffledSequenceFileInputFormat<K,V>

java.lang.Object

org.apache.hadoop.mapreduce.InputFormat<K,V>

org.apache.hadoop.mapreduce.lib.input.FileInputFormat<K,V>

org.apache.hadoop.mapreduce.lib.input.SequenceFileInputFormat<K,V>

com.google.cloud.bigtable.mapreduce.ShuffledSequenceFileInputFormat<K,V>

public class ShuffledSequenceFileInputFormat<K,V>
extends org.apache.hadoop.mapreduce.lib.input.SequenceFileInputFormat<K,V>

An InputFormat randomizes the order of SequenceFile splits. This is needed for the Bigtable import job to spread the load across all of the Bigtable tablet servers.

Version:

$Id: $Id

Author:

igorbernstein2

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.mapreduce.lib.input.FileInputFormat

org.apache.hadoop.mapreduce.lib.input.FileInputFormat.Counter

Field Summary

Fields inherited from class org.apache.hadoop.mapreduce.lib.input.FileInputFormat

DEFAULT_LIST_STATUS_NUM_THREADS, INPUT_DIR, INPUT_DIR_RECURSIVE, LIST_STATUS_NUM_THREADS, NUM_INPUT_FILES, PATHFILTER_CLASS, SPLIT_MAXSIZE, SPLIT_MINSIZE

Constructor Summary

Constructors

Constructor	Description
ShuffledSequenceFileInputFormat()

Method Summary

Modifier and Type	Method	Description
List<org.apache.hadoop.mapreduce.InputSplit>	getSplits(org.apache.hadoop.mapreduce.JobContext job)

Methods inherited from class org.apache.hadoop.mapreduce.lib.input.FileInputFormat

addInputPath, addInputPathRecursively, addInputPaths, computeSplitSize, getBlockIndex, getInputDirRecursive, getInputPathFilter, getInputPaths, getMaxSplitSize, getMinSplitSize, isSplitable, makeSplit, makeSplit, setInputDirRecursive, setInputPathFilter, setInputPaths, setInputPaths, setMaxInputSplitSize, setMinInputSplitSize

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from class org.apache.hadoop.mapreduce.lib.input.SequenceFileInputFormat

createRecordReader, getFormatMinSplitSize, listStatus

Constructor Detail

ShuffledSequenceFileInputFormat

public ShuffledSequenceFileInputFormat()

Method Detail

getSplits

public List<org.apache.hadoop.mapreduce.InputSplit> getSplits(org.apache.hadoop.mapreduce.JobContext job)
                                                       throws IOException

Overrides:

getSplits in class org.apache.hadoop.mapreduce.lib.input.FileInputFormat<K,V>

Throws:

IOException