org.apache.tez.runtime.library.common.task.local.output

Class TezTaskOutput

java.lang.Object

org.apache.tez.runtime.library.common.task.local.output.TezTaskOutput

Direct Known Subclasses:

TezLocalTaskOutputFiles, TezTaskOutputFiles

@InterfaceAudience.Private
@InterfaceStability.Unstable
public abstract class TezTaskOutput
extends Object

Manipulate the working area for the transient store for maps and reduces. This class is used by map and reduce tasks to identify the directories that they need to write to/read from for intermediate files. The callers of these methods are from child space and see mapreduce.cluster.local.dir as taskTracker/jobCache/jobId/attemptId This class should not be used from TaskTracker space.

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.hadoop.conf.Configuration	conf
protected String	uniqueId

Constructor Summary

Constructors

Constructor and Description
TezTaskOutput(org.apache.hadoop.conf.Configuration conf, String uniqueId)

Method Summary

Methods

Modifier and Type	Method and Description
abstract org.apache.hadoop.fs.Path	getInputFile(InputAttemptIdentifier attemptIdentifier) Return a local reduce input file created earlier
abstract org.apache.hadoop.fs.Path	getInputFileForWrite(int taskIdentifier, long size) Create a local reduce input file name.
abstract org.apache.hadoop.fs.Path	getOutputFile() Return the path to local map output file created earlier
abstract org.apache.hadoop.fs.Path	getOutputFileForWrite() Create a local output file name.
abstract org.apache.hadoop.fs.Path	getOutputFileForWrite(long size) Create a local map output file name.
abstract org.apache.hadoop.fs.Path	getOutputFileForWriteInVolume(org.apache.hadoop.fs.Path existing) Create a local map output file name on the same volume.
abstract org.apache.hadoop.fs.Path	getOutputIndexFile() Return the path to a local map output index file created earlier
abstract org.apache.hadoop.fs.Path	getOutputIndexFileForWrite(long size) Create a local map output index file name.
abstract org.apache.hadoop.fs.Path	getOutputIndexFileForWriteInVolume(org.apache.hadoop.fs.Path existing) Create a local map output index file name on the same volume.
abstract org.apache.hadoop.fs.Path	getSpillFile(int spillNumber) Return a local map spill file created earlier.
abstract org.apache.hadoop.fs.Path	getSpillFileForWrite(int spillNumber, long size) Create a local map spill file name.
abstract String	getSpillFileName(int spillNum) Construct a spill file name, given a spill number
abstract org.apache.hadoop.fs.Path	getSpillIndexFile(int spillNumber) Return a local map spill index file created earlier
abstract org.apache.hadoop.fs.Path	getSpillIndexFileForWrite(int spillNumber, long size) Create a local map spill index file name.
abstract void	removeAll() Removes all of the files related to a task.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

conf

protected org.apache.hadoop.conf.Configuration conf

uniqueId

protected String uniqueId

Constructor Detail

TezTaskOutput

public TezTaskOutput(org.apache.hadoop.conf.Configuration conf,
             String uniqueId)

Method Detail

getOutputFile

public abstract org.apache.hadoop.fs.Path getOutputFile()
                                                 throws IOException

Return the path to local map output file created earlier

Returns:

path

Throws:

IOException

getOutputFileForWrite

public abstract org.apache.hadoop.fs.Path getOutputFileForWrite(long size)
                                                         throws IOException

Create a local map output file name.

Parameters:

size - the size of the file

Returns:

path

Throws:

IOException

getOutputFileForWrite

public abstract org.apache.hadoop.fs.Path getOutputFileForWrite()
                                                         throws IOException

Create a local output file name. This method is meant to be used *only* if the size of the file is not know up front.

Returns:

path

Throws:

IOException

getOutputFileForWriteInVolume

public abstract org.apache.hadoop.fs.Path getOutputFileForWriteInVolume(org.apache.hadoop.fs.Path existing)

Create a local map output file name on the same volume.

getOutputIndexFile

public abstract org.apache.hadoop.fs.Path getOutputIndexFile()
                                                      throws IOException

Return the path to a local map output index file created earlier

Returns:

path

Throws:

IOException

getOutputIndexFileForWrite

public abstract org.apache.hadoop.fs.Path getOutputIndexFileForWrite(long size)
                                                              throws IOException

Create a local map output index file name.

Parameters:

size - the size of the file

Returns:

path

Throws:

IOException

getOutputIndexFileForWriteInVolume

public abstract org.apache.hadoop.fs.Path getOutputIndexFileForWriteInVolume(org.apache.hadoop.fs.Path existing)

Create a local map output index file name on the same volume.

getSpillFile

public abstract org.apache.hadoop.fs.Path getSpillFile(int spillNumber)
                                                throws IOException

Return a local map spill file created earlier.

Parameters:

spillNumber - the number

Returns:

path

Throws:

IOException

getSpillFileForWrite

public abstract org.apache.hadoop.fs.Path getSpillFileForWrite(int spillNumber,
                                             long size)
                                                        throws IOException

Create a local map spill file name.

Parameters:

spillNumber - the number

size - the size of the file

Returns:

path

Throws:

IOException

getSpillIndexFile

public abstract org.apache.hadoop.fs.Path getSpillIndexFile(int spillNumber)
                                                     throws IOException

Return a local map spill index file created earlier

Parameters:

spillNumber - the number

Returns:

path

Throws:

IOException

getSpillIndexFileForWrite

public abstract org.apache.hadoop.fs.Path getSpillIndexFileForWrite(int spillNumber,
                                                  long size)
                                                             throws IOException

Create a local map spill index file name.

Parameters:

spillNumber - the number

size - the size of the file

Returns:

path

Throws:

IOException

getInputFile

public abstract org.apache.hadoop.fs.Path getInputFile(InputAttemptIdentifier attemptIdentifier)
                                                throws IOException

Return a local reduce input file created earlier

Parameters:

attemptIdentifier - The identifier for the source task

Returns:

path

Throws:

IOException

getInputFileForWrite

public abstract org.apache.hadoop.fs.Path getInputFileForWrite(int taskIdentifier,
                                             long size)
                                                        throws IOException

Create a local reduce input file name.

Parameters:

taskIdentifier - The identifier for the source task

size - the size of the file

Returns:

path

Throws:

IOException

getSpillFileName

public abstract String getSpillFileName(int spillNum)

Construct a spill file name, given a spill number

Parameters:

spillNum -

Returns:

removeAll

public abstract void removeAll()
                        throws IOException

Removes all of the files related to a task.

Throws:

IOException