co.cask.cdap.data2.dataset2.lib.partitioned

Class TimePartitionedFileSetDataset

java.lang.Object

co.cask.cdap.api.dataset.lib.AbstractDataset

co.cask.cdap.data2.dataset2.lib.partitioned.PartitionedFileSetDataset

co.cask.cdap.data2.dataset2.lib.partitioned.TimePartitionedFileSetDataset

All Implemented Interfaces:

co.cask.cdap.api.data.batch.DatasetOutputCommitter, co.cask.cdap.api.data.batch.InputFormatProvider, co.cask.cdap.api.data.batch.OutputFormatProvider, co.cask.cdap.api.dataset.Dataset, co.cask.cdap.api.dataset.lib.PartitionedFileSet, co.cask.cdap.api.dataset.lib.TimePartitionedFileSet, co.cask.cdap.api.dataset.metrics.MeteredDataset, co.cask.tephra.TransactionAware, Closeable, AutoCloseable

public class TimePartitionedFileSetDataset
extends PartitionedFileSetDataset
implements co.cask.cdap.api.dataset.lib.TimePartitionedFileSet

Implementation of partitioned datasets using a Table to store the meta data.

Nested Class Summary

Nested classes/interfaces inherited from class co.cask.cdap.data2.dataset2.lib.partitioned.PartitionedFileSetDataset

PartitionedFileSetDataset.BasicPartitionOutput, PartitionedFileSetDataset.PartitionConsumer

Field Summary

Fields

Modifier and Type	Field and Description
static co.cask.cdap.api.dataset.lib.Partitioning	PARTITIONING

Fields inherited from class co.cask.cdap.data2.dataset2.lib.partitioned.PartitionedFileSetDataset

CREATION_TIME_COL, exploreFacadeProvider, FIELD_PREFIX, files, ignoreInvalidRowsSilently, isExternal, METADATA_PREFIX, partitioning, partitionsTable, RELATIVE_PATH, runtimeArguments, spec, WRITE_PTR_COL

Constructor Summary

Constructors

Constructor and Description
TimePartitionedFileSetDataset(co.cask.cdap.api.dataset.DatasetContext datasetContext, String name, co.cask.cdap.api.dataset.lib.FileSet fileSet, co.cask.cdap.api.dataset.lib.IndexedTable partitionTable, co.cask.cdap.api.dataset.DatasetSpecification spec, Map<String,String> arguments, com.google.inject.Provider<ExploreFacade> exploreFacadeProvider)

Method Summary

Methods

Modifier and Type	Method and Description
void	addMetadata(long time, Map<String,String> metadata)
void	addMetadata(long time, String metadataKey, String metadataValue)
void	addPartition(long time, String path)
void	addPartition(long time, String path, Map<String,String> metadata)
protected Collection<String>	computeFilterInputPaths() If a partition filter was specified, return the input locations of all partitions matching the filter.
void	dropPartition(long time)
co.cask.cdap.api.dataset.lib.TimePartitionDetail	getPartitionByTime(long time)
co.cask.cdap.api.dataset.lib.TimePartitionOutput	getPartitionOutput(long time)
Set<co.cask.cdap.api.dataset.lib.TimePartitionDetail>	getPartitionsByTime(long startTime, long endTime)

Methods inherited from class co.cask.cdap.data2.dataset2.lib.partitioned.PartitionedFileSetDataset

addMetadata, addMetadata, addPartition, addPartition, addPartition, addPartitionToExplore, close, commitTx, consumePartitions, consumePartitions, dropPartition, dropPartitionFromExplore, getEmbeddedFileSet, getInputFormatClassName, getInputFormatConfiguration, getOutputFormatClassName, getOutputFormatConfiguration, getOutputPath, getPartition, getPartitioning, getPartitionOutput, getPartitions, getPartitions, getPartitions, getRuntimeArguments, onFailure, onSuccess, rollbackTx, startTx

Methods inherited from class co.cask.cdap.api.dataset.lib.AbstractDataset

getName, getTransactionAwareName, getTxChanges, postTxCommit, setMetricsCollector, toString, updateTx

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface co.cask.cdap.api.dataset.lib.PartitionedFileSet

addMetadata, addMetadata, addPartition, addPartition, consumePartitions, consumePartitions, dropPartition, getEmbeddedFileSet, getPartition, getPartitioning, getPartitionOutput, getPartitions, getRuntimeArguments

Methods inherited from interface java.io.Closeable

close

Methods inherited from interface co.cask.cdap.api.data.batch.InputFormatProvider

getInputFormatClassName, getInputFormatConfiguration

Methods inherited from interface co.cask.cdap.api.data.batch.OutputFormatProvider

getOutputFormatClassName, getOutputFormatConfiguration

Field Detail

PARTITIONING

public static final co.cask.cdap.api.dataset.lib.Partitioning PARTITIONING

Constructor Detail

TimePartitionedFileSetDataset

public TimePartitionedFileSetDataset(co.cask.cdap.api.dataset.DatasetContext datasetContext,
                             String name,
                             co.cask.cdap.api.dataset.lib.FileSet fileSet,
                             co.cask.cdap.api.dataset.lib.IndexedTable partitionTable,
                             co.cask.cdap.api.dataset.DatasetSpecification spec,
                             Map<String,String> arguments,
                             com.google.inject.Provider<ExploreFacade> exploreFacadeProvider)

Method Detail

addPartition

public void addPartition(long time,
                String path)

Specified by:

addPartition in interface co.cask.cdap.api.dataset.lib.TimePartitionedFileSet

addPartition

public void addPartition(long time,
                String path,
                Map<String,String> metadata)

Specified by:

addPartition in interface co.cask.cdap.api.dataset.lib.TimePartitionedFileSet

addMetadata

public void addMetadata(long time,
               String metadataKey,
               String metadataValue)

Specified by:

addMetadata in interface co.cask.cdap.api.dataset.lib.TimePartitionedFileSet

addMetadata

public void addMetadata(long time,
               Map<String,String> metadata)

Specified by:

addMetadata in interface co.cask.cdap.api.dataset.lib.TimePartitionedFileSet

dropPartition

public void dropPartition(long time)

Specified by:

dropPartition in interface co.cask.cdap.api.dataset.lib.TimePartitionedFileSet

getPartitionByTime

@Nullable
public co.cask.cdap.api.dataset.lib.TimePartitionDetail getPartitionByTime(long time)

Specified by:

getPartitionByTime in interface co.cask.cdap.api.dataset.lib.TimePartitionedFileSet

getPartitionsByTime

public Set<co.cask.cdap.api.dataset.lib.TimePartitionDetail> getPartitionsByTime(long startTime,
                                                                        long endTime)

Specified by:

getPartitionsByTime in interface co.cask.cdap.api.dataset.lib.TimePartitionedFileSet

getPartitionOutput

public co.cask.cdap.api.dataset.lib.TimePartitionOutput getPartitionOutput(long time)

Specified by:

getPartitionOutput in interface co.cask.cdap.api.dataset.lib.TimePartitionedFileSet

computeFilterInputPaths

@Nullable
protected Collection<String> computeFilterInputPaths()

Description copied from class: PartitionedFileSetDataset

If a partition filter was specified, return the input locations of all partitions matching the filter. Otherwise return null.

Overrides:

computeFilterInputPaths in class PartitionedFileSetDataset