co.cask.cdap.data2.dataset2.lib.partitioned

Class PartitionedFileSetDataset

java.lang.Object

co.cask.cdap.api.dataset.lib.AbstractDataset

co.cask.cdap.data2.dataset2.lib.partitioned.PartitionedFileSetDataset

All Implemented Interfaces:

co.cask.cdap.api.data.batch.DatasetOutputCommitter, co.cask.cdap.api.data.batch.InputFormatProvider, co.cask.cdap.api.data.batch.OutputFormatProvider, co.cask.cdap.api.dataset.Dataset, co.cask.cdap.api.dataset.lib.PartitionedFileSet, co.cask.cdap.api.dataset.metrics.MeteredDataset, co.cask.tephra.TransactionAware, Closeable, AutoCloseable

Direct Known Subclasses:

TimePartitionedFileSetDataset

public class PartitionedFileSetDataset
extends co.cask.cdap.api.dataset.lib.AbstractDataset
implements co.cask.cdap.api.dataset.lib.PartitionedFileSet, co.cask.cdap.api.data.batch.DatasetOutputCommitter

Implementation of partitioned datasets using a Table to store the meta data.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static class	PartitionedFileSetDataset.BasicPartitionOutput Simple Implementation of PartitionOutput.
protected static interface	PartitionedFileSetDataset.PartitionConsumer Interface use internally to build different types of results when scanning partitions.

Field Summary

Fields

Modifier and Type	Field and Description
protected static byte[]	CREATION_TIME_COL
protected com.google.inject.Provider<ExploreFacade>	exploreFacadeProvider
protected static byte[]	FIELD_PREFIX
protected co.cask.cdap.api.dataset.lib.FileSet	files
protected boolean	ignoreInvalidRowsSilently
protected boolean	isExternal
protected static byte[]	METADATA_PREFIX
protected co.cask.cdap.api.dataset.lib.Partitioning	partitioning
protected co.cask.cdap.api.dataset.lib.IndexedTable	partitionsTable
protected static byte[]	RELATIVE_PATH
protected Map<String,String>	runtimeArguments
protected co.cask.cdap.api.dataset.DatasetSpecification	spec
protected static byte[]	WRITE_PTR_COL

Constructor Summary

Constructors

Constructor and Description

PartitionedFileSetDataset(co.cask.cdap.api.dataset.DatasetContext datasetContext, String name, co.cask.cdap.api.dataset.lib.Partitioning partitioning, co.cask.cdap.api.dataset.lib.FileSet fileSet, co.cask.cdap.api.dataset.lib.IndexedTable partitionTable, co.cask.cdap.api.dataset.DatasetSpecification spec, Map<String,String> arguments, com.google.inject.Provider<ExploreFacade> exploreFacadeProvider)

Method Summary

Methods

Modifier and Type	Method and Description
void	addMetadata(co.cask.cdap.api.dataset.lib.PartitionKey key, Map<String,String> metadata)
void	addMetadata(co.cask.cdap.api.dataset.lib.PartitionKey key, String metadataKey, String metadataValue)
void	addPartition(co.cask.cdap.api.dataset.lib.PartitionKey key, String path)
protected void	addPartition(co.cask.cdap.api.dataset.lib.PartitionKey key, String path, boolean explorable, Map<String,String> metadata) Add a partition for a given partition key, stored at a given path (relative to the file set's base path), with the given metadata.
void	addPartition(co.cask.cdap.api.dataset.lib.PartitionKey key, String path, Map<String,String> metadata)
protected void	addPartitionToExplore(co.cask.cdap.api.dataset.lib.PartitionKey key, String path)
void	close()
boolean	commitTx()
protected Collection<String>	computeFilterInputPaths() If a partition filter was specified, return the input locations of all partitions matching the filter.
co.cask.cdap.api.dataset.lib.PartitionConsumerResult	consumePartitions(co.cask.cdap.api.dataset.lib.PartitionConsumerState partitionConsumerState)
co.cask.cdap.api.dataset.lib.PartitionConsumerResult	consumePartitions(co.cask.cdap.api.dataset.lib.PartitionConsumerState partitionConsumerState, int limit, co.cask.cdap.api.Predicate<co.cask.cdap.api.dataset.lib.PartitionDetail> predicate)
void	dropPartition(co.cask.cdap.api.dataset.lib.PartitionKey key)
protected void	dropPartitionFromExplore(co.cask.cdap.api.dataset.lib.PartitionKey key)
co.cask.cdap.api.dataset.lib.FileSet	getEmbeddedFileSet()
String	getInputFormatClassName()
Map<String,String>	getInputFormatConfiguration()
String	getOutputFormatClassName()
Map<String,String>	getOutputFormatConfiguration()
static String	getOutputPath(co.cask.cdap.api.dataset.lib.PartitionKey key, co.cask.cdap.api.dataset.lib.Partitioning partitioning)
co.cask.cdap.api.dataset.lib.PartitionDetail	getPartition(co.cask.cdap.api.dataset.lib.PartitionKey key)
co.cask.cdap.api.dataset.lib.Partitioning	getPartitioning()
co.cask.cdap.api.dataset.lib.PartitionOutput	getPartitionOutput(co.cask.cdap.api.dataset.lib.PartitionKey key)
Set<co.cask.cdap.api.dataset.lib.PartitionDetail>	getPartitions(co.cask.cdap.api.dataset.lib.PartitionFilter filter)
protected void	getPartitions(co.cask.cdap.api.dataset.lib.PartitionFilter filter, PartitionedFileSetDataset.PartitionConsumer consumer)
protected void	getPartitions(co.cask.cdap.api.dataset.lib.PartitionFilter filter, PartitionedFileSetDataset.PartitionConsumer consumer, boolean decodeMetadata)
Map<String,String>	getRuntimeArguments()
void	onFailure()
void	onSuccess()
boolean	rollbackTx()
void	startTx(co.cask.tephra.Transaction tx)

Methods inherited from class co.cask.cdap.api.dataset.lib.AbstractDataset

getName, getTransactionAwareName, getTxChanges, postTxCommit, setMetricsCollector, toString, updateTx

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

RELATIVE_PATH

protected static final byte[] RELATIVE_PATH

FIELD_PREFIX

protected static final byte[] FIELD_PREFIX

METADATA_PREFIX

protected static final byte[] METADATA_PREFIX

CREATION_TIME_COL

protected static final byte[] CREATION_TIME_COL

WRITE_PTR_COL

protected static final byte[] WRITE_PTR_COL

files

protected final co.cask.cdap.api.dataset.lib.FileSet files

partitionsTable

protected final co.cask.cdap.api.dataset.lib.IndexedTable partitionsTable

spec

protected final co.cask.cdap.api.dataset.DatasetSpecification spec

isExternal

protected final boolean isExternal

runtimeArguments

protected final Map<String,String> runtimeArguments

exploreFacadeProvider

protected final com.google.inject.Provider<ExploreFacade> exploreFacadeProvider

partitioning

protected final co.cask.cdap.api.dataset.lib.Partitioning partitioning

ignoreInvalidRowsSilently

protected boolean ignoreInvalidRowsSilently

Constructor Detail

PartitionedFileSetDataset

public PartitionedFileSetDataset(co.cask.cdap.api.dataset.DatasetContext datasetContext,
                         String name,
                         co.cask.cdap.api.dataset.lib.Partitioning partitioning,
                         co.cask.cdap.api.dataset.lib.FileSet fileSet,
                         co.cask.cdap.api.dataset.lib.IndexedTable partitionTable,
                         co.cask.cdap.api.dataset.DatasetSpecification spec,
                         Map<String,String> arguments,
                         com.google.inject.Provider<ExploreFacade> exploreFacadeProvider)

Method Detail

startTx

public void startTx(co.cask.tephra.Transaction tx)

Specified by:

startTx in interface co.cask.tephra.TransactionAware

Overrides:

startTx in class co.cask.cdap.api.dataset.lib.AbstractDataset

commitTx

public boolean commitTx()
                 throws Exception

Specified by:

commitTx in interface co.cask.tephra.TransactionAware

Overrides:

commitTx in class co.cask.cdap.api.dataset.lib.AbstractDataset

Throws:

Exception

rollbackTx

public boolean rollbackTx()
                   throws Exception

Specified by:

rollbackTx in interface co.cask.tephra.TransactionAware

Overrides:

rollbackTx in class co.cask.cdap.api.dataset.lib.AbstractDataset

Throws:

Exception

getPartitioning

public co.cask.cdap.api.dataset.lib.Partitioning getPartitioning()

Specified by:

getPartitioning in interface co.cask.cdap.api.dataset.lib.PartitionedFileSet

addPartition

public void addPartition(co.cask.cdap.api.dataset.lib.PartitionKey key,
                String path)

Specified by:

addPartition in interface co.cask.cdap.api.dataset.lib.PartitionedFileSet

addPartition

public void addPartition(co.cask.cdap.api.dataset.lib.PartitionKey key,
                String path,
                Map<String,String> metadata)

Specified by:

addPartition in interface co.cask.cdap.api.dataset.lib.PartitionedFileSet

addPartition

protected void addPartition(co.cask.cdap.api.dataset.lib.PartitionKey key,
                String path,
                boolean explorable,
                Map<String,String> metadata)

Add a partition for a given partition key, stored at a given path (relative to the file set's base path), with the given metadata.

Parameters:

key - the partitionKey for the partition.

path - the path for the partition.

explorable - whether to add the partition to explore

metadata - the metadata associated with the partition.

consumePartitions

public co.cask.cdap.api.dataset.lib.PartitionConsumerResult consumePartitions(co.cask.cdap.api.dataset.lib.PartitionConsumerState partitionConsumerState)

Specified by:

consumePartitions in interface co.cask.cdap.api.dataset.lib.PartitionedFileSet

consumePartitions

public co.cask.cdap.api.dataset.lib.PartitionConsumerResult consumePartitions(co.cask.cdap.api.dataset.lib.PartitionConsumerState partitionConsumerState,
                                                                     int limit,
                                                                     co.cask.cdap.api.Predicate<co.cask.cdap.api.dataset.lib.PartitionDetail> predicate)

Specified by:

consumePartitions in interface co.cask.cdap.api.dataset.lib.PartitionedFileSet

addMetadata

public void addMetadata(co.cask.cdap.api.dataset.lib.PartitionKey key,
               String metadataKey,
               String metadataValue)

Specified by:

addMetadata in interface co.cask.cdap.api.dataset.lib.PartitionedFileSet

addMetadata

public void addMetadata(co.cask.cdap.api.dataset.lib.PartitionKey key,
               Map<String,String> metadata)

Specified by:

addMetadata in interface co.cask.cdap.api.dataset.lib.PartitionedFileSet

addPartitionToExplore

protected void addPartitionToExplore(co.cask.cdap.api.dataset.lib.PartitionKey key,
                         String path)

dropPartition

public void dropPartition(co.cask.cdap.api.dataset.lib.PartitionKey key)

Specified by:

dropPartition in interface co.cask.cdap.api.dataset.lib.PartitionedFileSet

dropPartitionFromExplore

protected void dropPartitionFromExplore(co.cask.cdap.api.dataset.lib.PartitionKey key)

getPartitionOutput

public co.cask.cdap.api.dataset.lib.PartitionOutput getPartitionOutput(co.cask.cdap.api.dataset.lib.PartitionKey key)

Specified by:

getPartitionOutput in interface co.cask.cdap.api.dataset.lib.PartitionedFileSet

getPartition

public co.cask.cdap.api.dataset.lib.PartitionDetail getPartition(co.cask.cdap.api.dataset.lib.PartitionKey key)

Specified by:

getPartition in interface co.cask.cdap.api.dataset.lib.PartitionedFileSet

getPartitions

public Set<co.cask.cdap.api.dataset.lib.PartitionDetail> getPartitions(@Nullable
                                                              co.cask.cdap.api.dataset.lib.PartitionFilter filter)

Specified by:

getPartitions in interface co.cask.cdap.api.dataset.lib.PartitionedFileSet

getPartitions

protected void getPartitions(@Nullable
                 co.cask.cdap.api.dataset.lib.PartitionFilter filter,
                 PartitionedFileSetDataset.PartitionConsumer consumer)

getPartitions

protected void getPartitions(@Nullable
                 co.cask.cdap.api.dataset.lib.PartitionFilter filter,
                 PartitionedFileSetDataset.PartitionConsumer consumer,
                 boolean decodeMetadata)

getOutputPath

public static String getOutputPath(co.cask.cdap.api.dataset.lib.PartitionKey key,
                   co.cask.cdap.api.dataset.lib.Partitioning partitioning)

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Overrides:

close in class co.cask.cdap.api.dataset.lib.AbstractDataset

Throws:

IOException

getInputFormatClassName

public String getInputFormatClassName()

Specified by:

getInputFormatClassName in interface co.cask.cdap.api.data.batch.InputFormatProvider

getInputFormatConfiguration

public Map<String,String> getInputFormatConfiguration()

Specified by:

getInputFormatConfiguration in interface co.cask.cdap.api.data.batch.InputFormatProvider

computeFilterInputPaths

@Nullable
protected Collection<String> computeFilterInputPaths()

If a partition filter was specified, return the input locations of all partitions matching the filter. Otherwise return null.

getOutputFormatClassName

public String getOutputFormatClassName()

Specified by:

getOutputFormatClassName in interface co.cask.cdap.api.data.batch.OutputFormatProvider

getOutputFormatConfiguration

public Map<String,String> getOutputFormatConfiguration()

Specified by:

getOutputFormatConfiguration in interface co.cask.cdap.api.data.batch.OutputFormatProvider

onSuccess

public void onSuccess()
               throws co.cask.cdap.api.dataset.DataSetException

Specified by:

onSuccess in interface co.cask.cdap.api.data.batch.DatasetOutputCommitter

Throws:

co.cask.cdap.api.dataset.DataSetException

onFailure

public void onFailure()
               throws co.cask.cdap.api.dataset.DataSetException

Specified by:

onFailure in interface co.cask.cdap.api.data.batch.DatasetOutputCommitter

Throws:

co.cask.cdap.api.dataset.DataSetException

getEmbeddedFileSet

public co.cask.cdap.api.dataset.lib.FileSet getEmbeddedFileSet()

Specified by:

getEmbeddedFileSet in interface co.cask.cdap.api.dataset.lib.PartitionedFileSet

getRuntimeArguments

public Map<String,String> getRuntimeArguments()

Specified by:

getRuntimeArguments in interface co.cask.cdap.api.dataset.lib.PartitionedFileSet