co.cask.cdap.data2.metadata.lineage

Class LineageStore

java.lang.Object

co.cask.cdap.data2.metadata.lineage.LineageStore

public class LineageStore
extends Object

Store for storing/retrieving lineage information for a Dataset.

Constructor Summary

Constructors

Constructor and Description
LineageStore(co.cask.tephra.TransactionExecutorFactory executorFactory, DatasetFramework datasetFramework)
LineageStore(co.cask.tephra.TransactionExecutorFactory executorFactory, DatasetFramework datasetFramework, Id.DatasetInstance lineageDatasetId)

Method Summary

Methods

Modifier and Type	Method and Description
void	addAccess(Id.Run run, Id.DatasetInstance datasetInstance, AccessType accessType, long accessTimeMillis) Add a program-dataset access.
void	addAccess(Id.Run run, Id.DatasetInstance datasetInstance, AccessType accessType, long accessTimeMillis, Id.NamespacedId component) Add a program-dataset access.
void	addAccess(Id.Run run, Id.Stream stream, AccessType accessType, long accessTimeMillis) Add a program-stream access.
void	addAccess(Id.Run run, Id.Stream stream, AccessType accessType, long accessTimeMillis, Id.NamespacedId component) Add a program-stream access.
List<Long>	getAccessTimesForRun(Id.Run run)
Set<Id.NamespacedId>	getEntitiesForRun(Id.Run run)
Set<Relation>	getRelations(Id.DatasetInstance datasetInstance, long start, long end, com.google.common.base.Predicate<Relation> filter) Fetch program-dataset access information for a dataset for a given period.
Set<Relation>	getRelations(Id.Program program, long start, long end, com.google.common.base.Predicate<Relation> filter) Fetch program-dataset access information for a program for a given period.
Set<Relation>	getRelations(Id.Stream stream, long start, long end, com.google.common.base.Predicate<Relation> filter) Fetch program-stream access information for a dataset for a given period.
static void	setupDatasets(DatasetFramework framework) Adds datasets and types to the given DatasetFramework.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

LineageStore

@Inject
public LineageStore(co.cask.tephra.TransactionExecutorFactory executorFactory,
                   DatasetFramework datasetFramework)

LineageStore

public LineageStore(co.cask.tephra.TransactionExecutorFactory executorFactory,
            DatasetFramework datasetFramework,
            Id.DatasetInstance lineageDatasetId)

Method Detail

addAccess

public void addAccess(Id.Run run,
             Id.DatasetInstance datasetInstance,
             AccessType accessType,
             long accessTimeMillis)

Add a program-dataset access.

Parameters:

run - program run information

datasetInstance - dataset accessed by the program

accessType - access type

accessTimeMillis - time of access

addAccess

public void addAccess(Id.Run run,
             Id.DatasetInstance datasetInstance,
             AccessType accessType,
             long accessTimeMillis,
             @Nullable
             Id.NamespacedId component)

Add a program-dataset access.

Parameters:

run - program run information

datasetInstance - dataset accessed by the program

accessType - access type

accessTimeMillis - time of access

component - program component such as flowlet id, etc.

addAccess

public void addAccess(Id.Run run,
             Id.Stream stream,
             AccessType accessType,
             long accessTimeMillis)

Add a program-stream access.

Parameters:

run - program run information

stream - stream accessed by the program

accessType - access type

accessTimeMillis - time of access

addAccess

public void addAccess(Id.Run run,
             Id.Stream stream,
             AccessType accessType,
             long accessTimeMillis,
             @Nullable
             Id.NamespacedId component)

Add a program-stream access.

Parameters:

run - program run information

stream - stream accessed by the program

accessType - access type

accessTimeMillis - time of access

component - program component such as flowlet id, etc.

getEntitiesForRun

public Set<Id.NamespacedId> getEntitiesForRun(Id.Run run)

Returns:

a set of entities (program and data it accesses) associated with a program run.

getRelations

public Set<Relation> getRelations(Id.DatasetInstance datasetInstance,
                         long start,
                         long end,
                         com.google.common.base.Predicate<Relation> filter)

Fetch program-dataset access information for a dataset for a given period.

Parameters:

datasetInstance - dataset for which to fetch access information

start - start time period

end - end time period

filter - filter to be applied on result set

Returns:

program-dataset access information

getRelations

public Set<Relation> getRelations(Id.Stream stream,
                         long start,
                         long end,
                         com.google.common.base.Predicate<Relation> filter)

Fetch program-stream access information for a dataset for a given period.

Parameters:

stream - stream for which to fetch access information

start - start time period

end - end time period

filter - filter to be applied on result set

Returns:

program-stream access information

getRelations

public Set<Relation> getRelations(Id.Program program,
                         long start,
                         long end,
                         com.google.common.base.Predicate<Relation> filter)

Fetch program-dataset access information for a program for a given period.

Parameters:

program - program for which to fetch access information

start - start time period

end - end time period

filter - filter to be applied on result set

Returns:

program-dataset access information

getAccessTimesForRun

public List<Long> getAccessTimesForRun(Id.Run run)

Returns:

a set of access times (for program and data it accesses) associated with a program run.

setupDatasets

public static void setupDatasets(DatasetFramework framework)
                          throws IOException,
                                 DatasetManagementException

Adds datasets and types to the given DatasetFramework. Used by the upgrade tool to upgrade Lineage Dataset.

Parameters:

framework - framework to add types and datasets to

Throws:

IOException

DatasetManagementException