co.cask.cdap.data2.metadata.writer

Class LineageWriterDatasetFramework

java.lang.Object

co.cask.cdap.data2.metadata.writer.LineageWriterDatasetFramework

All Implemented Interfaces:

DatasetFramework, ProgramContextAware

public class LineageWriterDatasetFramework
extends Object
implements DatasetFramework, ProgramContextAware

DatasetFramework that also records lineage (program-dataset access) records.

Method Summary

Methods

Modifier and Type	Method and Description
void	addInstance(String datasetTypeName, Id.DatasetInstance datasetInstanceId, co.cask.cdap.api.dataset.DatasetProperties props) Adds information about dataset instance to the system.
void	addModule(Id.DatasetModule moduleId, co.cask.cdap.api.dataset.module.DatasetModule module) Adds dataset types by adding dataset module to the system.
void	addModule(Id.DatasetModule moduleId, co.cask.cdap.api.dataset.module.DatasetModule module, Location jarLocation) Adds dataset types by adding dataset module to the system with a jar location containing all dataset classes needed by the module.
void	createNamespace(Id.Namespace namespaceId) Creates a namespace in the Storage Providers - HBase/LevelDB, Hive and HDFS/Local File System.
void	deleteAllInstances(Id.Namespace namespaceId) Deletes all dataset instances in the specified namespace.
void	deleteAllModules(Id.Namespace namespaceId) Deletes dataset modules and its types in the specified namespace.
void	deleteInstance(Id.DatasetInstance datasetInstanceId) Deletes dataset instance from the system.
void	deleteModule(Id.DatasetModule moduleId) Deletes dataset module and its types from the system.
void	deleteNamespace(Id.Namespace namespaceId) Deletes a namespace in the Storage Providers - HBase/LevelDB, Hive and HDFS/Local File System.
<T extends co.cask.cdap.api.dataset.DatasetAdmin> T	getAdmin(Id.DatasetInstance datasetInstanceId, ClassLoader classLoader) Gets dataset instance admin to be used to perform administrative operations.
<T extends co.cask.cdap.api.dataset.DatasetAdmin> T	getAdmin(Id.DatasetInstance datasetInstanceId, ClassLoader classLoader, DatasetClassLoaderProvider classLoaderProvider) Gets dataset instance admin to be used to perform administrative operations.
<T extends co.cask.cdap.api.dataset.Dataset> T	getDataset(Id.DatasetInstance datasetInstanceId, Map<String,String> arguments, ClassLoader classLoader) Gets dataset to be used to perform data operations.
<T extends co.cask.cdap.api.dataset.Dataset> T	getDataset(Id.DatasetInstance datasetInstanceId, Map<String,String> arguments, ClassLoader classLoader, DatasetClassLoaderProvider classLoaderProvider, Iterable<? extends Id> owners) Gets dataset to be used to perform data operations.
<T extends co.cask.cdap.api.dataset.Dataset> T	getDataset(Id.DatasetInstance datasetInstanceId, Map<String,String> arguments, ClassLoader classLoader, Iterable<? extends Id> owners) Gets dataset to be used to perform data operations.
co.cask.cdap.api.dataset.DatasetSpecification	getDatasetSpec(Id.DatasetInstance datasetInstanceId) Gets the DatasetSpecification for the specified dataset instance id
Collection<DatasetSpecificationSummary>	getInstances(Id.Namespace namespaceId) Get all dataset instances in the specified namespace
boolean	hasInstance(Id.DatasetInstance datasetInstanceId)
boolean	hasSystemType(String typeName) Checks if the specified type exists in the 'system' namespace
boolean	hasType(Id.DatasetType datasetTypeId) Checks if the specified type exists in the specified namespace
void	initContext(Id.Run run) Initalize with program run information.
void	initContext(Id.Run run, Id.NamespacedId componentId) Intialize with program run and program component (i.e, flowet Id, etc.) information.
void	updateInstance(Id.DatasetInstance datasetInstanceId, co.cask.cdap.api.dataset.DatasetProperties props) Updates the existing dataset instance in the system.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

initContext

public void initContext(Id.Run run)

Description copied from interface: ProgramContextAware

Initalize with program run information.

Specified by:

initContext in interface ProgramContextAware

Parameters:

run - program run

initContext

public void initContext(Id.Run run,
               Id.NamespacedId componentId)

Description copied from interface: ProgramContextAware

Intialize with program run and program component (i.e, flowet Id, etc.) information.

Specified by:

initContext in interface ProgramContextAware

Parameters:

run - program run

componentId - program component

addModule

public void addModule(Id.DatasetModule moduleId,
             co.cask.cdap.api.dataset.module.DatasetModule module)
               throws DatasetManagementException

Description copied from interface: DatasetFramework

Adds dataset types by adding dataset module to the system. Calling this method to add DatasetModule may result in tracing class dependencies if the DatasetModule is not a system dataset, which can takes couple seconds for the tracing. If the jar Location containing the DatasetModule is known, it's better to call DatasetFramework.addModule(Id.DatasetModule, DatasetModule, Location) instead.

Specified by:

addModule in interface DatasetFramework

Parameters:

moduleId - dataset module id

module - dataset module

Throws:

ModuleConflictException - when module with same name is already registered or this module registers a type with a same name as one of the already registered by another module types

DatasetManagementException - in case of problems

addModule

public void addModule(Id.DatasetModule moduleId,
             co.cask.cdap.api.dataset.module.DatasetModule module,
             Location jarLocation)
               throws DatasetManagementException

Description copied from interface: DatasetFramework

Adds dataset types by adding dataset module to the system with a jar location containing all dataset classes needed by the module.

Specified by:

addModule in interface DatasetFramework

Parameters:

moduleId - dataset module id

module - dataset module

jarLocation - location of the jar file that contains the dataset classes needed by the module

Throws:

ModuleConflictException - when module with same name is already registered or this module registers a type with a same name as one of the already registered by another module types

DatasetManagementException - in case of problems

deleteModule

public void deleteModule(Id.DatasetModule moduleId)
                  throws DatasetManagementException

Description copied from interface: DatasetFramework

Deletes dataset module and its types from the system.

Specified by:

deleteModule in interface DatasetFramework

Parameters:

moduleId - dataset module id

Throws:

ModuleConflictException - when module cannot be deleted because of its dependant modules or instances

DatasetManagementException

deleteAllModules

public void deleteAllModules(Id.Namespace namespaceId)
                      throws DatasetManagementException

Description copied from interface: DatasetFramework

Deletes dataset modules and its types in the specified namespace.

Specified by:

deleteAllModules in interface DatasetFramework

Parameters:

namespaceId - the Id.Namespace to delete all modules from.

Throws:

ModuleConflictException - when some of modules can't be deleted because of its dependant modules or instances

DatasetManagementException

addInstance

public void addInstance(String datasetTypeName,
               Id.DatasetInstance datasetInstanceId,
               co.cask.cdap.api.dataset.DatasetProperties props)
                 throws DatasetManagementException,
                        IOException

Description copied from interface: DatasetFramework

Adds information about dataset instance to the system. This uses DatasetDefinition.configure(String, DatasetProperties) method to build DatasetSpecification which describes dataset instance and later used to initialize DatasetAdmin and Dataset for the dataset instance.

Specified by:

addInstance in interface DatasetFramework

Parameters:

datasetTypeName - dataset instance type name

datasetInstanceId - dataset instance name

props - dataset instance properties

Throws:

InstanceConflictException - if dataset instance with this name already exists

DatasetManagementException

IOException - when creation of dataset instance using its admin fails

updateInstance

public void updateInstance(Id.DatasetInstance datasetInstanceId,
                  co.cask.cdap.api.dataset.DatasetProperties props)
                    throws DatasetManagementException,
                           IOException

Description copied from interface: DatasetFramework

Updates the existing dataset instance in the system. This uses DatasetDefinition.configure(String, DatasetProperties) method to build DatasetSpecification with new properties, which describes dataset instance and DatasetAdmin is used to upgrade Dataset for the dataset instance.

Specified by:

updateInstance in interface DatasetFramework

Parameters:

datasetInstanceId - dataset instance name

props - dataset instance properties

Throws:

DatasetManagementException

IOException - when creation of dataset instance using its admin fails

getInstances

public Collection<DatasetSpecificationSummary> getInstances(Id.Namespace namespaceId)
                                                     throws DatasetManagementException

Description copied from interface: DatasetFramework

Get all dataset instances in the specified namespace

Specified by:

getInstances in interface DatasetFramework

Parameters:

namespaceId - the specified namespace id

Returns:

a collection of DatasetSpecifications for all datasets in the specified namespace

Throws:

DatasetManagementException

getDatasetSpec

@Nullable
public co.cask.cdap.api.dataset.DatasetSpecification getDatasetSpec(Id.DatasetInstance datasetInstanceId)
                                                             throws DatasetManagementException

Description copied from interface: DatasetFramework

Gets the DatasetSpecification for the specified dataset instance id

Specified by:

getDatasetSpec in interface DatasetFramework

Parameters:

datasetInstanceId - the Id.DatasetInstance for which the DatasetSpecification is desired

Returns:

DatasetSpecification of the dataset or null if dataset not not exist

Throws:

DatasetManagementException

hasInstance

public boolean hasInstance(Id.DatasetInstance datasetInstanceId)
                    throws DatasetManagementException

Specified by:

hasInstance in interface DatasetFramework

Parameters:

datasetInstanceId - the Id.DatasetInstance to check for existence

Returns:

true if instance exists, false otherwise

Throws:

DatasetManagementException

hasSystemType

public boolean hasSystemType(String typeName)
                      throws DatasetManagementException

Description copied from interface: DatasetFramework

Checks if the specified type exists in the 'system' namespace

Specified by:

hasSystemType in interface DatasetFramework

Returns:

true if type exists in the 'system' namespace, false otherwise

Throws:

DatasetManagementException

hasType

public boolean hasType(Id.DatasetType datasetTypeId)
                throws DatasetManagementException

Description copied from interface: DatasetFramework

Checks if the specified type exists in the specified namespace

Specified by:

hasType in interface DatasetFramework

Returns:

true if type exists in the specified namespace, false otherwise

Throws:

DatasetManagementException

deleteInstance

public void deleteInstance(Id.DatasetInstance datasetInstanceId)
                    throws DatasetManagementException,
                           IOException

Description copied from interface: DatasetFramework

Deletes dataset instance from the system.

Specified by:

deleteInstance in interface DatasetFramework

Parameters:

datasetInstanceId - dataset instance name

Throws:

InstanceConflictException - if dataset instance cannot be deleted because of its dependencies

DatasetManagementException

IOException - when deletion of dataset instance using its admin fails

deleteAllInstances

public void deleteAllInstances(Id.Namespace namespaceId)
                        throws DatasetManagementException,
                               IOException

Description copied from interface: DatasetFramework

Deletes all dataset instances in the specified namespace.

Specified by:

deleteAllInstances in interface DatasetFramework

Parameters:

namespaceId - the specified namespace id

Throws:

DatasetManagementException

IOException - when deletion of dataset instance using its admin fails

getAdmin

@Nullable
public <T extends co.cask.cdap.api.dataset.DatasetAdmin> T getAdmin(Id.DatasetInstance datasetInstanceId,
                                                                    @Nullable
                                                                    ClassLoader classLoader)
                                                         throws DatasetManagementException,
                                                                IOException

Description copied from interface: DatasetFramework

Gets dataset instance admin to be used to perform administrative operations. The given classloader must be able to load all classes needed to instantiate the dataset admin. This means if the system classloader is used, only system dataset admins can fetched.

Specified by:

getAdmin in interface DatasetFramework

Type Parameters:

T - dataset admin type

Parameters:

datasetInstanceId - dataset instance name

classLoader - classLoader to be used to load classes or null to use system classLoader

Returns:

instance of dataset admin or null if dataset instance of this name doesn't exist.

Throws:

DatasetManagementException - when there's trouble getting dataset meta info

IOException - when there's trouble to instantiate DatasetAdmin

getAdmin

@Nullable
public <T extends co.cask.cdap.api.dataset.DatasetAdmin> T getAdmin(Id.DatasetInstance datasetInstanceId,
                                                                    @Nullable
                                                                    ClassLoader classLoader,
                                                                    DatasetClassLoaderProvider classLoaderProvider)
                                                         throws DatasetManagementException,
                                                                IOException

Description copied from interface: DatasetFramework

Gets dataset instance admin to be used to perform administrative operations. The class loader provider is used get classloaders for any dataset modules used by the specified dataset admin. This is because the classloader(s) for a dataset admin may create some resources that need to be cleaned up on close.

Specified by:

getAdmin in interface DatasetFramework

Type Parameters:

T - dataset admin type

Parameters:

datasetInstanceId - dataset instance name

classLoader - parent classLoader to be used to load classes or null to use system classLoader

classLoaderProvider - provider to get classloaders for different dataset modules

Returns:

instance of dataset admin or null if dataset instance of this name doesn't exist.

Throws:

DatasetManagementException - when there's trouble getting dataset meta info

IOException - when there's trouble to instantiate DatasetAdmin

getDataset

@Nullable
public <T extends co.cask.cdap.api.dataset.Dataset> T getDataset(Id.DatasetInstance datasetInstanceId,
                                                                 @Nullable
                                                                 Map<String,String> arguments,
                                                                 @Nullable
                                                                 ClassLoader classLoader,
                                                                 @Nullable
                                                                 Iterable<? extends Id> owners)
                                                      throws DatasetManagementException,
                                                             IOException

Description copied from interface: DatasetFramework

Gets dataset to be used to perform data operations.

Specified by:

getDataset in interface DatasetFramework

Type Parameters:

T - dataset type to be returned

Parameters:

datasetInstanceId - dataset instance id

arguments - runtime arguments for the dataset instance

classLoader - classLoader to be used to load classes or null to use system classLoader

owners - owners of the dataset

Returns:

instance of dataset or null if dataset instance of this name doesn't exist.

Throws:

DatasetManagementException - when there's trouble getting dataset meta info

IOException - when there's trouble to instantiate Dataset

getDataset

@Nullable
public <T extends co.cask.cdap.api.dataset.Dataset> T getDataset(Id.DatasetInstance datasetInstanceId,
                                                                 @Nullable
                                                                 Map<String,String> arguments,
                                                                 @Nullable
                                                                 ClassLoader classLoader)
                                                      throws DatasetManagementException,
                                                             IOException

Description copied from interface: DatasetFramework

Gets dataset to be used to perform data operations.

Specified by:

getDataset in interface DatasetFramework

Type Parameters:

T - dataset type to be returned

Parameters:

datasetInstanceId - dataset instance id

arguments - runtime arguments for the dataset instance

classLoader - classLoader to be used to load classes or null to use system classLoader

Returns:

instance of dataset or null if dataset instance of this name doesn't exist.

Throws:

DatasetManagementException - when there's trouble getting dataset meta info

IOException - when there's trouble to instantiate Dataset

getDataset

@Nullable
public <T extends co.cask.cdap.api.dataset.Dataset> T getDataset(Id.DatasetInstance datasetInstanceId,
                                                                 @Nullable
                                                                 Map<String,String> arguments,
                                                                 @Nullable
                                                                 ClassLoader classLoader,
                                                                 DatasetClassLoaderProvider classLoaderProvider,
                                                                 @Nullable
                                                                 Iterable<? extends Id> owners)
                                                      throws DatasetManagementException,
                                                             IOException

Description copied from interface: DatasetFramework

Gets dataset to be used to perform data operations. This one is used when the classloader(s) for a dataset may create some resources that need to be cleaned up on close.

Specified by:

getDataset in interface DatasetFramework

Type Parameters:

T - dataset type to be returned

Parameters:

datasetInstanceId - dataset instance id

arguments - runtime arguments for the dataset instance

classLoader - parent classLoader to be used to load classes or null to use system classLoader

classLoaderProvider - provider to get classloaders for different dataset modules

owners - owners of the dataset

Returns:

instance of dataset or null if dataset instance of this name doesn't exist.

Throws:

DatasetManagementException - when there's trouble getting dataset meta info

IOException - when there's trouble to instantiate Dataset

createNamespace

public void createNamespace(Id.Namespace namespaceId)
                     throws DatasetManagementException

Description copied from interface: DatasetFramework

Creates a namespace in the Storage Providers - HBase/LevelDB, Hive and HDFS/Local File System.

Specified by:

createNamespace in interface DatasetFramework

Parameters:

namespaceId - the Id.Namespace to create

Throws:

DatasetManagementException

deleteNamespace

public void deleteNamespace(Id.Namespace namespaceId)
                     throws DatasetManagementException

Description copied from interface: DatasetFramework

Deletes a namespace in the Storage Providers - HBase/LevelDB, Hive and HDFS/Local File System.

Specified by:

deleteNamespace in interface DatasetFramework

Parameters:

namespaceId - the Id.Namespace to create

Throws:

DatasetManagementException