co.cask.cdap.data2.dataset2

Class InMemoryDatasetFramework

java.lang.Object

co.cask.cdap.data2.dataset2.InMemoryDatasetFramework

All Implemented Interfaces:

DatasetFramework

Direct Known Subclasses:

StaticDatasetFramework

public class InMemoryDatasetFramework
extends Object
implements DatasetFramework

A simple implementation of DatasetFramework that keeps its state in memory

Constructor Summary

Constructors

Constructor and Description
InMemoryDatasetFramework(DatasetDefinitionRegistryFactory registryFactory, CConfiguration configuration)
InMemoryDatasetFramework(DatasetDefinitionRegistryFactory registryFactory, Map<String,co.cask.cdap.api.dataset.module.DatasetModule> defaultModules, CConfiguration configuration)

Method Summary

Methods

Modifier and Type	Method and Description
void	addInstance(String datasetType, Id.DatasetInstance datasetInstanceId, co.cask.cdap.api.dataset.DatasetProperties props) Adds information about dataset instance to the system.
void	addModule(Id.DatasetModule moduleId, co.cask.cdap.api.dataset.module.DatasetModule module) Adds dataset types by adding dataset module to the system.
void	addModule(Id.DatasetModule moduleId, co.cask.cdap.api.dataset.module.DatasetModule module, Location jarLocation) Adds dataset types by adding dataset module to the system with a jar location containing all dataset classes needed by the module.
void	createNamespace(Id.Namespace namespaceId) Creates a namespace in the Storage Providers - HBase/LevelDB, Hive and HDFS/Local File System.
protected co.cask.cdap.api.dataset.module.DatasetDefinitionRegistry	createRegistry(LinkedHashSet<String> availableModuleClasses, ClassLoader classLoader)
void	deleteAllInstances(Id.Namespace namespaceId) Deletes all dataset instances in the specified namespace.
void	deleteAllModules(Id.Namespace namespaceId) Deletes dataset modules and its types in the specified namespace.
void	deleteInstance(Id.DatasetInstance datasetInstanceId) Deletes dataset instance from the system.
void	deleteModule(Id.DatasetModule moduleId) Deletes dataset module and its types from the system.
void	deleteNamespace(Id.Namespace namespaceId) Deletes a namespace in the Storage Providers - HBase/LevelDB, Hive and HDFS/Local File System.
<T extends co.cask.cdap.api.dataset.DatasetAdmin> T	getAdmin(Id.DatasetInstance datasetInstanceId, ClassLoader classLoader) Gets dataset instance admin to be used to perform administrative operations.
<T extends co.cask.cdap.api.dataset.DatasetAdmin> T	getAdmin(Id.DatasetInstance datasetInstanceId, ClassLoader classLoader, DatasetClassLoaderProvider classLoaderProvider) Gets dataset instance admin to be used to perform administrative operations.
protected LinkedHashSet<String>	getAvailableModuleClasses(Id.Namespace namespace)
<T extends co.cask.cdap.api.dataset.Dataset> T	getDataset(Id.DatasetInstance datasetInstanceId, Map<String,String> arguments, ClassLoader classLoader) Gets dataset to be used to perform data operations.
<T extends co.cask.cdap.api.dataset.Dataset> T	getDataset(Id.DatasetInstance datasetInstanceId, Map<String,String> arguments, ClassLoader parentClassLoader, DatasetClassLoaderProvider classLoaderProvider, Iterable<? extends Id> owners) Gets dataset to be used to perform data operations.
<T extends co.cask.cdap.api.dataset.Dataset> T	getDataset(Id.DatasetInstance datasetInstanceId, Map<String,String> arguments, ClassLoader classLoader, Iterable<? extends Id> owners) Gets dataset to be used to perform data operations.
co.cask.cdap.api.dataset.DatasetSpecification	getDatasetSpec(Id.DatasetInstance datasetInstanceId) Gets the DatasetSpecification for the specified dataset instance id
Collection<DatasetSpecificationSummary>	getInstances(Id.Namespace namespaceId) Get all dataset instances in the specified namespace
boolean	hasInstance(Id.DatasetInstance datasetInstanceId)
boolean	hasSystemType(String typeName) Checks if the specified type exists in the 'system' namespace
boolean	hasType(Id.DatasetType datasetTypeId) Checks if the specified type exists in the specified namespace
void	updateInstance(Id.DatasetInstance datasetInstanceId, co.cask.cdap.api.dataset.DatasetProperties props) Updates the existing dataset instance in the system.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

InMemoryDatasetFramework

public InMemoryDatasetFramework(DatasetDefinitionRegistryFactory registryFactory,
                        CConfiguration configuration)

InMemoryDatasetFramework

@Inject
public InMemoryDatasetFramework(DatasetDefinitionRegistryFactory registryFactory,
                               Map<String,co.cask.cdap.api.dataset.module.DatasetModule> defaultModules,
                               CConfiguration configuration)

Method Detail

addModule

public void addModule(Id.DatasetModule moduleId,
             co.cask.cdap.api.dataset.module.DatasetModule module)
               throws ModuleConflictException

Description copied from interface: DatasetFramework

Adds dataset types by adding dataset module to the system. Calling this method to add DatasetModule may result in tracing class dependencies if the DatasetModule is not a system dataset, which can takes couple seconds for the tracing. If the jar Location containing the DatasetModule is known, it's better to call DatasetFramework.addModule(Id.DatasetModule, DatasetModule, Location) instead.

Specified by:

addModule in interface DatasetFramework

Parameters:

moduleId - dataset module id

module - dataset module

Throws:

ModuleConflictException - when module with same name is already registered or this module registers a type with a same name as one of the already registered by another module types

addModule

public void addModule(Id.DatasetModule moduleId,
             co.cask.cdap.api.dataset.module.DatasetModule module,
             Location jarLocation)
               throws DatasetManagementException

Description copied from interface: DatasetFramework

Adds dataset types by adding dataset module to the system with a jar location containing all dataset classes needed by the module.

Specified by:

addModule in interface DatasetFramework

Parameters:

moduleId - dataset module id

module - dataset module

jarLocation - location of the jar file that contains the dataset classes needed by the module

Throws:

ModuleConflictException - when module with same name is already registered or this module registers a type with a same name as one of the already registered by another module types

DatasetManagementException - in case of problems

deleteModule

public void deleteModule(Id.DatasetModule moduleId)

Description copied from interface: DatasetFramework

Deletes dataset module and its types from the system.

Specified by:

deleteModule in interface DatasetFramework

Parameters:

moduleId - dataset module id

deleteAllModules

public void deleteAllModules(Id.Namespace namespaceId)
                      throws ModuleConflictException

Description copied from interface: DatasetFramework

Deletes dataset modules and its types in the specified namespace.

Specified by:

deleteAllModules in interface DatasetFramework

Parameters:

namespaceId - the Id.Namespace to delete all modules from.

Throws:

ModuleConflictException - when some of modules can't be deleted because of its dependant modules or instances

addInstance

public void addInstance(String datasetType,
               Id.DatasetInstance datasetInstanceId,
               co.cask.cdap.api.dataset.DatasetProperties props)
                 throws DatasetManagementException,
                        IOException

Description copied from interface: DatasetFramework

Adds information about dataset instance to the system. This uses DatasetDefinition.configure(String, DatasetProperties) method to build DatasetSpecification which describes dataset instance and later used to initialize DatasetAdmin and Dataset for the dataset instance.

Specified by:

addInstance in interface DatasetFramework

Parameters:

datasetType - dataset instance type name

datasetInstanceId - dataset instance name

props - dataset instance properties

Throws:

InstanceConflictException - if dataset instance with this name already exists

DatasetManagementException

IOException - when creation of dataset instance using its admin fails

updateInstance

public void updateInstance(Id.DatasetInstance datasetInstanceId,
                  co.cask.cdap.api.dataset.DatasetProperties props)
                    throws DatasetManagementException,
                           IOException

Description copied from interface: DatasetFramework

Updates the existing dataset instance in the system. This uses DatasetDefinition.configure(String, DatasetProperties) method to build DatasetSpecification with new properties, which describes dataset instance and DatasetAdmin is used to upgrade Dataset for the dataset instance.

Specified by:

updateInstance in interface DatasetFramework

Parameters:

datasetInstanceId - dataset instance name

props - dataset instance properties

Throws:

DatasetManagementException

IOException - when creation of dataset instance using its admin fails

getInstances

public Collection<DatasetSpecificationSummary> getInstances(Id.Namespace namespaceId)

Description copied from interface: DatasetFramework

Get all dataset instances in the specified namespace

Specified by:

getInstances in interface DatasetFramework

Parameters:

namespaceId - the specified namespace id

Returns:

a collection of DatasetSpecifications for all datasets in the specified namespace

getDatasetSpec

@Nullable
public co.cask.cdap.api.dataset.DatasetSpecification getDatasetSpec(Id.DatasetInstance datasetInstanceId)

Description copied from interface: DatasetFramework

Gets the DatasetSpecification for the specified dataset instance id

Specified by:

getDatasetSpec in interface DatasetFramework

Parameters:

datasetInstanceId - the Id.DatasetInstance for which the DatasetSpecification is desired

Returns:

DatasetSpecification of the dataset or null if dataset not not exist

hasInstance

public boolean hasInstance(Id.DatasetInstance datasetInstanceId)

Specified by:

hasInstance in interface DatasetFramework

Parameters:

datasetInstanceId - the Id.DatasetInstance to check for existence

Returns:

true if instance exists, false otherwise

hasSystemType

public boolean hasSystemType(String typeName)

Description copied from interface: DatasetFramework

Checks if the specified type exists in the 'system' namespace

Specified by:

hasSystemType in interface DatasetFramework

Returns:

true if type exists in the 'system' namespace, false otherwise

hasType

public boolean hasType(Id.DatasetType datasetTypeId)

Description copied from interface: DatasetFramework

Checks if the specified type exists in the specified namespace

Specified by:

hasType in interface DatasetFramework

Returns:

true if type exists in the specified namespace, false otherwise

deleteInstance

public void deleteInstance(Id.DatasetInstance datasetInstanceId)
                    throws DatasetManagementException,
                           IOException

Description copied from interface: DatasetFramework

Deletes dataset instance from the system.

Specified by:

deleteInstance in interface DatasetFramework

Parameters:

datasetInstanceId - dataset instance name

Throws:

InstanceConflictException - if dataset instance cannot be deleted because of its dependencies

DatasetManagementException

IOException - when deletion of dataset instance using its admin fails

deleteAllInstances

public void deleteAllInstances(Id.Namespace namespaceId)
                        throws DatasetManagementException,
                               IOException

Description copied from interface: DatasetFramework

Deletes all dataset instances in the specified namespace.

Specified by:

deleteAllInstances in interface DatasetFramework

Parameters:

namespaceId - the specified namespace id

Throws:

DatasetManagementException

IOException - when deletion of dataset instance using its admin fails

getAdmin

public <T extends co.cask.cdap.api.dataset.DatasetAdmin> T getAdmin(Id.DatasetInstance datasetInstanceId,
                                                           @Nullable
                                                           ClassLoader classLoader)
                                                         throws IOException

Description copied from interface: DatasetFramework

Gets dataset instance admin to be used to perform administrative operations. The given classloader must be able to load all classes needed to instantiate the dataset admin. This means if the system classloader is used, only system dataset admins can fetched.

Specified by:

getAdmin in interface DatasetFramework

Type Parameters:

T - dataset admin type

Parameters:

datasetInstanceId - dataset instance name

classLoader - classLoader to be used to load classes or null to use system classLoader

Returns:

instance of dataset admin or null if dataset instance of this name doesn't exist.

Throws:

IOException - when there's trouble to instantiate DatasetAdmin

getAdmin

@Nullable
public <T extends co.cask.cdap.api.dataset.DatasetAdmin> T getAdmin(Id.DatasetInstance datasetInstanceId,
                                                                    @Nullable
                                                                    ClassLoader classLoader,
                                                                    DatasetClassLoaderProvider classLoaderProvider)
                                                         throws IOException

Description copied from interface: DatasetFramework

Gets dataset instance admin to be used to perform administrative operations. The class loader provider is used get classloaders for any dataset modules used by the specified dataset admin. This is because the classloader(s) for a dataset admin may create some resources that need to be cleaned up on close.

Specified by:

getAdmin in interface DatasetFramework

Type Parameters:

T - dataset admin type

Parameters:

datasetInstanceId - dataset instance name

classLoader - parent classLoader to be used to load classes or null to use system classLoader

classLoaderProvider - provider to get classloaders for different dataset modules

Returns:

instance of dataset admin or null if dataset instance of this name doesn't exist.

Throws:

IOException - when there's trouble to instantiate DatasetAdmin

getDataset

public <T extends co.cask.cdap.api.dataset.Dataset> T getDataset(Id.DatasetInstance datasetInstanceId,
                                                        Map<String,String> arguments,
                                                        @Nullable
                                                        ClassLoader classLoader,
                                                        @Nullable
                                                        Iterable<? extends Id> owners)
                                                      throws IOException

Description copied from interface: DatasetFramework

Gets dataset to be used to perform data operations.

Specified by:

getDataset in interface DatasetFramework

Type Parameters:

T - dataset type to be returned

Parameters:

datasetInstanceId - dataset instance id

arguments - runtime arguments for the dataset instance

classLoader - classLoader to be used to load classes or null to use system classLoader

owners - owners of the dataset

Returns:

instance of dataset or null if dataset instance of this name doesn't exist.

Throws:

IOException - when there's trouble to instantiate Dataset

getDataset

public <T extends co.cask.cdap.api.dataset.Dataset> T getDataset(Id.DatasetInstance datasetInstanceId,
                                                        Map<String,String> arguments,
                                                        @Nullable
                                                        ClassLoader classLoader)
                                                      throws IOException

Description copied from interface: DatasetFramework

Gets dataset to be used to perform data operations.

Specified by:

getDataset in interface DatasetFramework

Type Parameters:

T - dataset type to be returned

Parameters:

datasetInstanceId - dataset instance id

arguments - runtime arguments for the dataset instance

classLoader - classLoader to be used to load classes or null to use system classLoader

Returns:

instance of dataset or null if dataset instance of this name doesn't exist.

Throws:

IOException - when there's trouble to instantiate Dataset

getDataset

@Nullable
public <T extends co.cask.cdap.api.dataset.Dataset> T getDataset(Id.DatasetInstance datasetInstanceId,
                                                                 @Nullable
                                                                 Map<String,String> arguments,
                                                                 @Nullable
                                                                 ClassLoader parentClassLoader,
                                                                 DatasetClassLoaderProvider classLoaderProvider,
                                                                 @Nullable
                                                                 Iterable<? extends Id> owners)
                                                      throws IOException

Description copied from interface: DatasetFramework

Gets dataset to be used to perform data operations. This one is used when the classloader(s) for a dataset may create some resources that need to be cleaned up on close.

Specified by:

getDataset in interface DatasetFramework

Type Parameters:

T - dataset type to be returned

Parameters:

datasetInstanceId - dataset instance id

arguments - runtime arguments for the dataset instance

parentClassLoader - parent classLoader to be used to load classes or null to use system classLoader

classLoaderProvider - provider to get classloaders for different dataset modules

owners - owners of the dataset

Returns:

instance of dataset or null if dataset instance of this name doesn't exist.

Throws:

IOException - when there's trouble to instantiate Dataset

createNamespace

public void createNamespace(Id.Namespace namespaceId)
                     throws DatasetManagementException

Description copied from interface: DatasetFramework

Creates a namespace in the Storage Providers - HBase/LevelDB, Hive and HDFS/Local File System.

Specified by:

createNamespace in interface DatasetFramework

Parameters:

namespaceId - the Id.Namespace to create

Throws:

DatasetManagementException

deleteNamespace

public void deleteNamespace(Id.Namespace namespaceId)
                     throws DatasetManagementException

Description copied from interface: DatasetFramework

Deletes a namespace in the Storage Providers - HBase/LevelDB, Hive and HDFS/Local File System.

Specified by:

deleteNamespace in interface DatasetFramework

Parameters:

namespaceId - the Id.Namespace to create

Throws:

DatasetManagementException

createRegistry

protected co.cask.cdap.api.dataset.module.DatasetDefinitionRegistry createRegistry(LinkedHashSet<String> availableModuleClasses,
                                                                       @Nullable
                                                                       ClassLoader classLoader)

getAvailableModuleClasses

protected LinkedHashSet<String> getAvailableModuleClasses(Id.Namespace namespace)