Package com.google.cloud.hadoop.gcsio.testing

Class InMemoryGoogleCloudStorage

java.lang.Object

com.google.cloud.hadoop.gcsio.testing.InMemoryGoogleCloudStorage

All Implemented Interfaces:

GoogleCloudStorage

public class InMemoryGoogleCloudStorage
extends Object
implements GoogleCloudStorage

InMemoryGoogleCloudStorage overrides the public methods of GoogleCloudStorage by implementing all the equivalent bucket/object semantics with local in-memory storage.

Nested Class Summary

Nested classes/interfaces inherited from interface com.google.cloud.hadoop.gcsio.GoogleCloudStorage

GoogleCloudStorage.ListPage<T>

Field Summary

Fields inherited from interface com.google.cloud.hadoop.gcsio.GoogleCloudStorage

LIST_MAX_RESULTS, MAX_COMPOSE_OBJECTS, MAX_RESULTS_UNLIMITED, PATH_DELIMITER

Constructor Summary

Constructors

Constructor	Description
InMemoryGoogleCloudStorage()
InMemoryGoogleCloudStorage(GoogleCloudStorageOptions storageOptions)
InMemoryGoogleCloudStorage(GoogleCloudStorageOptions storageOptions, com.google.api.client.util.Clock clock)

Method Summary

Modifier and Type	Method	Description
void	close()	Releases resources used by this instance.
void	compose(String bucketName, List<String> sources, String destination, String contentType)	Composes inputs into a single GCS object.
GoogleCloudStorageItemInfo	composeObjects(List<StorageResourceId> sources, StorageResourceId destination, CreateObjectOptions options)	Composes inputs into a single GCS object.
void	copy(String srcBucketName, List<String> srcObjectNames, String dstBucketName, List<String> dstObjectNames)	Copies metadata of the given objects.
WritableByteChannel	create(StorageResourceId resourceId, CreateObjectOptions options)	Creates and opens an object for writing.
void	createBucket(String bucketName, CreateBucketOptions options)	Creates a bucket.
void	createEmptyObject(StorageResourceId resourceId)	Creates an empty object, useful for placeholders representing, for example, directories.
void	createEmptyObject(StorageResourceId resourceId, CreateObjectOptions options)	Creates an empty object, useful for placeholders representing, for example, directories.
void	createEmptyObjects(List<StorageResourceId> resourceIds)	Creates a list of empty objects; see GoogleCloudStorage.createEmptyObject(StorageResourceId) for the single-item version of this method.
void	createEmptyObjects(List<StorageResourceId> resourceIds, CreateObjectOptions options)	Creates a list of empty objects; see GoogleCloudStorage.createEmptyObject(StorageResourceId) for the single-item version of this method.
void	createFolder(StorageResourceId resourceId, boolean recursive)	Creates a native GCS folder resource.
void	deleteBuckets(List<String> bucketNames)	Deletes a list of buckets.
void	deleteFolders(List<FolderInfo> folders)	Deletes the given folder resources.
void	deleteObjects(List<StorageResourceId> fullObjectNames)	Deletes the given objects.
GoogleCloudStorageItemInfo	getFolderInfo(StorageResourceId resourceId)	Gets the metadata for a single folder.
static GoogleCloudStorageOptions	getInMemoryGoogleCloudStorageOptions()
GoogleCloudStorageItemInfo	getItemInfo(StorageResourceId resourceId)	Gets information about an object or a bucket.
List<GoogleCloudStorageItemInfo>	getItemInfos(List<StorageResourceId> resourceIds)	Gets information about multiple objects and/or buckets.
GoogleCloudStorageOptions	getOptions()	Retrieve the options that were used to create this GoogleCloudStorage.
Map<String,Long>	getStatistics()	Retrieves the statistics associated with the GoogleCloudStorage instance.
boolean	isHnBucket(URI src)	Checks if resourceId belongs to a Hierarchical namespace enabled bucket.
List<GoogleCloudStorageItemInfo>	listBucketInfo()	Gets a list of GoogleCloudStorageItemInfo for all buckets of this project.
List<String>	listBucketNames()	Gets a list of names of buckets in this project.
GoogleCloudStorage.ListPage<FolderInfo>	listFolderInfoForPrefixPage(String bucketName, String objectNamePrefix, ListFolderOptions listFolderOptions, String pageToken)	Returns the list of folder resources.
List<GoogleCloudStorageItemInfo>	listObjectInfo(String bucketName, String objectNamePrefix, ListObjectOptions listOptions)	Lists GoogleCloudStorageItemInfo of objects contained in the given bucket and whose names begin with the given prefix.
GoogleCloudStorage.ListPage<GoogleCloudStorageItemInfo>	listObjectInfoPage(String bucketName, String objectNamePrefix, ListObjectOptions listOptions, String pageToken)	The same semantics as GoogleCloudStorage.listObjectInfo(java.lang.String, java.lang.String), but returns only result of single list request (1 page).
List<GoogleCloudStorageItemInfo>	listObjectInfoStartingFrom(String bucketName, String startOffset, ListObjectOptions listOptions)	Lists GoogleCloudStorageItemInfo of objects contained in the given bucket and their name is lexicographically greater than or equal the provided offset.
void	move(Map<StorageResourceId,StorageResourceId> sourceToDestinationObjectsMap)	Moves objects within the same bucket.
SeekableByteChannel	open(GoogleCloudStorageItemInfo itemInfo, GoogleCloudStorageReadOptions readOptions)	Opens an object for reading using GoogleCloudStorageItemInfo.
SeekableByteChannel	open(StorageResourceId resourceId, GoogleCloudStorageReadOptions readOptions)	Opens an object for reading.
void	renameHnFolder(URI src, URI dst)	Renames src to dst using the rename LRO API.
List<GoogleCloudStorageItemInfo>	updateItems(List<UpdatableItemInfo> itemInfoList)	Attempt to update metadata of the objects referenced within the passed itemInfo objects.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.google.cloud.hadoop.gcsio.GoogleCloudStorage

copy, create, createBucket, listObjectInfo, listObjectInfoPage, listObjectInfoStartingFrom, open, open

Constructor Detail

InMemoryGoogleCloudStorage

public InMemoryGoogleCloudStorage()

InMemoryGoogleCloudStorage

public InMemoryGoogleCloudStorage(GoogleCloudStorageOptions storageOptions)

InMemoryGoogleCloudStorage

public InMemoryGoogleCloudStorage(GoogleCloudStorageOptions storageOptions,
                                  com.google.api.client.util.Clock clock)

Method Detail

getInMemoryGoogleCloudStorageOptions

public static GoogleCloudStorageOptions getInMemoryGoogleCloudStorageOptions()

getOptions

public GoogleCloudStorageOptions getOptions()

Description copied from interface: GoogleCloudStorage

Retrieve the options that were used to create this GoogleCloudStorage.

Specified by:

getOptions in interface GoogleCloudStorage

create

public WritableByteChannel create(StorageResourceId resourceId,
                                  CreateObjectOptions options)
                           throws IOException

Description copied from interface: GoogleCloudStorage

Creates and opens an object for writing. The bucket must already exist. If resourceId contains a known generationId or a generationId of 0, it is used instead of any "overwrite" settings in the provided CreateObjectOptions (overwrite will only succeed if the provided generationId matches the existing object, where a generationId of 0 indicates no existing object expected).

Specified by:

create in interface GoogleCloudStorage

Parameters:

resourceId - identifies a StorageObject

options - Options to use when creating the object

Returns:

a channel for writing to the given object

Throws:

IOException - on IO error

createBucket

public void createBucket(String bucketName,
                         CreateBucketOptions options)
                  throws IOException

Description copied from interface: GoogleCloudStorage

Creates a bucket.

Specified by:

createBucket in interface GoogleCloudStorage

Parameters:

bucketName - name of the bucket to create

options - options to use when creating bucket

Throws:

IOException - on IO error

createEmptyObject

public void createEmptyObject(StorageResourceId resourceId)
                       throws IOException

Description copied from interface: GoogleCloudStorage

Creates an empty object, useful for placeholders representing, for example, directories. The bucket must already exist. If the object already exists, it is overwritten. See GoogleCloudStorage.create(StorageResourceId) for the behavior if StorageResourceId.getGenerationId() is explicitly set.

Specified by:

createEmptyObject in interface GoogleCloudStorage

Parameters:

resourceId - identifies a StorageObject

Throws:

IOException - on IO error

createEmptyObject

public void createEmptyObject(StorageResourceId resourceId,
                              CreateObjectOptions options)
                       throws IOException

Description copied from interface: GoogleCloudStorage

Creates an empty object, useful for placeholders representing, for example, directories. The bucket must already exist. If the object already exists, it is overwritten. See GoogleCloudStorage.create(StorageResourceId) for the behavior if StorageResourceId.getGenerationId() is explicitly set.

Specified by:

createEmptyObject in interface GoogleCloudStorage

Parameters:

resourceId - identifies a StorageObject

options - options to use when creating the object

Throws:

IOException - on IO error

createEmptyObjects

public void createEmptyObjects(List<StorageResourceId> resourceIds)
                        throws IOException

Description copied from interface: GoogleCloudStorage

Creates a list of empty objects; see GoogleCloudStorage.createEmptyObject(StorageResourceId) for the single-item version of this method. Implementations may use different flow than the single-item version for greater efficiency. See GoogleCloudStorage.create(StorageResourceId) for the behavior if StorageResourceId.getGenerationId() is explicitly set.

Specified by:

createEmptyObjects in interface GoogleCloudStorage

Throws:

IOException

createEmptyObjects

public void createEmptyObjects(List<StorageResourceId> resourceIds,
                               CreateObjectOptions options)
                        throws IOException

Description copied from interface: GoogleCloudStorage

Creates a list of empty objects; see GoogleCloudStorage.createEmptyObject(StorageResourceId) for the single-item version of this method. Implementations may use different flow than the single-item version for greater efficiency. See GoogleCloudStorage.create(StorageResourceId) for the behavior if StorageResourceId.getGenerationId() is explicitly set.

Specified by:

createEmptyObjects in interface GoogleCloudStorage

Throws:

IOException

open

public SeekableByteChannel open(StorageResourceId resourceId,
                                GoogleCloudStorageReadOptions readOptions)
                         throws IOException

Description copied from interface: GoogleCloudStorage

Opens an object for reading.

Specified by:

open in interface GoogleCloudStorage

Parameters:

resourceId - identifies a StorageObject

readOptions - Fine-grained options for behaviors of retries, buffering, etc.

Returns:

a channel for reading from the given object

Throws:

FileNotFoundException - if the given object does not exist

IOException - if object exists but cannot be opened

open

public SeekableByteChannel open(GoogleCloudStorageItemInfo itemInfo,
                                GoogleCloudStorageReadOptions readOptions)
                         throws IOException

Description copied from interface: GoogleCloudStorage

Opens an object for reading using GoogleCloudStorageItemInfo.

Specified by:

open in interface GoogleCloudStorage

Parameters:

itemInfo - identifies a StorageObject

readOptions - Fine-grained options for behaviors of retries, buffering, etc.

Returns:

a channel for reading from the given object

Throws:

FileNotFoundException - if the given object does not exist

IOException - if object exists but cannot be opened

deleteBuckets

public void deleteBuckets(List<String> bucketNames)
                   throws IOException

Description copied from interface: GoogleCloudStorage

Deletes a list of buckets. Does not throw any exception for "bucket not found" errors.

Specified by:

deleteBuckets in interface GoogleCloudStorage

Parameters:

bucketNames - name of the buckets to delete

Throws:

FileNotFoundException - if the given bucket does not exist

IOException - on IO error

deleteObjects

public void deleteObjects(List<StorageResourceId> fullObjectNames)
                   throws IOException

Description copied from interface: GoogleCloudStorage

Deletes the given objects. Does not throw any exception for "object not found" errors.

Specified by:

deleteObjects in interface GoogleCloudStorage

Parameters:

fullObjectNames - names of objects to delete with their respective bucketNames.

Throws:

FileNotFoundException - if the given object does not exist

IOException - if object exists but cannot be deleted

deleteFolders

public void deleteFolders(List<FolderInfo> folders)
                   throws IOException

Description copied from interface: GoogleCloudStorage

Deletes the given folder resources. Does not throw any exception for "folders not found" errors.

Specified by:

deleteFolders in interface GoogleCloudStorage

Parameters:

folders - names of folder resources to delete

Throws:

IOException - if folder exists but cannot be deleted

createFolder

public void createFolder(StorageResourceId resourceId,
                         boolean recursive)
                  throws IOException

Description copied from interface: GoogleCloudStorage

Creates a native GCS folder resource. Applicable only for Hierarchical Namespace (HNS) enabled buckets.

Specified by:

createFolder in interface GoogleCloudStorage

Parameters:

resourceId - The StorageResourceId of the folder to create.

recursive - If true, creates all nonexistent parent folders in the path.

Throws:

IOException - if the folder cannot be created, or if it already exists and is not a folder.

getFolderInfo

public GoogleCloudStorageItemInfo getFolderInfo(StorageResourceId resourceId)
                                         throws IOException

Description copied from interface: GoogleCloudStorage

Gets the metadata for a single folder.

Specified by:

getFolderInfo in interface GoogleCloudStorage

Parameters:

resourceId - The StorageResourceId of the folder.

Throws:

IOException - on IO error.

move

public void move(Map<StorageResourceId,StorageResourceId> sourceToDestinationObjectsMap)
          throws IOException

Description copied from interface: GoogleCloudStorage

Moves objects within the same bucket. Moving objects between different buckets is not allowed.

Specified by:

move in interface GoogleCloudStorage

Parameters:

sourceToDestinationObjectsMap - map of destination objects to be moved, keyed by source

Throws:

FileNotFoundException - if the source object or the destination bucket does not exist

IOException - in all other error cases

copy

public void copy(String srcBucketName,
                 List<String> srcObjectNames,
                 String dstBucketName,
                 List<String> dstObjectNames)
          throws IOException

Description copied from interface: GoogleCloudStorage

Copies metadata of the given objects. After the copy is successfully complete, each object blob is reachable by two different names. Copying between two different locations or between two different storage classes is not allowed.

Specified by:

copy in interface GoogleCloudStorage

Parameters:

srcBucketName - name of the bucket containing the objects to copy

srcObjectNames - names of the objects to copy

dstBucketName - name of the bucket to copy to

dstObjectNames - names of the objects after copy

Throws:

FileNotFoundException - if the source object or the destination bucket does not exist

IOException - in all other error cases

listBucketNames

public List<String> listBucketNames()

Description copied from interface: GoogleCloudStorage

Gets a list of names of buckets in this project.

Specified by:

listBucketNames in interface GoogleCloudStorage

listBucketInfo

public List<GoogleCloudStorageItemInfo> listBucketInfo()

Description copied from interface: GoogleCloudStorage

Gets a list of GoogleCloudStorageItemInfo for all buckets of this project. This is no more expensive than calling listBucketNames(), since the list API for buckets already retrieves all the relevant bucket metadata.

Specified by:

listBucketInfo in interface GoogleCloudStorage

listObjectInfoPage

public GoogleCloudStorage.ListPage<GoogleCloudStorageItemInfo> listObjectInfoPage(String bucketName,
                                                                                  String objectNamePrefix,
                                                                                  ListObjectOptions listOptions,
                                                                                  String pageToken)
                                                                           throws IOException

Description copied from interface: GoogleCloudStorage

The same semantics as GoogleCloudStorage.listObjectInfo(java.lang.String, java.lang.String), but returns only result of single list request (1 page).

Specified by:

listObjectInfoPage in interface GoogleCloudStorage

Parameters:

bucketName - bucket name

objectNamePrefix - object name prefix or null if all objects in the bucket are desired

listOptions - options to use when listing objects

pageToken - the page token

Returns:

GoogleCloudStorage.ListPage object with listed GoogleCloudStorageItemInfos and next page token if any

Throws:

IOException - on IO error

listFolderInfoForPrefixPage

public GoogleCloudStorage.ListPage<FolderInfo> listFolderInfoForPrefixPage(String bucketName,
                                                                           String objectNamePrefix,
                                                                           ListFolderOptions listFolderOptions,
                                                                           String pageToken)
                                                                    throws IOException

Description copied from interface: GoogleCloudStorage

Returns the list of folder resources. Applicable only for HN enabled bucket

Specified by:

listFolderInfoForPrefixPage in interface GoogleCloudStorage

Parameters:

bucketName - bucket name

objectNamePrefix - folder resource name prefix

listFolderOptions - options to use when listing folder resources

pageToken - the page token

Returns:

GoogleCloudStorage.ListPage folder resources with listed Folders

Throws:

IOException - on IO error

listObjectInfo

public List<GoogleCloudStorageItemInfo> listObjectInfo(String bucketName,
                                                       String objectNamePrefix,
                                                       ListObjectOptions listOptions)
                                                throws IOException

Description copied from interface: GoogleCloudStorage

Lists GoogleCloudStorageItemInfo of objects contained in the given bucket and whose names begin with the given prefix.

Note: Although GCS does not implement a file system, it treats objects that contain a delimiter (ListObjectOptions.getDelimiter()) as different from other objects when listing objects. This will be clearer with an example.

Consider a bucket with objects: o1, d1/, d1/o1, d1/o2

• With prefix == null and delimiter == /, we get: d1/, o1
• With prefix == null and delimiter == null, we get: o1, d1/, d1/o1, d1/o2

Thus when delimiter is null, the entire key name is considered an opaque string, otherwise only the part up to the first delimiter is considered.

Specified by:

listObjectInfo in interface GoogleCloudStorage

Parameters:

bucketName - bucket name

objectNamePrefix - object name prefix or null if all objects in the bucket are desired

listOptions - options to use when listing objects

Returns:

list of objects

Throws:

IOException - on IO error

listObjectInfoStartingFrom

public List<GoogleCloudStorageItemInfo> listObjectInfoStartingFrom(String bucketName,
                                                                   String startOffset,
                                                                   ListObjectOptions listOptions)
                                                            throws IOException

Description copied from interface: GoogleCloudStorage

Lists GoogleCloudStorageItemInfo of objects contained in the given bucket and their name is lexicographically greater than or equal the provided offset.

Note: As GCS doesn't implement a file system, directory is also treated as an object (if it's been created). This APi filters out all those directory object and maintain the order of items. This APi strictly expects delimiter in listOptions to be not set.

Consider a bucket with objects: o1, d1/, d1/o1, d1/o2

• With startOffset == "o1" , we get: o1
• With startOffset == "d1/" , we get: d1/o1 d1/o2
• With startOffset == "d1/"o1 , we get: d1/o1 d1/o2

  This is an experimental API and can change without notice.

Specified by:

listObjectInfoStartingFrom in interface GoogleCloudStorage

Parameters:

bucketName - bucket name

startOffset - offset sting, all object with name greater and equal will be listed.

listOptions - options to use when listing objects

Returns:

list of objects

Throws:

IOException - on IO error

renameHnFolder

public void renameHnFolder(URI src,
                           URI dst)
                    throws IOException

Description copied from interface: GoogleCloudStorage

Renames src to dst using the rename LRO API. This should be called only on an Hierarchical namespace enabled bucket.

Specified by:

renameHnFolder in interface GoogleCloudStorage

Parameters:

src - source path

dst - destination path

Throws:

IOException

isHnBucket

public boolean isHnBucket(URI src)
                   throws IOException

Description copied from interface: GoogleCloudStorage

Checks if resourceId belongs to a Hierarchical namespace enabled bucket. This takes a path and not the bucket name since the caller may not have permission to query the bucket.

Specified by:

isHnBucket in interface GoogleCloudStorage

Parameters:

src - Path for which the check need to be performed

Returns:

Throws:

IOException

getItemInfo

public GoogleCloudStorageItemInfo getItemInfo(StorageResourceId resourceId)
                                       throws IOException

Description copied from interface: GoogleCloudStorage

Gets information about an object or a bucket.

Specified by:

getItemInfo in interface GoogleCloudStorage

Parameters:

resourceId - identifies either root, a Bucket, or a StorageObject

Returns:

information about the given item

Throws:

IOException - on IO error

getItemInfos

public List<GoogleCloudStorageItemInfo> getItemInfos(List<StorageResourceId> resourceIds)
                                              throws IOException

Description copied from interface: GoogleCloudStorage

Gets information about multiple objects and/or buckets. Items that are "not found" will still have an entry in the returned list; exists() will return false for these entries. The return list will have the same ordering as the input arguments.

Specified by:

getItemInfos in interface GoogleCloudStorage

Parameters:

resourceIds - names of the GCS StorageObjects or Buckets for which to retrieve info.

Returns:

information about the given resourceIds.

Throws:

IOException - on IO error

updateItems

public List<GoogleCloudStorageItemInfo> updateItems(List<UpdatableItemInfo> itemInfoList)
                                             throws IOException

Description copied from interface: GoogleCloudStorage

Attempt to update metadata of the objects referenced within the passed itemInfo objects.

Specified by:

updateItems in interface GoogleCloudStorage

Returns:

Updated GoogleCloudStorageItemInfo objects for the referenced objects.

Throws:

IOException - on IO error

close

public void close()

Description copied from interface: GoogleCloudStorage

Releases resources used by this instance.

Specified by:

close in interface GoogleCloudStorage

compose

public void compose(String bucketName,
                    List<String> sources,
                    String destination,
                    String contentType)
             throws IOException

Description copied from interface: GoogleCloudStorage

Composes inputs into a single GCS object. This performs a GCS Compose. Objects will be composed according to the order they appear in the input. The destination object, if already present, will be overwritten. Sources and destination are assumed to be in the same bucket.

Specified by:

compose in interface GoogleCloudStorage

Parameters:

bucketName - name of the bucket containing the sources and the destination

sources - names of the objects to be composed

destination - name of the resulting object

contentType - content-type of composed object

Throws:

IOException - if the Compose operation was unsuccessful

composeObjects

public GoogleCloudStorageItemInfo composeObjects(List<StorageResourceId> sources,
                                                 StorageResourceId destination,
                                                 CreateObjectOptions options)
                                          throws IOException

Description copied from interface: GoogleCloudStorage

Composes inputs into a single GCS object. This performs a GCS Compose. Objects will be composed according to the order they appear in the input. The destination object will have metadata set according to options. Overwrite semantics for the destination object will follow the same semantics as GoogleCloudStorage.create(StorageResourceId, CreateObjectOptions). See GoogleCloudStorage.create(StorageResourceId) for the behavior if StorageResourceId.getGenerationId() is explicitly set. The bucket must be the same for all sources and the destination.

Specified by:

composeObjects in interface GoogleCloudStorage

Throws:

IOException

getStatistics

public Map<String,Long> getStatistics()

Description copied from interface: GoogleCloudStorage

Retrieves the statistics associated with the GoogleCloudStorage instance.

Specified by:

getStatistics in interface GoogleCloudStorage