Package com.google.cloud.hadoop.gcsio

Class GoogleCloudStorageFileSystemImpl

java.lang.Object

com.google.cloud.hadoop.gcsio.GoogleCloudStorageFileSystemImpl

All Implemented Interfaces:

GoogleCloudStorageFileSystem

public class GoogleCloudStorageFileSystemImpl
extends Object
implements GoogleCloudStorageFileSystem

Provides FS semantics over GCS based on Objects API

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	GoogleCloudStorageFileSystemImpl.PathTypeHint

Field Summary

Fields

Modifier and Type	Field	Description
static ListFileOptions	DELETE_RENAME_LIST_OPTIONS

Fields inherited from interface com.google.cloud.hadoop.gcsio.GoogleCloudStorageFileSystem

GCS_ROOT, SCHEME

Constructor Summary

Constructors

Constructor	Description
GoogleCloudStorageFileSystemImpl(com.google.auth.Credentials credentials, GoogleCloudStorageFileSystemOptions options)	Constructs an instance of GoogleCloudStorageFileSystem.
GoogleCloudStorageFileSystemImpl(com.google.auth.Credentials credentials, Function<List<AccessBoundary>,String> downscopedAccessTokenFn, GoogleCloudStorageFileSystemOptions options)	Constructs an instance of GoogleCloudStorageFileSystem.
GoogleCloudStorageFileSystemImpl(GoogleCloudStorage gcs, GoogleCloudStorageFileSystemOptions options)
GoogleCloudStorageFileSystemImpl(CheckedFunction<GoogleCloudStorageOptions,GoogleCloudStorage,IOException> gcsFn, GoogleCloudStorageFileSystemOptions options)	Constructs a GoogleCloudStorageFilesystem based on an already-configured underlying GoogleCloudStorage gcs.

Method Summary

Modifier and Type	Method	Description
void	close()	Releases resources used by this instance.
void	compose(List<URI> sources, URI destination, String contentType)	Composes inputs into a single GCS object.
WritableByteChannel	create(URI path, CreateFileOptions createOptions)	Creates and opens an object for writing.
void	delete(URI path, boolean recursive)	Deletes one or more items indicated by the given path.
boolean	exists(URI path)	Indicates whether the given item exists.
FileInfo	getFileInfo(URI path)	Gets information about the given path item.
FileInfo	getFileInfoObject(URI path)	Gets information about the given path item.
List<FileInfo>	getFileInfos(List<URI> paths)	Gets information about each path in the given list; more efficient than calling getFileInfo() on each path individually in a loop.
FileInfo	getFileInfoWithHint(URI path, GoogleCloudStorageFileSystemImpl.PathTypeHint pathTypeHint)	Gets information about the given path item with hint providing the path type (file vs directory).
GoogleCloudStorage	getGcs()	Retrieve our internal gcs.
GoogleCloudStorageFileSystemOptions	getOptions()	Retrieve the options that were used to create this GoogleCloudStorageFileSystem.
List<FileInfo>	listFileInfo(URI path, ListFileOptions listOptions)	If the given path points to a directory then the information about its children is returned, otherwise information about the given file is returned.
List<FileInfo>	listFileInfoForPrefix(URI prefix, ListFileOptions listOptions)	Equivalent to a recursive listing of prefix, except that prefix doesn't have to represent an actual object but can just be a partial prefix string.
GoogleCloudStorage.ListPage<FileInfo>	listFileInfoForPrefixPage(URI prefix, ListFileOptions listOptions, String pageToken)	Equivalent to GoogleCloudStorageFileSystem.listFileInfoForPrefix(java.net.URI) but returns FileInfos listed by single request (1 page).
List<FileInfo>	listFileInfoStartingFrom(URI startsFrom, ListFileOptions listOptions)	Return all the files which are lexicographically equal or greater than the provided path.
GoogleCloudStorage.ListPage<FolderInfo>	listFoldersInfoForPrefixPage(URI prefix, ListFolderOptions listFolderOptions, String pageToken)	Returns the list of folder resources in the prefix.
void	mkdir(URI path)	Creates a directory at the specified path.
void	mkdirs(URI path)	Creates a directory at the specified path.
static CreateObjectOptions	objectOptionsFromFileOptions(CreateFileOptions options)	Convert CreateFileOptions to CreateObjectOptions.
SeekableByteChannel	open(FileInfo fileInfo, GoogleCloudStorageReadOptions readOptions)	Opens an object for reading using FileInfo.
SeekableByteChannel	open(URI path, GoogleCloudStorageReadOptions readOptions)	Opens an object for reading.
void	rename(URI src, URI dst)	Renames the given item's path.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.google.cloud.hadoop.gcsio.GoogleCloudStorageFileSystem

create, listFileInfo, listFileInfoForPrefix, listFileInfoForPrefixPage, listFileInfoStartingFrom, open

Field Detail

DELETE_RENAME_LIST_OPTIONS

public static final ListFileOptions DELETE_RENAME_LIST_OPTIONS

Constructor Detail

GoogleCloudStorageFileSystemImpl

public GoogleCloudStorageFileSystemImpl(com.google.auth.Credentials credentials,
                                        GoogleCloudStorageFileSystemOptions options)
                                 throws IOException

Constructs an instance of GoogleCloudStorageFileSystem.

Parameters:

credentials - OAuth2 credentials that allows access to GCS.

options - Options for how this filesystem should operate and configure its underlying storage.

Throws:

IOException

GoogleCloudStorageFileSystemImpl

public GoogleCloudStorageFileSystemImpl(com.google.auth.Credentials credentials,
                                        Function<List<AccessBoundary>,String> downscopedAccessTokenFn,
                                        GoogleCloudStorageFileSystemOptions options)
                                 throws IOException

Constructs an instance of GoogleCloudStorageFileSystem.

Parameters:

credentials - OAuth2 credentials that allows access to GCS.

downscopedAccessTokenFn - Function that generates downscoped access token.

options - Options for how this filesystem should operate and configure its underlying storage.

Throws:

IOException

GoogleCloudStorageFileSystemImpl

public GoogleCloudStorageFileSystemImpl(CheckedFunction<GoogleCloudStorageOptions,GoogleCloudStorage,IOException> gcsFn,
                                        GoogleCloudStorageFileSystemOptions options)
                                 throws IOException

Constructs a GoogleCloudStorageFilesystem based on an already-configured underlying GoogleCloudStorage gcs. Any options pertaining to GCS creation will be ignored.

Throws:

IOException

GoogleCloudStorageFileSystemImpl

public GoogleCloudStorageFileSystemImpl(GoogleCloudStorage gcs,
                                        GoogleCloudStorageFileSystemOptions options)

Method Detail

getOptions

public GoogleCloudStorageFileSystemOptions getOptions()

Description copied from interface: GoogleCloudStorageFileSystem

Retrieve the options that were used to create this GoogleCloudStorageFileSystem.

Specified by:

getOptions in interface GoogleCloudStorageFileSystem

objectOptionsFromFileOptions

public static CreateObjectOptions objectOptionsFromFileOptions(CreateFileOptions options)

Convert CreateFileOptions to CreateObjectOptions.

create

public WritableByteChannel create(URI path,
                                  CreateFileOptions createOptions)
                           throws IOException

Description copied from interface: GoogleCloudStorageFileSystem

Creates and opens an object for writing.

Specified by:

create in interface GoogleCloudStorageFileSystem

Parameters:

path - Object full path of the form gs://bucket/object-path.

Returns:

A channel for writing to the given object.

Throws:

IOException

open

public SeekableByteChannel open(URI path,
                                GoogleCloudStorageReadOptions readOptions)
                         throws IOException

Description copied from interface: GoogleCloudStorageFileSystem

Opens an object for reading.

Specified by:

open in interface GoogleCloudStorageFileSystem

Parameters:

path - Object full path of the form gs://bucket/object-path.

readOptions - Fine-grained options for behaviors of retries, buffering, etc.

Returns:

A channel for reading from the given object.

Throws:

FileNotFoundException - if the given path does not exist.

IOException - if object exists but cannot be opened.

open

public SeekableByteChannel open(FileInfo fileInfo,
                                GoogleCloudStorageReadOptions readOptions)
                         throws IOException

Description copied from interface: GoogleCloudStorageFileSystem

Opens an object for reading using FileInfo.

Specified by:

open in interface GoogleCloudStorageFileSystem

Parameters:

fileInfo - contains metadata information about the file

readOptions - readOptions fine-grained options specifying things like retry settings, buffering, etc.

Returns:

Seekable Byte Channel to enable file open

Throws:

IOException - IOException on IO Error

delete

public void delete(URI path,
                   boolean recursive)
            throws IOException

Description copied from interface: GoogleCloudStorageFileSystem

Deletes one or more items indicated by the given path.

If path points to a directory:

• if recursive is true, all items under that path are recursively deleted followed by deletion of the directory.
• else,
  • the directory is deleted if it is empty,
  • else, an IOException is thrown.

The recursive parameter is ignored for a file.

Specified by:

delete in interface GoogleCloudStorageFileSystem

Parameters:

path - Path of the item to delete.

recursive - If true, all sub-items are also deleted.

Throws:

FileNotFoundException - if the given path does not exist.

IOException

listFoldersInfoForPrefixPage

public GoogleCloudStorage.ListPage<FolderInfo> listFoldersInfoForPrefixPage(URI prefix,
                                                                            ListFolderOptions listFolderOptions,
                                                                            String pageToken)
                                                                     throws IOException

Returns the list of folder resources in the prefix. It lists all the folder resources

Specified by:

listFoldersInfoForPrefixPage in interface GoogleCloudStorageFileSystem

Parameters:

prefix - the prefix to use to list all matching folder resources.

pageToken - the page token to list

listFolderOptions - the page token to list

Throws:

IOException

exists

public boolean exists(URI path)
               throws IOException

Description copied from interface: GoogleCloudStorageFileSystem

Indicates whether the given item exists.

Specified by:

exists in interface GoogleCloudStorageFileSystem

Parameters:

path - Path of the item to check.

Returns:

true if the given item exists, false otherwise.

Throws:

IOException

mkdirs

public void mkdirs(URI path)
            throws IOException

Description copied from interface: GoogleCloudStorageFileSystem

Creates a directory at the specified path. Also creates any parent directories as necessary. Similar to 'mkdir -p' command.

Specified by:

mkdirs in interface GoogleCloudStorageFileSystem

Parameters:

path - Path of the directory to create.

Throws:

IOException

rename

public void rename(URI src,
                   URI dst)
            throws IOException

Description copied from interface: GoogleCloudStorageFileSystem

Renames the given item's path.

The operation is disallowed if any of the following is true:

• src equals GCS_ROOT
• src is a file and dst equals GCS_ROOT
• src does not exist
• dst is a file that already exists
• parent of the destination does not exist

Otherwise, the expected behavior is as follows:

• if src is a directory:
  • if dst is an existing file then disallowed
  • if dst is a directory then rename the directory

• if src is a file:
  • if dst is a file then rename the file.
  • if dst is a directory then similar to the previous case after appending src file-name to dst

Note: This function is very expensive to call for directories that have many sub-items.

Specified by:

rename in interface GoogleCloudStorageFileSystem

Parameters:

src - Path of the item to rename.

dst - New path of the item.

Throws:

FileNotFoundException - if src does not exist.

IOException

compose

public void compose(List<URI> sources,
                    URI destination,
                    String contentType)
             throws IOException

Description copied from interface: GoogleCloudStorageFileSystem

Composes inputs into a single GCS object. This performs a GCS Compose. Objects will be composed according to the order they appear in the input. The destination object, if already present, will be overwritten. Sources and destination are assumed to be in the same bucket.

Specified by:

compose in interface GoogleCloudStorageFileSystem

Parameters:

sources - the list of URIs to be composed

destination - the resulting URI with composed sources

contentType - content-type of the composed object

Throws:

IOException - if the Compose operation was unsuccessful

listFileInfoForPrefix

public List<FileInfo> listFileInfoForPrefix(URI prefix,
                                            ListFileOptions listOptions)
                                     throws IOException

Description copied from interface: GoogleCloudStorageFileSystem

Equivalent to a recursive listing of prefix, except that prefix doesn't have to represent an actual object but can just be a partial prefix string. The 'authority' component of the prefix must be the complete authority, however; we can only list prefixes of objects, not buckets.

Specified by:

listFileInfoForPrefix in interface GoogleCloudStorageFileSystem

Parameters:

prefix - the prefix to use to list all matching objects.

Throws:

IOException

listFileInfoForPrefixPage

public GoogleCloudStorage.ListPage<FileInfo> listFileInfoForPrefixPage(URI prefix,
                                                                       ListFileOptions listOptions,
                                                                       String pageToken)
                                                                throws IOException

Description copied from interface: GoogleCloudStorageFileSystem

Equivalent to GoogleCloudStorageFileSystem.listFileInfoForPrefix(java.net.URI) but returns FileInfos listed by single request (1 page).

Specified by:

listFileInfoForPrefixPage in interface GoogleCloudStorageFileSystem

Parameters:

prefix - the prefix to use to list all matching objects.

pageToken - the page token to list

Throws:

IOException

listFileInfo

public List<FileInfo> listFileInfo(URI path,
                                   ListFileOptions listOptions)
                            throws IOException

Description copied from interface: GoogleCloudStorageFileSystem

If the given path points to a directory then the information about its children is returned, otherwise information about the given file is returned.

Specified by:

listFileInfo in interface GoogleCloudStorageFileSystem

Parameters:

path - Given path.

Returns:

Information about a file or children of a directory.

Throws:

FileNotFoundException - if the given path does not exist.

IOException

listFileInfoStartingFrom

public List<FileInfo> listFileInfoStartingFrom(URI startsFrom,
                                               ListFileOptions listOptions)
                                        throws IOException

Description copied from interface: GoogleCloudStorageFileSystem

Return all the files which are lexicographically equal or greater than the provided path. This is an experimental API and can change without notice.

Specified by:

listFileInfoStartingFrom in interface GoogleCloudStorageFileSystem

Parameters:

startsFrom - Given path.

Returns:

Information about all the files which satisfies the criteria.

Throws:

IOException

getFileInfo

public FileInfo getFileInfo(URI path)
                     throws IOException

Description copied from interface: GoogleCloudStorageFileSystem

Gets information about the given path item.

Specified by:

getFileInfo in interface GoogleCloudStorageFileSystem

Parameters:

path - The path we want information about.

Returns:

Information about the given path item.

Throws:

IOException

getFileInfoWithHint

public FileInfo getFileInfoWithHint(URI path,
                                    GoogleCloudStorageFileSystemImpl.PathTypeHint pathTypeHint)
                             throws IOException

Description copied from interface: GoogleCloudStorageFileSystem

Gets information about the given path item with hint providing the path type (file vs directory).

Specified by:

getFileInfoWithHint in interface GoogleCloudStorageFileSystem

Parameters:

path - The path we want information about.

Returns:

Information about the given path item.

Throws:

IOException

getFileInfoObject

public FileInfo getFileInfoObject(URI path)
                           throws IOException

Description copied from interface: GoogleCloudStorageFileSystem

Gets information about the given path item. Here path should be pointing to a gcs object and can't be a directory

Specified by:

getFileInfoObject in interface GoogleCloudStorageFileSystem

Parameters:

path - The path we want information about.

Returns:

Information about the given path item.

Throws:

IOException

getFileInfos

public List<FileInfo> getFileInfos(List<URI> paths)
                            throws IOException

Description copied from interface: GoogleCloudStorageFileSystem

Gets information about each path in the given list; more efficient than calling getFileInfo() on each path individually in a loop.

Specified by:

getFileInfos in interface GoogleCloudStorageFileSystem

Parameters:

paths - List of paths.

Returns:

Information about each path in the given list.

Throws:

IOException

close

public void close()

Description copied from interface: GoogleCloudStorageFileSystem

Releases resources used by this instance.

Specified by:

close in interface GoogleCloudStorageFileSystem

mkdir

public void mkdir(URI path)
           throws IOException

Description copied from interface: GoogleCloudStorageFileSystem

Creates a directory at the specified path.

There are two conventions for using objects as directories in GCS. 1. An object of zero size with name ending in / 2. An object of zero size with name ending in _$folder$

#1 is the recommended convention by the GCS team. We use it when creating a directory.

However, some old tools still use #2. We will decide based on customer use cases whether to support #2 as well. For now, we only support #1.

Note that a bucket is always considered a directory. Doesn't create parent directories; normal use cases should only call mkdirs().

Specified by:

mkdir in interface GoogleCloudStorageFileSystem

Throws:

IOException

getGcs

public GoogleCloudStorage getGcs()

Description copied from interface: GoogleCloudStorageFileSystem

Retrieve our internal gcs.

Specified by:

getGcs in interface GoogleCloudStorageFileSystem