Package com.google.cloud.hadoop.gcsio

Interface GoogleCloudStorageFileSystem

All Known Implementing Classes:

GoogleCloudStorageFileSystemImpl

public interface GoogleCloudStorageFileSystem

Provides a POSIX like file system layered on top of Google Cloud Storage (GCS).

All file system aspects (eg, path) are encapsulated in this class, they are not exposed to the underlying layer. That is, all interactions with the underlying layer are strictly in terms of buckets and objects.

See Also:

Hadoop FileSystem specification.

Field Summary

Fields

Modifier and Type	Field	Description
static URI	GCS_ROOT
static String	SCHEME

Method Summary

Modifier and Type	Method	Description
void	close()	Releases resources used by this instance.
void	compose(List<URI> sources, URI destination, String contentType)	Composes inputs into a single GCS object.
default WritableByteChannel	create(URI path)	Creates and opens an object for writing.
WritableByteChannel	create(URI path, CreateFileOptions createOptions)	Creates and opens an object for writing.
void	delete(URI path, boolean recursive)	Deletes one or more items indicated by the given path.
boolean	exists(URI path)	Indicates whether the given item exists.
FileInfo	getFileInfo(URI path)	Gets information about the given path item.
FileInfo	getFileInfoObject(URI path)	Gets information about the given path item.
List<FileInfo>	getFileInfos(List<URI> paths)	Gets information about each path in the given list; more efficient than calling getFileInfo() on each path individually in a loop.
FileInfo	getFileInfoWithHint(URI path, GoogleCloudStorageFileSystemImpl.PathTypeHint pathTypeHint)	Gets information about the given path item with hint providing the path type (file vs directory).
GoogleCloudStorage	getGcs()	Retrieve our internal gcs.
GoogleCloudStorageFileSystemOptions	getOptions()	Retrieve the options that were used to create this GoogleCloudStorageFileSystem.
default List<FileInfo>	listFileInfo(URI path)	If the given path points to a directory then the information about its children is returned, otherwise information about the given file is returned.
List<FileInfo>	listFileInfo(URI path, ListFileOptions listOptions)	If the given path points to a directory then the information about its children is returned, otherwise information about the given file is returned.
default List<FileInfo>	listFileInfoForPrefix(URI prefix)	Equivalent to a recursive listing of prefix, except that prefix doesn't have to represent an actual object but can just be a partial prefix string.
List<FileInfo>	listFileInfoForPrefix(URI prefix, ListFileOptions listOptions)	Equivalent to a recursive listing of prefix, except that prefix doesn't have to represent an actual object but can just be a partial prefix string.
GoogleCloudStorage.ListPage<FileInfo>	listFileInfoForPrefixPage(URI prefix, ListFileOptions listOptions, String pageToken)	Equivalent to listFileInfoForPrefix(java.net.URI) but returns FileInfos listed by single request (1 page).
default GoogleCloudStorage.ListPage<FileInfo>	listFileInfoForPrefixPage(URI prefix, String pageToken)	Equivalent to listFileInfoForPrefix(java.net.URI) but returns FileInfos listed by single request (1 page).
default List<FileInfo>	listFileInfoStartingFrom(URI startsFrom)	Return all the files which are lexicographically equal or greater than the provided path.
List<FileInfo>	listFileInfoStartingFrom(URI startsFrom, ListFileOptions listOptions)	Return all the files which are lexicographically equal or greater than the provided path.
GoogleCloudStorage.ListPage<FolderInfo>	listFoldersInfoForPrefixPage(URI prefix, ListFolderOptions listFolderOptions, String pageToken)	Returns the list of folder resources in the prefix.
void	mkdir(URI path)	Creates a directory at the specified path.
void	mkdirs(URI path)	Creates a directory at the specified path.
SeekableByteChannel	open(FileInfo fileInfo, GoogleCloudStorageReadOptions readOptions)	Opens an object for reading using FileInfo.
default SeekableByteChannel	open(URI path)	Opens an object for reading.
SeekableByteChannel	open(URI path, GoogleCloudStorageReadOptions readOptions)	Opens an object for reading.
void	rename(URI src, URI dst)	Renames the given item's path.

Field Detail

SCHEME

static final String SCHEME

See Also:

Constant Field Values

GCS_ROOT

static final URI GCS_ROOT

Method Detail

create

default WritableByteChannel create(URI path)
                            throws IOException

Creates and opens an object for writing. If the object already exists, it is overwritten.

Parameters:

path - Object full path of the form gs://bucket/object-path.

Returns:

A channel for writing to the given object.

Throws:

IOException

create

WritableByteChannel create(URI path,
                           CreateFileOptions createOptions)
                    throws IOException

Creates and opens an object for writing.

Parameters:

path - Object full path of the form gs://bucket/object-path.

Returns:

A channel for writing to the given object.

Throws:

IOException

open

default SeekableByteChannel open(URI path)
                          throws IOException

Opens an object for reading.

Parameters:

path - Object full path of the form gs://bucket/object-path.

Returns:

A channel for reading from the given object.

Throws:

FileNotFoundException - if the given path does not exist.

IOException - if object exists but cannot be opened.

open

SeekableByteChannel open(URI path,
                         GoogleCloudStorageReadOptions readOptions)
                  throws IOException

Opens an object for reading.

Parameters:

path - Object full path of the form gs://bucket/object-path.

readOptions - Fine-grained options for behaviors of retries, buffering, etc.

Returns:

A channel for reading from the given object.

Throws:

FileNotFoundException - if the given path does not exist.

IOException - if object exists but cannot be opened.

open

SeekableByteChannel open(FileInfo fileInfo,
                         GoogleCloudStorageReadOptions readOptions)
                  throws IOException

Opens an object for reading using FileInfo.

Parameters:

fileInfo - contains metadata information about the file

readOptions - readOptions fine-grained options specifying things like retry settings, buffering, etc.

Returns:

Seekable Byte Channel to enable file open

Throws:

IOException - IOException on IO Error

delete

void delete(URI path,
            boolean recursive)
     throws IOException

Deletes one or more items indicated by the given path.

If path points to a directory:

• if recursive is true, all items under that path are recursively deleted followed by deletion of the directory.
• else,
  • the directory is deleted if it is empty,
  • else, an IOException is thrown.

The recursive parameter is ignored for a file.

Parameters:

path - Path of the item to delete.

recursive - If true, all sub-items are also deleted.

Throws:

FileNotFoundException - if the given path does not exist.

IOException

exists

boolean exists(URI path)
        throws IOException

Indicates whether the given item exists.

Parameters:

path - Path of the item to check.

Returns:

true if the given item exists, false otherwise.

Throws:

IOException

mkdirs

void mkdirs(URI path)
     throws IOException

Creates a directory at the specified path. Also creates any parent directories as necessary. Similar to 'mkdir -p' command.

Parameters:

path - Path of the directory to create.

Throws:

IOException

rename

void rename(URI src,
            URI dst)
     throws IOException

Renames the given item's path.

The operation is disallowed if any of the following is true:

• src equals GCS_ROOT
• src is a file and dst equals GCS_ROOT
• src does not exist
• dst is a file that already exists
• parent of the destination does not exist

Otherwise, the expected behavior is as follows:

• if src is a directory:
  • if dst is an existing file then disallowed
  • if dst is a directory then rename the directory

• if src is a file:
  • if dst is a file then rename the file.
  • if dst is a directory then similar to the previous case after appending src file-name to dst

Note: This function is very expensive to call for directories that have many sub-items.

Parameters:

src - Path of the item to rename.

dst - New path of the item.

Throws:

FileNotFoundException - if src does not exist.

IOException

compose

void compose(List<URI> sources,
             URI destination,
             String contentType)
      throws IOException

Composes inputs into a single GCS object. This performs a GCS Compose. Objects will be composed according to the order they appear in the input. The destination object, if already present, will be overwritten. Sources and destination are assumed to be in the same bucket.

Parameters:

sources - the list of URIs to be composed

destination - the resulting URI with composed sources

contentType - content-type of the composed object

Throws:

IOException - if the Compose operation was unsuccessful

listFileInfoForPrefix

default List<FileInfo> listFileInfoForPrefix(URI prefix)
                                      throws IOException

Equivalent to a recursive listing of prefix, except that prefix doesn't have to represent an actual object but can just be a partial prefix string. The 'authority' component of the prefix must be the complete authority, however; we can only list prefixes of objects, not buckets.

Parameters:

prefix - the prefix to use to list all matching objects.

Throws:

IOException

listFileInfoForPrefix

List<FileInfo> listFileInfoForPrefix(URI prefix,
                                     ListFileOptions listOptions)
                              throws IOException

Equivalent to a recursive listing of prefix, except that prefix doesn't have to represent an actual object but can just be a partial prefix string. The 'authority' component of the prefix must be the complete authority, however; we can only list prefixes of objects, not buckets.

Parameters:

prefix - the prefix to use to list all matching objects.

Throws:

IOException

listFileInfoForPrefixPage

default GoogleCloudStorage.ListPage<FileInfo> listFileInfoForPrefixPage(URI prefix,
                                                                        String pageToken)
                                                                 throws IOException

Equivalent to listFileInfoForPrefix(java.net.URI) but returns FileInfos listed by single request (1 page).

Parameters:

prefix - the prefix to use to list all matching objects.

pageToken - the page token to list

Throws:

IOException

listFileInfoForPrefixPage

GoogleCloudStorage.ListPage<FileInfo> listFileInfoForPrefixPage(URI prefix,
                                                                ListFileOptions listOptions,
                                                                String pageToken)
                                                         throws IOException

Equivalent to listFileInfoForPrefix(java.net.URI) but returns FileInfos listed by single request (1 page).

Parameters:

prefix - the prefix to use to list all matching objects.

pageToken - the page token to list

Throws:

IOException

listFileInfo

default List<FileInfo> listFileInfo(URI path)
                             throws IOException

If the given path points to a directory then the information about its children is returned, otherwise information about the given file is returned.

Parameters:

path - Given path.

Returns:

Information about a file or children of a directory.

Throws:

FileNotFoundException - if the given path does not exist.

IOException

listFileInfoStartingFrom

default List<FileInfo> listFileInfoStartingFrom(URI startsFrom)
                                         throws IOException

Return all the files which are lexicographically equal or greater than the provided path. This is an experimental API and can change without notice.

Parameters:

startsFrom - Given path.

Returns:

Information about all the files which satisfies the criteria.

Throws:

IOException

listFileInfo

List<FileInfo> listFileInfo(URI path,
                            ListFileOptions listOptions)
                     throws IOException

If the given path points to a directory then the information about its children is returned, otherwise information about the given file is returned.

Parameters:

path - Given path.

Returns:

Information about a file or children of a directory.

Throws:

FileNotFoundException - if the given path does not exist.

IOException

listFileInfoStartingFrom

List<FileInfo> listFileInfoStartingFrom(URI startsFrom,
                                        ListFileOptions listOptions)
                                 throws IOException

Return all the files which are lexicographically equal or greater than the provided path. This is an experimental API and can change without notice.

Parameters:

startsFrom - Given path.

Returns:

Information about all the files which satisfies the criteria.

Throws:

IOException

listFoldersInfoForPrefixPage

GoogleCloudStorage.ListPage<FolderInfo> listFoldersInfoForPrefixPage(URI prefix,
                                                                     ListFolderOptions listFolderOptions,
                                                                     String pageToken)
                                                              throws IOException

Returns the list of folder resources in the prefix. It lists all the folder resources

Parameters:

prefix - the prefix to use to list all matching folder resources.

pageToken - the page token to list

listFolderOptions - the page token to list

Throws:

IOException

getFileInfo

FileInfo getFileInfo(URI path)
              throws IOException

Gets information about the given path item.

Parameters:

path - The path we want information about.

Returns:

Information about the given path item.

Throws:

IOException

getFileInfoWithHint

FileInfo getFileInfoWithHint(URI path,
                             GoogleCloudStorageFileSystemImpl.PathTypeHint pathTypeHint)
                      throws IOException

Gets information about the given path item with hint providing the path type (file vs directory).

Parameters:

path - The path we want information about.

Returns:

Information about the given path item.

Throws:

IOException

getFileInfoObject

FileInfo getFileInfoObject(URI path)
                    throws IOException

Gets information about the given path item. Here path should be pointing to a gcs object and can't be a directory

Parameters:

path - The path we want information about.

Returns:

Information about the given path item.

Throws:

IOException

getFileInfos

List<FileInfo> getFileInfos(List<URI> paths)
                     throws IOException

Gets information about each path in the given list; more efficient than calling getFileInfo() on each path individually in a loop.

Parameters:

paths - List of paths.

Returns:

Information about each path in the given list.

Throws:

IOException

close

void close()

Releases resources used by this instance.

mkdir

void mkdir(URI path)
    throws IOException

Creates a directory at the specified path.

There are two conventions for using objects as directories in GCS. 1. An object of zero size with name ending in / 2. An object of zero size with name ending in _$folder$

#1 is the recommended convention by the GCS team. We use it when creating a directory.

However, some old tools still use #2. We will decide based on customer use cases whether to support #2 as well. For now, we only support #1.

Note that a bucket is always considered a directory. Doesn't create parent directories; normal use cases should only call mkdirs().

Throws:

IOException

getGcs

GoogleCloudStorage getGcs()

Retrieve our internal gcs.

getOptions

GoogleCloudStorageFileSystemOptions getOptions()

Retrieve the options that were used to create this GoogleCloudStorageFileSystem.