Package com.google.cloud.hadoop.gcsio

Interface GoogleCloudStorage

    • Field Detail

      • PATH_DELIMITER

        static final String PATH_DELIMITER
        Pseudo path delimiter.

        GCS does not implement full concept of file system paths but it does expose some notion of a delimiter that can be used with ${code Storage.Objects.List} to control which items are listed.

        See Also:
        Constant Field Values

      • MAX_RESULTS_UNLIMITED

        static final long MAX_RESULTS_UNLIMITED
        Value indicating all objects should be returned from GCSFileSystem. No limit i.e. get all possible items. Used while listing all files in a directory
        See Also:
        Constant Field Values

      • LIST_MAX_RESULTS

        static final long LIST_MAX_RESULTS
        Value indicates the maxResult returned by gcs List API.
        See Also:
        Constant Field Values

      • MAX_COMPOSE_OBJECTS

        static final int MAX_COMPOSE_OBJECTS
        The maximum number of objects that can be composed in one operation.
        See Also:
        Constant Field Values

    • Method Detail

      • create

        default WritableByteChannel create​(StorageResourceId resourceId)
                            throws IOException
        Creates and opens an object for writing. The bucket must already exist. If the object already exists and resourceId doesn't have a explicit generationId set, it is deleted. If a generationId is provided in resourceId, that is used in place of normal overwrite semantics (overwrite will only succeed if the provided generationId matches the existing object, where a generationId of 0 indicates no existing object expected).
        Parameters:
        resourceId - identifies a StorageObject
        Returns:
        a channel for writing to the given object
        Throws:
        IOException - on IO error

      • create

        WritableByteChannel create​(StorageResourceId resourceId,
                           CreateObjectOptions options)
                    throws IOException
        Creates and opens an object for writing. The bucket must already exist. If resourceId contains a known generationId or a generationId of 0, it is used instead of any "overwrite" settings in the provided CreateObjectOptions (overwrite will only succeed if the provided generationId matches the existing object, where a generationId of 0 indicates no existing object expected).
        Parameters:
        resourceId - identifies a StorageObject
        options - Options to use when creating the object
        Returns:
        a channel for writing to the given object
        Throws:
        IOException - on IO error

      • createBucket

        default void createBucket​(String bucketName)
                   throws IOException
        Creates a bucket.
        Parameters:
        bucketName - name of the bucket to create
        Throws:
        IOException - on IO error

      • createBucket

        void createBucket​(String bucketName,
                  CreateBucketOptions options)
           throws IOException
        Creates a bucket.
        Parameters:
        bucketName - name of the bucket to create
        options - options to use when creating bucket
        Throws:
        IOException - on IO error

      • createEmptyObject

        void createEmptyObject​(StorageResourceId resourceId)
                throws IOException
        Creates an empty object, useful for placeholders representing, for example, directories. The bucket must already exist. If the object already exists, it is overwritten. See create(StorageResourceId) for the behavior if StorageResourceId.getGenerationId() is explicitly set.
        Parameters:
        resourceId - identifies a StorageObject
        Throws:
        IOException - on IO error

      • createEmptyObject

        void createEmptyObject​(StorageResourceId resourceId,
                       CreateObjectOptions options)
                throws IOException
        Creates an empty object, useful for placeholders representing, for example, directories. The bucket must already exist. If the object already exists, it is overwritten. See create(StorageResourceId) for the behavior if StorageResourceId.getGenerationId() is explicitly set.
        Parameters:
        resourceId - identifies a StorageObject
        options - options to use when creating the object
        Throws:
        IOException - on IO error

      • deleteBuckets

        void deleteBuckets​(List<String> bucketNames)
            throws IOException
        Deletes a list of buckets. Does not throw any exception for "bucket not found" errors.
        Parameters:
        bucketNames - name of the buckets to delete
        Throws:
        FileNotFoundException - if the given bucket does not exist
        IOException - on IO error

      • deleteObjects

        void deleteObjects​(List<StorageResourceId> fullObjectNames)
            throws IOException
        Deletes the given objects. Does not throw any exception for "object not found" errors.
        Parameters:
        fullObjectNames - names of objects to delete with their respective bucketNames.
        Throws:
        FileNotFoundException - if the given object does not exist
        IOException - if object exists but cannot be deleted

      • createFolder

        void createFolder​(StorageResourceId folderId,
                  boolean recursive)
           throws IOException
        Creates a native GCS folder resource. Applicable only for Hierarchical Namespace (HNS) enabled buckets.
        Parameters:
        folderId - The StorageResourceId of the folder to create.
        recursive - If true, creates all nonexistent parent folders in the path.
        Throws:
        IOException - if the folder cannot be created, or if it already exists and is not a folder.

      • deleteFolders

        void deleteFolders​(List<FolderInfo> folders)
            throws IOException
        Deletes the given folder resources. Does not throw any exception for "folders not found" errors.
        Parameters:
        folders - names of folder resources to delete
        Throws:
        IOException - if folder exists but cannot be deleted

      • copy

        void copy​(String srcBucketName,
          List<String> srcObjectNames,
          String dstBucketName,
          List<String> dstObjectNames)
   throws IOException
        Copies metadata of the given objects. After the copy is successfully complete, each object blob is reachable by two different names. Copying between two different locations or between two different storage classes is not allowed.
        Parameters:
        srcBucketName - name of the bucket containing the objects to copy
        srcObjectNames - names of the objects to copy
        dstBucketName - name of the bucket to copy to
        dstObjectNames - names of the objects after copy
        Throws:
        FileNotFoundException - if the source object or the destination bucket does not exist
        IOException - in all other error cases

      • copy

        default void copy​(Map<StorageResourceId,​StorageResourceId> sourceToDestinationObjectsMap)
           throws IOException
        Copies metadata of the given objects. After the copy is successfully complete, each object blob is reachable by two different names. Copying between two different locations or between two different storage classes is not allowed.
        Parameters:
        sourceToDestinationObjectsMap - map of destination objects to be copied, keyed by source
        Throws:
        FileNotFoundException - if the source object or the destination bucket does not exist
        IOException - in all other error cases

      • move

        void move​(Map<StorageResourceId,​StorageResourceId> sourceToDestinationObjectsMap)
   throws IOException
        Moves objects within the same bucket. Moving objects between different buckets is not allowed.
        Parameters:
        sourceToDestinationObjectsMap - map of destination objects to be moved, keyed by source
        Throws:
        FileNotFoundException - if the source object or the destination bucket does not exist
        IOException - in all other error cases

      • isHnBucket

        boolean isHnBucket​(URI path)
            throws IOException
        Checks if resourceId belongs to a Hierarchical namespace enabled bucket. This takes a path and not the bucket name since the caller may not have permission to query the bucket.
        Parameters:
        path - Path for which the check need to be performed
        Returns:
        Throws:
        IOException

      • renameHnFolder

        void renameHnFolder​(URI src,
                    URI dst)
             throws IOException
        Renames src to dst using the rename LRO API. This should be called only on an Hierarchical namespace enabled bucket.
        Parameters:
        src - source path
        dst - destination path
        Throws:
        IOException

      • listBucketInfo

        List<GoogleCloudStorageItemInfo> listBucketInfo()
                                         throws IOException
        Gets a list of GoogleCloudStorageItemInfo for all buckets of this project. This is no more expensive than calling listBucketNames(), since the list API for buckets already retrieves all the relevant bucket metadata.
        Throws:
        IOException

      • listObjectInfo

        default List<GoogleCloudStorageItemInfo> listObjectInfo​(String bucketName,
                                                        String objectNamePrefix)
                                                 throws IOException
        Lists GoogleCloudStorageItemInfo of objects contained in the given bucket and whose names begin with the given prefix.

        Note: Although GCS does not implement a file system, it treats objects that contain a delimiter (ListObjectOptions.getDelimiter()) as different from other objects when listing objects. This will be clearer with an example.

        Consider a bucket with objects: o1, d1/, d1/o1, d1/o2

        • With prefix == null and delimiter == /, we get: d1/, o1
        • With prefix == null and delimiter == null, we get: o1, d1/, d1/o1, d1/o2

        Thus when delimiter is null, the entire key name is considered an opaque string, otherwise only the part up to the first delimiter is considered.

        Parameters:
        bucketName - bucket name
        objectNamePrefix - object name prefix or null if all objects in the bucket are desired
        Returns:
        list of objects
        Throws:
        IOException - on IO error

      • listObjectInfoStartingFrom

        default List<GoogleCloudStorageItemInfo> listObjectInfoStartingFrom​(String bucketName,
                                                                    String startOffset)
                                                             throws IOException
        Lists GoogleCloudStorageItemInfo of objects contained in the given bucket and their name is lexicographically greater than or equal the provided offset.

        Note: As GCS doesn't implement a file system, directory is also treated as an object (if it's been created). This APi filters out all those directory object and maintain the order of items. This APi strictly expects delimiter in listOptions to be not set.

        Consider a bucket with objects: o1, d1/, d1/o1, d1/o2

        • With startOffset == "o1" , we get: o1
        • With startOffset == "d1/" , we get: d1/o1 d1/o2
        • With startOffset == "d1/"o1 , we get: d1/o1 d1/o2

          This is an experimental API and can change without notice.

        Parameters:
        bucketName - bucket name
        startOffset - offset sting, all object with name greater and equal will be listed.
        Returns:
        list of objects
        Throws:
        IOException - on IO error

      • listObjectInfo

        List<GoogleCloudStorageItemInfo> listObjectInfo​(String bucketName,
                                                String objectNamePrefix,
                                                ListObjectOptions listOptions)
                                         throws IOException
        Lists GoogleCloudStorageItemInfo of objects contained in the given bucket and whose names begin with the given prefix.

        Note: Although GCS does not implement a file system, it treats objects that contain a delimiter (ListObjectOptions.getDelimiter()) as different from other objects when listing objects. This will be clearer with an example.

        Consider a bucket with objects: o1, d1/, d1/o1, d1/o2

        • With prefix == null and delimiter == /, we get: d1/, o1
        • With prefix == null and delimiter == null, we get: o1, d1/, d1/o1, d1/o2

        Thus when delimiter is null, the entire key name is considered an opaque string, otherwise only the part up to the first delimiter is considered.

        Parameters:
        bucketName - bucket name
        objectNamePrefix - object name prefix or null if all objects in the bucket are desired
        listOptions - options to use when listing objects
        Returns:
        list of objects
        Throws:
        IOException - on IO error

      • listObjectInfoStartingFrom

        List<GoogleCloudStorageItemInfo> listObjectInfoStartingFrom​(String bucketName,
                                                            String startOffset,
                                                            ListObjectOptions listOptions)
                                                     throws IOException
        Lists GoogleCloudStorageItemInfo of objects contained in the given bucket and their name is lexicographically greater than or equal the provided offset.

        Note: As GCS doesn't implement a file system, directory is also treated as an object (if it's been created). This APi filters out all those directory object and maintain the order of items. This APi strictly expects delimiter in listOptions to be not set.

        Consider a bucket with objects: o1, d1/, d1/o1, d1/o2

        • With startOffset == "o1" , we get: o1
        • With startOffset == "d1/" , we get: d1/o1 d1/o2
        • With startOffset == "d1/"o1 , we get: d1/o1 d1/o2

          This is an experimental API and can change without notice.

        Parameters:
        bucketName - bucket name
        startOffset - offset sting, all object with name greater and equal will be listed.
        listOptions - options to use when listing objects
        Returns:
        list of objects
        Throws:
        IOException - on IO error

      • getItemInfos

        List<GoogleCloudStorageItemInfo> getItemInfos​(List<StorageResourceId> resourceIds)
                                       throws IOException
        Gets information about multiple objects and/or buckets. Items that are "not found" will still have an entry in the returned list; exists() will return false for these entries. The return list will have the same ordering as the input arguments.
        Parameters:
        resourceIds - names of the GCS StorageObjects or Buckets for which to retrieve info.
        Returns:
        information about the given resourceIds.
        Throws:
        IOException - on IO error

      • compose

        void compose​(String bucketName,
             List<String> sources,
             String destination,
             String contentType)
      throws IOException
        Composes inputs into a single GCS object. This performs a GCS Compose. Objects will be composed according to the order they appear in the input. The destination object, if already present, will be overwritten. Sources and destination are assumed to be in the same bucket.
        Parameters:
        bucketName - name of the bucket containing the sources and the destination
        sources - names of the objects to be composed
        destination - name of the resulting object
        contentType - content-type of composed object
        Throws:
        IOException - if the Compose operation was unsuccessful

      • close

        void close()
        Releases resources used by this instance.