com.google.cloud.hadoop.gcsio

Class GoogleCloudStorageReadChannel

java.lang.Object

com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadChannel

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable, java.nio.channels.ByteChannel, java.nio.channels.Channel, java.nio.channels.ReadableByteChannel, java.nio.channels.SeekableByteChannel, java.nio.channels.WritableByteChannel

Direct Known Subclasses:

InMemoryObjectReadChannel

public class GoogleCloudStorageReadChannel
extends java.lang.Object
implements java.nio.channels.SeekableByteChannel

Provides seekable read access to GCS.

Field Summary

Fields

Modifier and Type	Field and Description
protected long	contentChannelPosition
protected long	currentPosition
static int	DEFAULT_BACKOFF_INITIAL_INTERVAL_MILLIS Deprecated. use GoogleCloudStorageReadOptions.DEFAULT_BACKOFF_INITIAL_INTERVAL_MILLIS
static int	DEFAULT_BACKOFF_MAX_ELAPSED_TIME_MILLIS Deprecated. use GoogleCloudStorageReadOptions.DEFAULT_BACKOFF_MAX_ELAPSED_TIME_MILLIS
static int	DEFAULT_BACKOFF_MAX_INTERVAL_MILLIS Deprecated. use GoogleCloudStorageReadOptions.DEFAULT_BACKOFF_MAX_INTERVAL_MILLIS
static double	DEFAULT_BACKOFF_MULTIPLIER Deprecated. use GoogleCloudStorageReadOptions.DEFAULT_BACKOFF_MULTIPLIER instead
static double	DEFAULT_BACKOFF_RANDOMIZATION_FACTOR Deprecated. use GoogleCloudStorageReadOptions.DEFAULT_BACKOFF_RANDOMIZATION_FACTOR
protected boolean	metadataInitialized

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	GoogleCloudStorageReadChannel(GoogleCloudStorageReadOptions readOptions) Used for unit testing only.
	GoogleCloudStorageReadChannel(com.google.api.services.storage.Storage gcs, java.lang.String bucketName, java.lang.String objectName, ApiErrorExtractor errorExtractor, ClientRequestHelper<com.google.api.services.storage.model.StorageObject> requestHelper) Constructs an instance of GoogleCloudStorageReadChannel.
	GoogleCloudStorageReadChannel(com.google.api.services.storage.Storage gcs, java.lang.String bucketName, java.lang.String objectName, ApiErrorExtractor errorExtractor, ClientRequestHelper<com.google.api.services.storage.model.StorageObject> requestHelper, GoogleCloudStorageReadOptions readOptions) Constructs an instance of GoogleCloudStorageReadChannel.

Method Summary

Modifier and Type	Method and Description
void	close() Closes this channel.
protected void	closeContentChannel() Closes the underlying ReadableByteChannel.
protected com.google.api.services.storage.Storage.Objects.Get	createRequest()
protected GoogleCloudStorageItemInfo	getInitialMetadata() Returns GoogleCloudStorageItemInfo used to initialize metadata in constructor or null if GoogleCloudStorageReadOptions.getFastFailOnNotFound() is set to false.
protected void	initMetadata(com.google.api.client.http.HttpHeaders headers) Initializes metadata (size, encoding, etc) from HTTP headers.
protected void	initMetadata(java.lang.String encoding, long sizeFromMetadata, java.lang.String generation) Initializes metadata (size, encoding, etc) from passed parameters.
boolean	isOpen() Tells whether this channel is open.
protected java.io.InputStream	openStream(long bytesToRead) Opens the underlying stream, sets its position to the currentPosition.
long	position() Returns this channel's current position.
java.nio.channels.SeekableByteChannel	position(long newPosition) Sets this channel's position.
int	read(java.nio.ByteBuffer buffer) Reads from this channel and stores read data in the given buffer.
void	setMaxRetries(int maxRetries) Sets the number of times to automatically retry by re-opening the underlying contentChannel whenever an exception occurs while reading from it.
protected void	setSize(long size) Sets size of this channel to the given value.
long	size() Returns size of the object to which this channel is connected.
java.nio.channels.SeekableByteChannel	truncate(long size)
protected void	validatePosition(long position) Validates that the given position is valid for this channel.
int	write(java.nio.ByteBuffer src)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

currentPosition

protected long currentPosition

contentChannelPosition

protected long contentChannelPosition

metadataInitialized

protected boolean metadataInitialized

DEFAULT_BACKOFF_INITIAL_INTERVAL_MILLIS

@Deprecated
public static final int DEFAULT_BACKOFF_INITIAL_INTERVAL_MILLIS

Deprecated. use GoogleCloudStorageReadOptions.DEFAULT_BACKOFF_INITIAL_INTERVAL_MILLIS

See Also:

Constant Field Values

DEFAULT_BACKOFF_RANDOMIZATION_FACTOR

@Deprecated
public static final double DEFAULT_BACKOFF_RANDOMIZATION_FACTOR

Deprecated. use GoogleCloudStorageReadOptions.DEFAULT_BACKOFF_RANDOMIZATION_FACTOR

See Also:

Constant Field Values

DEFAULT_BACKOFF_MULTIPLIER

@Deprecated
public static final double DEFAULT_BACKOFF_MULTIPLIER

Deprecated. use GoogleCloudStorageReadOptions.DEFAULT_BACKOFF_MULTIPLIER instead

See Also:

Constant Field Values

DEFAULT_BACKOFF_MAX_INTERVAL_MILLIS

@Deprecated
public static final int DEFAULT_BACKOFF_MAX_INTERVAL_MILLIS

Deprecated. use GoogleCloudStorageReadOptions.DEFAULT_BACKOFF_MAX_INTERVAL_MILLIS

See Also:

Constant Field Values

DEFAULT_BACKOFF_MAX_ELAPSED_TIME_MILLIS

@Deprecated
public static final int DEFAULT_BACKOFF_MAX_ELAPSED_TIME_MILLIS

Deprecated. use GoogleCloudStorageReadOptions.DEFAULT_BACKOFF_MAX_ELAPSED_TIME_MILLIS

See Also:

Constant Field Values

Constructor Detail

GoogleCloudStorageReadChannel

public GoogleCloudStorageReadChannel(com.google.api.services.storage.Storage gcs,
                                     java.lang.String bucketName,
                                     java.lang.String objectName,
                                     ApiErrorExtractor errorExtractor,
                                     ClientRequestHelper<com.google.api.services.storage.model.StorageObject> requestHelper)
                              throws java.io.IOException

Constructs an instance of GoogleCloudStorageReadChannel.

Parameters:

gcs - storage object instance

bucketName - name of the bucket containing the object to read

objectName - name of the object to read

requestHelper - a ClientRequestHelper used to set any extra headers

Throws:

java.io.IOException - on IO error

GoogleCloudStorageReadChannel

public GoogleCloudStorageReadChannel(com.google.api.services.storage.Storage gcs,
                                     java.lang.String bucketName,
                                     java.lang.String objectName,
                                     ApiErrorExtractor errorExtractor,
                                     ClientRequestHelper<com.google.api.services.storage.model.StorageObject> requestHelper,
                                     @Nonnull
                                     GoogleCloudStorageReadOptions readOptions)
                              throws java.io.IOException

Constructs an instance of GoogleCloudStorageReadChannel.

Parameters:

gcs - storage object instance

bucketName - name of the bucket containing the object to read

objectName - name of the object to read

requestHelper - a ClientRequestHelper used to set any extra headers

readOptions - fine-grained options specifying things like retry settings, buffering, etc. Could not be null.

Throws:

java.io.IOException - on IO error

GoogleCloudStorageReadChannel

protected GoogleCloudStorageReadChannel(@Nonnull
                                        GoogleCloudStorageReadOptions readOptions)
                                 throws java.io.IOException

Used for unit testing only. Do not use elsewhere.

Constructs an instance of GoogleCloudStorageReadChannel.

Parameters:

readOptions - fine-grained options specifying things like retry settings, buffering, etc. Could not be null.

Throws:

java.io.IOException - on IO error

Method Detail

getInitialMetadata

@Nullable
protected GoogleCloudStorageItemInfo getInitialMetadata()
                                                           throws java.io.IOException

Returns GoogleCloudStorageItemInfo used to initialize metadata in constructor or null if GoogleCloudStorageReadOptions.getFastFailOnNotFound() is set to false.

Throws:

java.io.IOException

setMaxRetries

public void setMaxRetries(int maxRetries)

Sets the number of times to automatically retry by re-opening the underlying contentChannel whenever an exception occurs while reading from it. The count of attempted retries is reset whenever at least one byte is successfully read, so this number of retries refers to retries made without achieving any forward progress.

read

public int read(java.nio.ByteBuffer buffer)
         throws java.io.IOException

Reads from this channel and stores read data in the given buffer.

On unexpected failure, will attempt to close the channel and clean up state.

Specified by:

read in interface java.nio.channels.ReadableByteChannel

Specified by:

read in interface java.nio.channels.SeekableByteChannel

Parameters:

buffer - buffer to read data into

Returns:

number of bytes read or -1 on end-of-stream

Throws:

java.io.IOException - on IO error

truncate

public java.nio.channels.SeekableByteChannel truncate(long size)
                                               throws java.io.IOException

Specified by:

truncate in interface java.nio.channels.SeekableByteChannel

Throws:

java.io.IOException

write

public int write(java.nio.ByteBuffer src)
          throws java.io.IOException

Specified by:

write in interface java.nio.channels.SeekableByteChannel

Specified by:

write in interface java.nio.channels.WritableByteChannel

Throws:

java.io.IOException

isOpen

public boolean isOpen()

Tells whether this channel is open.

Specified by:

isOpen in interface java.nio.channels.Channel

Returns:

a value indicating whether this channel is open

closeContentChannel

protected void closeContentChannel()

Closes the underlying ReadableByteChannel.

Catches and ignores all exceptions as there is not a lot the user can do to fix errors here and a new connection will be needed. Especially SSLExceptions since the there's a high probability that SSL connections would be broken in a way that causes Channel.close() itself to throw an exception, even though underlying sockets have already been cleaned up; close() on an SSLSocketImpl requires a shutdown handshake in order to shutdown cleanly, and if the connection has been broken already, then this is not possible, and the SSLSocketImpl was already responsible for performing local cleanup at the time the exception was raised.

close

public void close()
           throws java.io.IOException

Closes this channel.

Specified by:

close in interface java.io.Closeable

Specified by:

close in interface java.lang.AutoCloseable

Specified by:

close in interface java.nio.channels.Channel

Throws:

java.io.IOException - on IO error

position

public long position()
              throws java.io.IOException

Returns this channel's current position.

Specified by:

position in interface java.nio.channels.SeekableByteChannel

Returns:

this channel's current position

Throws:

java.io.IOException

position

public java.nio.channels.SeekableByteChannel position(long newPosition)
                                               throws java.io.IOException

Sets this channel's position.

This method will throw an exception if newPosition is greater than object size, which contradicts SeekableByteChannel contract. TODO(user): decide if this needs to be fixed.

Specified by:

position in interface java.nio.channels.SeekableByteChannel

Parameters:

newPosition - the new position, counting the number of bytes from the beginning.

Returns:

this channel instance

Throws:

java.io.FileNotFoundException - if the underlying object does not exist.

java.io.IOException - on IO error

size

public long size()
          throws java.io.IOException

Returns size of the object to which this channel is connected.

Note: this method will return -1 until metadata will be lazily initialized during first read(java.nio.ByteBuffer) method call.

Specified by:

size in interface java.nio.channels.SeekableByteChannel

Returns:

size of the object to which this channel is connected after metadata was initialized (during first read) or -1 otherwise.

Throws:

java.io.IOException - on IO error

setSize

protected void setSize(long size)

Sets size of this channel to the given value.

validatePosition

protected void validatePosition(long position)
                         throws java.io.IOException

Validates that the given position is valid for this channel.

Throws:

java.io.IOException

initMetadata

protected void initMetadata(com.google.api.client.http.HttpHeaders headers)
                     throws java.io.IOException

Initializes metadata (size, encoding, etc) from HTTP headers.

Throws:

java.io.IOException

initMetadata

protected void initMetadata(@Nullable
                            java.lang.String encoding,
                            long sizeFromMetadata,
                            @Nullable
                            java.lang.String generation)
                     throws java.io.IOException

Initializes metadata (size, encoding, etc) from passed parameters.

Throws:

java.io.IOException

openStream

protected java.io.InputStream openStream(long bytesToRead)
                                  throws java.io.IOException

Opens the underlying stream, sets its position to the currentPosition.

If the file encoding in GCS is gzip (and therefore the HTTP client will decompress it), the entire file is always requested and we seek to the position requested. If the file encoding is not gzip, only the remaining bytes to be read are requested from GCS.

Parameters:

bytesToRead - number of bytes to read from new stream. Ignored if GoogleCloudStorageReadOptions.getFadvise() is equal to GoogleCloudStorageReadOptions.Fadvise.SEQUENTIAL.

Throws:

java.io.IOException - on IO error

createRequest

protected com.google.api.services.storage.Storage.Objects.Get createRequest()
                                                                     throws java.io.IOException

Throws:

java.io.IOException