co.cask.cdap.data2.transaction.queue.hbase

Class ShardedHBaseQueueStrategy

java.lang.Object

co.cask.cdap.data2.transaction.queue.hbase.ShardedHBaseQueueStrategy

All Implemented Interfaces:

HBaseQueueStrategy, Closeable, AutoCloseable

public final class ShardedHBaseQueueStrategy
extends Object
implements HBaseQueueStrategy, Closeable

An implementation of HBaseQueueStrategy with sharded keys. The row key has structure of:

 row_key = <shard> <queue_prefix> <write_pointer> <counter>
 shard = <salt> <consumer_group_id> <consumer_instance_id>
 salt = First byte of MD5 of <consumer_group_id>, <consumer_instance_id> and <queue_name>
 consumer_group_id = 8 bytes long value of the target consumer group or 0 if it is FIFO
 consumer_instance_id = 4 bytes int value of target consumer instance id or -1 if FIFO
 queue_prefix = <name_hash> <queue_name>
 name_hash = First byte of MD5 of <queue_name>
 queue_name = flowlet_name + "/" + output_name
 write_pointer = 8 bytes long value of the write pointer of the transaction
 counter = 4 bytes int value of a monotonic increasing number assigned for each entry written in the same transaction
 
 

Field Summary

Fields

Modifier and Type	Field and Description
static int	PREFIX_BYTES

Constructor Summary

Constructors

Constructor and Description
ShardedHBaseQueueStrategy(HBaseTableUtil tableUtil, int distributorBuckets) Constructs a new instance with the given number of buckets for distributed scan.

Method Summary

Methods

Modifier and Type	Method and Description
void	close()
QueueScanner	createScanner(ConsumerConfig consumerConfig, org.apache.hadoop.hbase.client.HTable hTable, org.apache.hadoop.hbase.client.Scan scan, int numRows) Creates a QueueScanner from the given Scan on the HBase table.
byte[]	getActualRowKey(ConsumerConfig consumerConfig, byte[] originalRowKey) Creates the actual row key used for accessing the HBase table from the given queue entry row key.
void	getRowKeys(Iterable<ConsumerGroupConfig> consumerGroupConfigs, QueueEntry queueEntry, byte[] rowKeyPrefix, long writePointer, int counter, Collection<byte[]> rowKeys) Get all the row keys that the queue entries need to write to.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PREFIX_BYTES

public static final int PREFIX_BYTES

See Also:

Constant Field Values

Constructor Detail

ShardedHBaseQueueStrategy

public ShardedHBaseQueueStrategy(HBaseTableUtil tableUtil,
                         int distributorBuckets)

Constructs a new instance with the given number of buckets for distributed scan.

Method Detail

createScanner

public QueueScanner createScanner(ConsumerConfig consumerConfig,
                         org.apache.hadoop.hbase.client.HTable hTable,
                         org.apache.hadoop.hbase.client.Scan scan,
                         int numRows)
                           throws IOException

Description copied from interface: HBaseQueueStrategy

Creates a QueueScanner from the given Scan on the HBase table.

Specified by:

createScanner in interface HBaseQueueStrategy

hTable - HTable for talking to HBase

scan - The scan request

numRows - Maximum number of rows to scan for

Returns:

A QueueScanner that scans over the give table

Throws:

IOException

getActualRowKey

public byte[] getActualRowKey(ConsumerConfig consumerConfig,
                     byte[] originalRowKey)

Description copied from interface: HBaseQueueStrategy

Creates the actual row key used for accessing the HBase table from the given queue entry row key.

Specified by:

getActualRowKey in interface HBaseQueueStrategy

getRowKeys

public void getRowKeys(Iterable<ConsumerGroupConfig> consumerGroupConfigs,
              QueueEntry queueEntry,
              byte[] rowKeyPrefix,
              long writePointer,
              int counter,
              Collection<byte[]> rowKeys)

Description copied from interface: HBaseQueueStrategy

Get all the row keys that the queue entries need to write to.

Specified by:

getRowKeys in interface HBaseQueueStrategy

Parameters:

consumerGroupConfigs - Consumer groups' configurations for the queue

queueEntry - Entry to enqueue

rowKeyPrefix - Prefix for row keys

writePointer - The writer pointer of the current transaction

counter - The counter of the given entry in this transaction

rowKeys - Collection for storing all the row keys generated by this entry

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException