parquet.column.values.delta

Class DeltaBinaryPackingValuesWriter

java.lang.Object

parquet.column.values.ValuesWriter

parquet.column.values.delta.DeltaBinaryPackingValuesWriter

public class DeltaBinaryPackingValuesWriter
extends ValuesWriter

Write integers with delta encoding and binary packing The format is as follows:

   delta-binary-packing: <page-header> <block>*
     page-header := <block size in values> <number of miniblocks in a block> <total value count> <first value>
     block := <min delta> <list of bitwidths of miniblocks> <miniblocks>

     min delta : zig-zag var int encoded
     bitWidthsOfMiniBlock : 1 byte little endian
     blockSizeInValues,blockSizeInValues,totalValueCount,firstValue : unsigned varint
   
 

The algorithm and format is inspired by D. Lemire's paper: http://lemire.me/blog/archives/2012/09/12/fast-integer-compression-decoding-billions-of-integers-per-second/

Author:

Tianshuo Deng

Field Summary

Fields

Modifier and Type	Field and Description
static int	DEFAULT_NUM_BLOCK_VALUES
static int	DEFAULT_NUM_MINIBLOCKS
static int	MAX_BITWIDTH max bitwidth for a mini block, it is used to allocate miniBlockByteBuffer which is reused between flushes.

Constructor Summary

Constructors

Constructor and Description
DeltaBinaryPackingValuesWriter(int slabSize)
DeltaBinaryPackingValuesWriter(int blockSizeInValues, int miniBlockNum, int slabSize)

Method Summary

Methods

Modifier and Type	Method and Description
long	getAllocatedSize()
long	getBufferedSize() used to decide if we want to work to the next page
BytesInput	getBytes() getBytes will trigger flushing block buffer, DO NOT write after getBytes() is called without calling reset()
Encoding	getEncoding() called after getBytes() and before reset()
String	memUsageString(String prefix)
void	reset() called after getBytes() to reset the current buffer and start writing the next page
void	writeInteger(int v)

Methods inherited from class parquet.column.values.ValuesWriter

createDictionaryPage, resetDictionary, writeBoolean, writeByte, writeBytes, writeDouble, writeFloat, writeLong

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

MAX_BITWIDTH

public static final int MAX_BITWIDTH

max bitwidth for a mini block, it is used to allocate miniBlockByteBuffer which is reused between flushes.

See Also:

Constant Field Values

DEFAULT_NUM_BLOCK_VALUES

public static final int DEFAULT_NUM_BLOCK_VALUES

See Also:

Constant Field Values

DEFAULT_NUM_MINIBLOCKS

public static final int DEFAULT_NUM_MINIBLOCKS

See Also:

Constant Field Values

Constructor Detail

DeltaBinaryPackingValuesWriter

public DeltaBinaryPackingValuesWriter(int slabSize)

DeltaBinaryPackingValuesWriter

public DeltaBinaryPackingValuesWriter(int blockSizeInValues,
                              int miniBlockNum,
                              int slabSize)

Method Detail

getBufferedSize

public long getBufferedSize()

Description copied from class: ValuesWriter

used to decide if we want to work to the next page

Specified by:

getBufferedSize in class ValuesWriter

Returns:

the size of the currently buffered data (in bytes)

writeInteger

public void writeInteger(int v)

Overrides:

writeInteger in class ValuesWriter

getBytes

public BytesInput getBytes()

getBytes will trigger flushing block buffer, DO NOT write after getBytes() is called without calling reset()

Specified by:

getBytes in class ValuesWriter

Returns:

getEncoding

public Encoding getEncoding()

Description copied from class: ValuesWriter

called after getBytes() and before reset()

Specified by:

getEncoding in class ValuesWriter

Returns:

the encoding that was used to encode the bytes

reset

public void reset()

Description copied from class: ValuesWriter

called after getBytes() to reset the current buffer and start writing the next page

Specified by:

reset in class ValuesWriter

getAllocatedSize

public long getAllocatedSize()

Specified by:

getAllocatedSize in class ValuesWriter

Returns:

the allocated size of the buffer ( > {@link #getBufferedMemorySize()() )

memUsageString

public String memUsageString(String prefix)

Specified by:

memUsageString in class ValuesWriter