parquet.column.values.dictionary

Class DictionaryValuesWriter

java.lang.Object

parquet.column.values.ValuesWriter

parquet.column.values.dictionary.DictionaryValuesWriter

Direct Known Subclasses:

DictionaryValuesWriter.PlainBinaryDictionaryValuesWriter, DictionaryValuesWriter.PlainDoubleDictionaryValuesWriter, DictionaryValuesWriter.PlainFloatDictionaryValuesWriter, DictionaryValuesWriter.PlainIntegerDictionaryValuesWriter, DictionaryValuesWriter.PlainLongDictionaryValuesWriter

public abstract class DictionaryValuesWriter
extends ValuesWriter

Will attempt to encode values using a dictionary and fall back to plain encoding if the dictionary gets too big

Author:

Julien Le Dem

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	DictionaryValuesWriter.PlainBinaryDictionaryValuesWriter
static class	DictionaryValuesWriter.PlainDoubleDictionaryValuesWriter
static class	DictionaryValuesWriter.PlainFixedLenArrayDictionaryValuesWriter
static class	DictionaryValuesWriter.PlainFloatDictionaryValuesWriter
static class	DictionaryValuesWriter.PlainIntegerDictionaryValuesWriter
static class	DictionaryValuesWriter.PlainLongDictionaryValuesWriter

Field Summary

Fields

Modifier and Type	Field and Description
protected int	dictionaryByteSize
protected boolean	dictionaryTooBig
protected IntList	encodedValues
protected boolean	firstPage indicates if this is the first page being processed
protected int	lastUsedDictionaryByteSize
protected int	lastUsedDictionarySize
protected int	maxDictionaryByteSize
protected ValuesWriter	plainValuesWriter
protected long	rawDataByteSize

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	DictionaryValuesWriter(int maxDictionaryByteSize, int initialSize)
protected	DictionaryValuesWriter(int maxDictionaryByteSize, int initialSize, int fixedLength) Construct a DictionaryValuesWriter for fixed-length byte array values.

Method Summary

Methods

Modifier and Type	Method and Description
protected void	checkAndFallbackIfNeeded() check the size constraints of the dictionary and fail over to plain values encoding if threshold reached
protected abstract void	clearDictionaryContent() clear/free the underlying dictionary content
protected abstract void	fallBackDictionaryEncodedData()
long	getAllocatedSize()
long	getBufferedSize() used to decide if we want to work to the next page
BytesInput	getBytes()
protected abstract int	getDictionarySize()
Encoding	getEncoding() called after getBytes() and before reset()
String	memUsageString(String prefix)
void	reset() called after getBytes() to reset the current buffer and start writing the next page
void	resetDictionary() reset the dictionary when a new block starts

Methods inherited from class parquet.column.values.ValuesWriter

createDictionaryPage, writeBoolean, writeByte, writeBytes, writeDouble, writeFloat, writeInteger, writeLong

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

maxDictionaryByteSize

protected final int maxDictionaryByteSize

plainValuesWriter

protected final ValuesWriter plainValuesWriter

dictionaryTooBig

protected boolean dictionaryTooBig

dictionaryByteSize

protected int dictionaryByteSize

lastUsedDictionaryByteSize

protected int lastUsedDictionaryByteSize

lastUsedDictionarySize

protected int lastUsedDictionarySize

encodedValues

protected IntList encodedValues

rawDataByteSize

protected long rawDataByteSize

firstPage

protected boolean firstPage

indicates if this is the first page being processed

Constructor Detail

DictionaryValuesWriter

protected DictionaryValuesWriter(int maxDictionaryByteSize,
                      int initialSize)

Parameters:

maxDictionaryByteSize -

initialSize -

DictionaryValuesWriter

protected DictionaryValuesWriter(int maxDictionaryByteSize,
                      int initialSize,
                      int fixedLength)

Construct a DictionaryValuesWriter for fixed-length byte array values.

Parameters:

maxDictionaryByteSize -

initialSize -

fixedLength - Fixed length for byte arrays

Method Detail

checkAndFallbackIfNeeded

protected void checkAndFallbackIfNeeded()

check the size constraints of the dictionary and fail over to plain values encoding if threshold reached

fallBackDictionaryEncodedData

protected abstract void fallBackDictionaryEncodedData()

getBufferedSize

public long getBufferedSize()

Description copied from class: ValuesWriter

used to decide if we want to work to the next page

Specified by:

getBufferedSize in class ValuesWriter

Returns:

the size of the currently buffered data (in bytes)

getAllocatedSize

public long getAllocatedSize()

Specified by:

getAllocatedSize in class ValuesWriter

Returns:

the allocated size of the buffer ( > {@link #getBufferedMemorySize()() )

getBytes

public BytesInput getBytes()

Specified by:

getBytes in class ValuesWriter

Returns:

the bytes buffered so far to write to the current page

getEncoding

public Encoding getEncoding()

Description copied from class: ValuesWriter

called after getBytes() and before reset()

Specified by:

getEncoding in class ValuesWriter

Returns:

the encoding that was used to encode the bytes

reset

public void reset()

Description copied from class: ValuesWriter

called after getBytes() to reset the current buffer and start writing the next page

Specified by:

reset in class ValuesWriter

resetDictionary

public void resetDictionary()

Description copied from class: ValuesWriter

reset the dictionary when a new block starts

Overrides:

resetDictionary in class ValuesWriter

clearDictionaryContent

protected abstract void clearDictionaryContent()

clear/free the underlying dictionary content

getDictionarySize

protected abstract int getDictionarySize()

Returns:

size in items

memUsageString

public String memUsageString(String prefix)

Specified by:

memUsageString in class ValuesWriter