com.kamikaze.pfordelta

Class LCPForDelta

java.lang.Object

com.kamikaze.pfordelta.LCPForDelta

public class LCPForDelta
extends Object

Implementation of the optimized PForDelta algorithm for sorted integer arrays. The basic ideas are based on 1. Original algorithm from http://homepages.cwi.nl/~heman/downloads/msthesis.pdf 2. Optimization and variation from http://www2008.org/papers/pdf/p387-zhangA.pdf 3. Further optimization http://www2009.org/proceedings/pdf/p401.pdf As a part of the PForDelta implementation, Simple16 is used to compress exceptions. The original Simple16 algorithm can also be found in the above literature. This implementation overcomes the problem that Simple16 cannot deal with greater than 2^28 numbers. This implementation is almost same as PForDelta in the same package, except that it is tuned especially for Lucene-4.0 Codec to achieve the best performance in Lucene-4.0.

Author:

hao yan, hyan2008@gmail.com

Constructor Summary

Constructors

Constructor and Description
LCPForDelta()

Method Summary

Modifier and Type	Method and Description
static boolean	checkBigNumbers(int[] inputBlock, int blockSize, int bits) Check if the block contains big numbers that is greater than ((1<< bits)-1)
int	compress(int[] inBlock, int blockSize) Compress one block of blockSize integers using PForDelta with the optimal parameter b
int	compressOneBlockCore(int[] inputBlock, int blockSize, int bits) The core implementation of compressing a block with blockSize integers using PForDelta with the given parameter b
protected int	compressOneBlockCore2(int[] inputBlock, int blockSize, int bits)
static int	decompressBBitSlots(int[] outDecompSlots, int[] inCompBlock, int blockSize, int bits) Decompress b-bit slots
protected static int	decompressBBitSlotsWithHardCodes(int[] outDecompSlots, int[] inCompBlock, int blockSize, int bits)
protected static int	decompressBBitSlotsWithHardCodesWithIntBuffer(int[] outDecompSlots, IntBuffer inCompBlock, int blockSize, int bits)
static int	decompressBlockByS16(int[] outDecompBlock, int[] inCompBlock, int inStartOffsetInBits, int blockSize) Decompress a block of blockSize integers using Simple16 algorithm
protected static void	decompressBlockByS16WithIntBuffer(int[] outDecompBlock, IntBuffer inCompBlock, int blockSize)
protected static void	decompressBlockByS16WithIntBufferIntegrated(int[] outDecompBlock, IntBuffer inCompBlock, int blockSize, int[] expPosBuffer, int oribits)
static void	decompressOneBlock(int[] decompBlock, int[] inBlock, int blockSize) Decompress one block using PForDelta
protected static void	decompressOneBlockWithSize(int[] decompBlock, int[] inBlock, int blockSize, int[] expPosBuffer, int[] expHighBitsBuffer, int inBlockLen)
protected static void	decompressOneBlockWithSizeWithIntBuffer(int[] decompBlock, IntBuffer inBlock, int blockSize, int[] expPosBuffer, int[] expHighBitsBuffer, int inBlockLen)
static int	estimateCompressedSize(int[] inputBlock, int blockSize, int bits) Estimate the compressed size in ints of a block
protected int[]	getCompBuffer()
static int	readBits(int[] in, int inOffset, int bits) Read a certain number of bits of an integer into an integer array starting from the given start offset
protected static int	readBitsWithBuffer(int[] in, int inOffset, int bits)
protected void	setCompBuffer(int[] buffer)
static void	writeBits(int[] out, int val, int outOffset, int bits) Write a certain number of bits of an integer into an integer array starting from the given start offset

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

LCPForDelta

public LCPForDelta()

Method Detail

getCompBuffer

protected int[] getCompBuffer()

setCompBuffer

protected void setCompBuffer(int[] buffer)

compress

public int compress(int[] inBlock,
                    int blockSize)

Compress one block of blockSize integers using PForDelta with the optimal parameter b

Parameters:

inBlock - the block to be compressed

blockSize - the block size

Returns:

the compressed block

compressOneBlockCore

public int compressOneBlockCore(int[] inputBlock,
                                int blockSize,
                                int bits)

The core implementation of compressing a block with blockSize integers using PForDelta with the given parameter b

Parameters:

inputBlock - the block to be compressed

blockSize - the block size

bits - the the value of the parameter b

Returns:

the size of the compressed block (the number of ints)

compressOneBlockCore2

protected int compressOneBlockCore2(int[] inputBlock,
                                    int blockSize,
                                    int bits)
                             throws IllegalArgumentException

Throws:

IllegalArgumentException

decompressOneBlock

public static void decompressOneBlock(int[] decompBlock,
                                      int[] inBlock,
                                      int blockSize)

Decompress one block using PForDelta

Parameters:

decompBlock - the block that was decompressed

inBlock - the block to be decompressed

blockSize - the number of elements in the decompressed block

decompressOneBlockWithSize

protected static void decompressOneBlockWithSize(int[] decompBlock,
                                                 int[] inBlock,
                                                 int blockSize,
                                                 int[] expPosBuffer,
                                                 int[] expHighBitsBuffer,
                                                 int inBlockLen)

decompressOneBlockWithSizeWithIntBuffer

protected static void decompressOneBlockWithSizeWithIntBuffer(int[] decompBlock,
                                                              IntBuffer inBlock,
                                                              int blockSize,
                                                              int[] expPosBuffer,
                                                              int[] expHighBitsBuffer,
                                                              int inBlockLen)

estimateCompressedSize

public static int estimateCompressedSize(int[] inputBlock,
                                         int blockSize,
                                         int bits)

Estimate the compressed size in ints of a block

Parameters:

inputBlock - the block to be compressed

bits - the value of the parameter b

blockSize - the block size

Returns:

the compressed size in ints

checkBigNumbers

public static boolean checkBigNumbers(int[] inputBlock,
                                      int blockSize,
                                      int bits)

Check if the block contains big numbers that is greater than ((1<< bits)-1)

Parameters:

inputBlock - the block to be compressed

bits - the numbers of bits to decide whether a number is a big number

blockSize - the block size

Returns:

true if there is any big numbers in the block

decompressBBitSlots

public static int decompressBBitSlots(int[] outDecompSlots,
                                      int[] inCompBlock,
                                      int blockSize,
                                      int bits)

Decompress b-bit slots

Parameters:

outDecompSlots - decompressed block which is the output

inCompBlock - the compressed block which is the input

blockSize - the block size

bits - the value of the parameter b

Returns:

the compressed size in bits of the data that has been decompressed

decompressBBitSlotsWithHardCodes

protected static int decompressBBitSlotsWithHardCodes(int[] outDecompSlots,
                                                      int[] inCompBlock,
                                                      int blockSize,
                                                      int bits)

decompressBBitSlotsWithHardCodesWithIntBuffer

protected static int decompressBBitSlotsWithHardCodesWithIntBuffer(int[] outDecompSlots,
                                                                   IntBuffer inCompBlock,
                                                                   int blockSize,
                                                                   int bits)

decompressBlockByS16

public static int decompressBlockByS16(int[] outDecompBlock,
                                       int[] inCompBlock,
                                       int inStartOffsetInBits,
                                       int blockSize)

Decompress a block of blockSize integers using Simple16 algorithm

Parameters:

outDecompBlock - the decompressed block which is the output

inCompBlock - the compressed block which is the input

blockSize - the block size

inStartOffsetInBits - the start offset in bits of the compressed block

Returns:

the compressed size in bits of the data that has been decompressed

decompressBlockByS16WithIntBuffer

protected static void decompressBlockByS16WithIntBuffer(int[] outDecompBlock,
                                                        IntBuffer inCompBlock,
                                                        int blockSize)

decompressBlockByS16WithIntBufferIntegrated

protected static void decompressBlockByS16WithIntBufferIntegrated(int[] outDecompBlock,
                                                                  IntBuffer inCompBlock,
                                                                  int blockSize,
                                                                  int[] expPosBuffer,
                                                                  int oribits)

writeBits

public static final void writeBits(int[] out,
                                   int val,
                                   int outOffset,
                                   int bits)

Write a certain number of bits of an integer into an integer array starting from the given start offset

Parameters:

out - the output array

val - the integer to be written

outOffset - the start offset in bits in the output array

bits - the number of bits to be written (bits greater or equal to 0)

readBits

public static final int readBits(int[] in,
                                 int inOffset,
                                 int bits)

Read a certain number of bits of an integer into an integer array starting from the given start offset

Parameters:

in - the input array

inOffset - the start offset in bits in the input array

bits - the number of bits to be read, unlike writeBits(), readBits() does not deal with bits==0 and thus bits must be greater than 0. When bits ==0, the calling functions will just skip the entire bits-bit slots without decoding them

Returns:

the bits bits of the input

readBitsWithBuffer

protected static final int readBitsWithBuffer(int[] in,
                                              int inOffset,
                                              int bits)