me.lemire.integercompression

Class FastPFOR

java.lang.Object

me.lemire.integercompression.FastPFOR

All Implemented Interfaces:

IntegerCODEC, SkippableIntegerCODEC

public final class FastPFOR
extends Object
implements IntegerCODEC, SkippableIntegerCODEC

This is a patching scheme designed for speed. It encodes integers in blocks of integers within pages of up to 65536 integers. Note that it is important, to get good compression and good performance, to use sizeable blocks (greater than 1024 integers). For arrays containing a number of integers that is not divisible by BLOCK_SIZE, you should use it in conjunction with another CODEC: IntegerCODEC ic = new Composition(new FastPFOR(), new VariableByte()).

For details, please see:

Daniel Lemire and Leonid Boytsov, Decoding billions of integers per second through vectorization Software: Practice & Experience http://onlinelibrary.wiley.com/doi/10.1002/spe.2203/abstract http://arxiv.org/abs/1209.2137

For sufficiently compressible and long arrays, it is faster and better than other PFOR schemes.

Note that this does not use differential coding: if you are working on sorted lists, use IntegratedFastPFOR instead. For multi-threaded applications, each thread should use its own FastPFOR object.

Author:

Daniel Lemire

Field Summary

Fields

Modifier and Type	Field and Description
static int	BLOCK_SIZE
static int	DEFAULT_PAGE_SIZE

Constructor Summary

Constructors

Constructor and Description
FastPFOR() Construct the fastPFOR CODEC with default parameters.

Method Summary

Modifier and Type	Method and Description
void	compress(int[] in, IntWrapper inpos, int inlength, int[] out, IntWrapper outpos) Compress data from an array to another array.
void	headlessCompress(int[] in, IntWrapper inpos, int inlength, int[] out, IntWrapper outpos) Compress data in blocks of BLOCK_SIZE integers (if fewer than BLOCK_SIZE integers are provided, nothing is done).
void	headlessUncompress(int[] in, IntWrapper inpos, int inlength, int[] out, IntWrapper outpos, int mynvalue) Uncompress data in blocks of integers.
String	toString()
void	uncompress(int[] in, IntWrapper inpos, int inlength, int[] out, IntWrapper outpos) Uncompress data from an array to another array.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

DEFAULT_PAGE_SIZE

public static final int DEFAULT_PAGE_SIZE

See Also:

Constant Field Values

BLOCK_SIZE

public static final int BLOCK_SIZE

See Also:

Constant Field Values

Constructor Detail

FastPFOR

public FastPFOR()

Construct the fastPFOR CODEC with default parameters.

Method Detail

headlessCompress

public void headlessCompress(int[] in,
                             IntWrapper inpos,
                             int inlength,
                             int[] out,
                             IntWrapper outpos)

Compress data in blocks of BLOCK_SIZE integers (if fewer than BLOCK_SIZE integers are provided, nothing is done).

Specified by:

headlessCompress in interface SkippableIntegerCODEC

Parameters:

in - input array

inpos - location in the input array

inlength - how many integers to compress

out - output array

outpos - where to write in the output array

See Also:

IntegerCODEC.compress(int[], IntWrapper, int, int[], IntWrapper)

headlessUncompress

public void headlessUncompress(int[] in,
                               IntWrapper inpos,
                               int inlength,
                               int[] out,
                               IntWrapper outpos,
                               int mynvalue)

Uncompress data in blocks of integers. In this particular case, the inlength parameter is ignored: it is deduced from the compressed data.

Specified by:

headlessUncompress in interface SkippableIntegerCODEC

Parameters:

in - array containing data in compressed form

inpos - where to start reading in the array

inlength - length of the compressed data (ignored by some schemes)

out - array where to write the compressed output

outpos - where to write the compressed output in out

mynvalue - number of integers we want to decode, the actual number of integers decoded can be less

See Also:

IntegerCODEC.compress(int[], IntWrapper, int, int[], IntWrapper)

compress

public void compress(int[] in,
                     IntWrapper inpos,
                     int inlength,
                     int[] out,
                     IntWrapper outpos)

Description copied from interface: IntegerCODEC

Compress data from an array to another array. Both inpos and outpos are modified to represent how much data was read and written to if 12 ints (inlength = 12) are compressed to 3 ints, then inpos will be incremented by 12 while outpos will be incremented by 3 we use IntWrapper to pass the values by reference.

Specified by:

compress in interface IntegerCODEC

Parameters:

in - input array

inpos - location in the input array

inlength - how many integers to compress

out - output array

outpos - where to write in the output array

uncompress

public void uncompress(int[] in,
                       IntWrapper inpos,
                       int inlength,
                       int[] out,
                       IntWrapper outpos)

Description copied from interface: IntegerCODEC

Uncompress data from an array to another array. Both inpos and outpos parameters are modified to indicate new positions after read/write.

Specified by:

uncompress in interface IntegerCODEC

Parameters:

in - array containing data in compressed form

inpos - where to start reading in the array

inlength - length of the compressed data (ignored by some schemes)

out - array where to write the compressed output

outpos - where to write the compressed output in out

toString

public String toString()

Overrides:

toString in class Object