HyperLogLog (stream-lib 2.5.1 API)

java.lang.Object
- com.clearspring.analytics.stream.cardinality.HyperLogLog

All Implemented Interfaces:

ICardinality
```
public class HyperLogLog
extends Object
implements ICardinality
```
Java implementation of HyperLogLog (HLL) algorithm from this paper:
http://algo.inria.fr/flajolet/Publications/FlFuGaMe07.pdf
HLL is an improved version of LogLog that is capable of estimating the cardinality of a set with accuracy = 1.04/sqrt(m) where m = 2^b. So we can control accuracy vs space usage by increasing or decreasing b.
The main benefit of using HLL over LL is that it only requires 64% of the space that LL does to get the same accuracy.
This implementation implements a single counter. If a large (millions) number of counters are required you may want to refer to:
http://dsiutils.dsi.unimi.it/
It has a more complex implementation of HLL that supports multiple counters in a single object, drastically reducing the java overhead from creating a large number of objects.
This implementation leveraged a javascript implementation that Yammer has been working on:
https://github.com/yammer/probablyjs
Note that this implementation does not include the long range correction function defined in the original paper. Empirical evidence shows that the correction function causes more harm than good.

Users have different motivations to use different types of hashing functions. Rather than try to keep up with all available hash functions and to remove the concern of causing future binary incompatibilities this class allows clients to offer the value in hashed int or long form. This way clients are free to change their hash function on their own time line. We recommend using Google's Guava Murmur3_128 implementation as it provides good performance and speed when high precision is required. In our tests the 32bit MurmurHash function included in this project is faster and produces better results than the 32 bit murmur3 implementation google provides.

Nested Class Summary

Nested Classes
Modifier and Type Class and Description

static class HyperLogLog.Builder

protected static class HyperLogLog.HyperLogLogMergeException

Nested Classes
Modifier and Type	Class and Description
`static class`	`HyperLogLog.Builder`
`protected static class`	`HyperLogLog.HyperLogLogMergeException`

Constructor Summary

Constructors
Constructor and Description
`HyperLogLog(double rsd)` Create a new HyperLogLog instance using the specified standard deviation.
`HyperLogLog(int log2m)` Create a new HyperLogLog instance.
`HyperLogLog(int log2m, RegisterSet registerSet)` Creates a new HyperLogLog instance using the given registers.

Method Summary

Methods
Modifier and Type	Method and Description
`void`	`addAll(HyperLogLog other)` Add all the elements of the other set to this set.
`long`	`cardinality()`
`byte[]`	`getBytes()`
`ICardinality`	`merge(ICardinality... estimators)` Merges estimators to produce a new estimator for the combined streams of this estimator and those passed as arguments.
`boolean`	`offer(Object o)`
`boolean`	`offerHashed(int hashedValue)` Offer the value as a hashed long value
`boolean`	`offerHashed(long hashedValue)` Offer the value as a hashed long value
`int`	`sizeof()`

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Constructor Detail
  - HyperLogLog
```
public HyperLogLog(double rsd)
```
    Create a new HyperLogLog instance using the specified standard deviation.
    
    Parameters:
    rsd - - the relative standard deviation for the counter. smaller values create counters that require more space.
  - HyperLogLog
```
public HyperLogLog(int log2m)
```
    Create a new HyperLogLog instance. The log2m parameter defines the accuracy of the counter. The larger the log2m the better the accuracy.
    accuracy = 1.04/sqrt(2^log2m)
    
    Parameters:
    log2m - - the number of bits to use as the basis for the HLL instance
  - HyperLogLog
```
public HyperLogLog(int log2m,
           RegisterSet registerSet)
```
    Creates a new HyperLogLog instance using the given registers. Used for unmarshalling a serialized instance and for merging multiple counters together.
    
    Parameters:
    registerSet - - the initial values for the register set
- Method Detail
  - offerHashed
```
public boolean offerHashed(long hashedValue)
```
    Description copied from interface: ICardinality
    
    Offer the value as a hashed long value
    
    Specified by:
    
    offerHashed in interface ICardinality
    
    Parameters:
    hashedValue - - the hash of the item to offer to the estimator
    
    Returns:
    false if the value returned by cardinality() is unaffected by the appearance of hashedLong in the stream
  - offerHashed
```
public boolean offerHashed(int hashedValue)
```
    Description copied from interface: ICardinality
    
    Offer the value as a hashed long value
    
    Specified by:
    
    offerHashed in interface ICardinality
    
    Parameters:
    hashedValue - - the hash of the item to offer to the estimator
    
    Returns:
    false if the value returned by cardinality() is unaffected by the appearance of hashedInt in the stream
  - offer
```
public boolean offer(Object o)
```
    Specified by:
    
    offer in interface ICardinality
    
    Parameters:
    o - stream element
    
    Returns:
    false if the value returned by cardinality() is unaffected by the appearance of o in the stream.
  - cardinality
```
public long cardinality()
```
    Specified by:
    
    cardinality in interface ICardinality
    
    Returns:
    the number of unique elements in the stream or an estimate thereof
  - sizeof
```
public int sizeof()
```
    Specified by:
    
    sizeof in interface ICardinality
    
    Returns:
    size in bytes needed for serialization
  - getBytes
```
public byte[] getBytes()
                throws IOException
```
    Specified by:
    
    getBytes in interface ICardinality
    
    Returns:
    
    Throws:
    
    IOException
  - addAll
```
public void addAll(HyperLogLog other)
            throws CardinalityMergeException
```
    Add all the elements of the other set to this set. This operation does not imply a loss of precision.
    
    Parameters:
    other - A compatible Hyperloglog instance (same log2m)
    
    Throws:
    
    CardinalityMergeException - if other is not compatible
  - merge
```
public ICardinality merge(ICardinality... estimators)
                   throws CardinalityMergeException
```
    Description copied from interface: ICardinality
    
    Merges estimators to produce a new estimator for the combined streams of this estimator and those passed as arguments. Nor this estimator nor the one passed as parameters are modified.
    
    Specified by:
    
    merge in interface ICardinality
    
    Parameters:
    estimators - Zero or more compatible estimators
    
    Throws:
    
    CardinalityMergeException - If at least one of the estimators is not compatible with this one

Class HyperLogLog

Nested Class Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Constructor Detail

HyperLogLog

HyperLogLog

HyperLogLog

Method Detail

offerHashed

offerHashed

offer

cardinality

sizeof

getBytes

addAll

merge