Tokenizers (Simmetrics Core 4.1.1 API)

java.lang.Object
- org.simmetrics.tokenizers.Tokenizers

```
public final class Tokenizers
extends Object
```
Construct simple tokenizers, chains multiple tokenizers into a single tokenizer or creates a tokenizers that apply filters and transforms to tokens.
The created tokenizers are immutable and thread-safe provided all their components are also immutable and thread-safe.

Method Summary

All Methods Static Methods Concrete Methods
Modifier and Type	Method and Description
`static Tokenizer`	`chain(List<Tokenizer> tokenizers)` Chains tokenizers together.
`static Tokenizer`	`chain(Tokenizer tokenizer, Tokenizer... tokenizers)` Chains tokenizers together.
`static Tokenizer`	`filter(Tokenizer tokenizer, com.google.common.base.Predicate<String> predicate)` Constructs a new filtering tokenizer.
`static Tokenizer`	`pattern(Pattern pattern)` Returns a tokenizer that splits a string into tokens around the pattern as if calling `pattern.split(input,-1)`.
`static Tokenizer`	`pattern(String regex)` Returns a tokenizer that splits a string into tokens around the pattern as if calling `Pattern.compile(regex).split(input,-1)`.
`static Tokenizer`	`qGram(int q)` Returns a q-gram tokenizer for a variable `q`.
`static Tokenizer`	`qGramWithFilter(int q)` Returns a q-gram tokenizer for a variable `q`.The tokenizer will return an empty collection if the input is empty or shorter then `q`.
`static Tokenizer`	`qGramWithPadding(int q)` Returns a q-gram tokenizer for a variable `q`.
`static Tokenizer`	`qGramWithPadding(int q, String padding)` Returns a q-gram tokenizer for a variable `q`.
`static Tokenizer`	`qGramWithPadding(int q, String startPadding, String endPadding)` Returns a q-gram tokenizer for a variable `q`.The q-gram is extended beyond the length of the string with padding.
`static Tokenizer`	`transform(Tokenizer tokenizer, com.google.common.base.Function<String,String> function)` Constructs a new transforming tokenizer.
`static Tokenizer`	`whitespace()` Returns a tokenizer that splits a string into tokens around whitespace.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Method Detail
  - pattern
```
public static Tokenizer pattern(Pattern pattern)
```
    Returns a tokenizer that splits a string into tokens around the pattern as if calling pattern.split(input,-1).
    
    Parameters:
    
    pattern - to split the the string around
    
    Returns:
    
    a pattern tokenizer
  - pattern
```
public static Tokenizer pattern(String regex)
```
    Returns a tokenizer that splits a string into tokens around the pattern as if calling Pattern.compile(regex).split(input,-1).
    
    Parameters:
    
    regex - to split the the string around
    
    Returns:
    
    a pattern tokenizer
  - qGram
```
public static Tokenizer qGram(int q)
```
    Returns a q-gram tokenizer for a variable q. The tokenizer will return an empty collection if the input is empty. A collection with the original input is returned for tokens shorter then q.
    The tokenizer takes care to split the string on Unicode code points, not separating valid surrogate pairs.
    
    Parameters:
    
    q - size of the tokens
    
    Returns:
    
    a q-gram tokenizer
  - qGramWithFilter
```
public static Tokenizer qGramWithFilter(int q)
```
    Returns a q-gram tokenizer for a variable q.The tokenizer will return an empty collection if the input is empty or shorter then q.
    The tokenizer takes care to split the string on Unicode code points, not separating valid surrogate pairs.
    
    Parameters:
    
    q - size of the tokens
    
    Returns:
    
    a q-gram tokenizer
  - qGramWithPadding
```
public static Tokenizer qGramWithPadding(int q)
```
    Returns a q-gram tokenizer for a variable q. The input is padded with q-1 special characters before being tokenized. Uses # as the default padding.
    The tokenizer takes care to split the string on Unicode code points, not separating valid surrogate pairs.
    
    Parameters:
    
    q - size of the tokens
    
    Returns:
    
    a q-gram tokenizer
  - qGramWithPadding
```
public static Tokenizer qGramWithPadding(int q,
                                         String padding)
```
    Returns a q-gram tokenizer for a variable q. The q-gram is extended beyond the length of the string with padding.
    The tokenizer takes care to split the string on Unicode code points, not separating valid surrogate pairs.
    
    Parameters:
    
    q - size of the tokens
    
    padding - padding to pad start and end of string with
    
    Returns:
    
    a q-gram tokenizer
  - qGramWithPadding
```
public static Tokenizer qGramWithPadding(int q,
                                         String startPadding,
                                         String endPadding)
```
    Returns a q-gram tokenizer for a variable q.The q-gram is extended beyond the length of the string with padding.
    The tokenizer takes care to split the string on Unicode code points, not separating valid surrogate pairs.
    
    Parameters:
    
    q - size of the tokens
    
    startPadding - padding to pad start of string with
    
    endPadding - padding to pad end of string with
    
    Returns:
    
    a q-gram tokenizer
  - whitespace
```
public static Tokenizer whitespace()
```
    Returns a tokenizer that splits a string into tokens around whitespace. Does not return leading or trailing empty tokens.
    To create tokenizer that returns leading and trailing empty tokens use Tokenizers.pattern("\\s+")
    
    Returns:
    
    a white space tokenizer
  - transform
```
public static Tokenizer transform(Tokenizer tokenizer,
                                  com.google.common.base.Function<String,String> function)
```
    Constructs a new transforming tokenizer. After tokenization, all tokens are transformed by the function.
    
    Parameters:
    
    tokenizer - delegate tokenizer
    
    function - to transform tokens
    
    Returns:
    
    a new transforming tokenizer.
  - chain
```
public static Tokenizer chain(List<Tokenizer> tokenizers)
```
    Chains tokenizers together. The output of each tokenizer is tokenized by the next. The tokenizers are applied in order.
    If only a single tokenizer is provided, that tokenizer is returned.
    
    Parameters:
    
    tokenizers - a non-empty list of tokenizers
    
    Returns:
    
    a chain of tokenizers
  - chain
```
public static Tokenizer chain(Tokenizer tokenizer,
                              Tokenizer... tokenizers)
```
    Chains tokenizers together. The output of each tokenizer is tokenized by the next. The tokenizers are applied in order.
    If only a single tokenizer is provided, that tokenizer is returned.
    
    Parameters:
    
    tokenizer - the first tokenizer
    
    tokenizers - a the other tokenizers
    
    Returns:
    
    a chain of tokenizers
  - filter
```
public static Tokenizer filter(Tokenizer tokenizer,
                               com.google.common.base.Predicate<String> predicate)
```
    Constructs a new filtering tokenizer. After tokenization, all tokens that don't match predicate are removed.
    
    Parameters:
    
    tokenizer - delegate tokenizer
    
    predicate - for tokens to keep
    
    Returns:
    
    a new filtering tokenizer.

Class Tokenizers

Method Summary

Methods inherited from class java.lang.Object

Method Detail

pattern

pattern

qGram

qGramWithFilter

qGramWithPadding

qGramWithPadding

qGramWithPadding

whitespace

transform

chain

chain

filter