org.simmetrics.tokenizers

Class AbstractTokenizer

java.lang.Object

org.simmetrics.tokenizers.AbstractTokenizer

All Implemented Interfaces:

Tokenizer

public abstract class AbstractTokenizer
extends Object
implements Tokenizer

Convenience tokenizer. Provides default implementation to tokenize to set and multiset by calling Tokenizer.tokenizeToList(String).

Constructor Summary

Constructors

Constructor and Description
AbstractTokenizer()

Method Summary

Modifier and Type	Method and Description
com.google.common.collect.Multiset<String>	tokenizeToMultiset(String input) Return tokenized version of a string as a multiset of tokens.
Set<String>	tokenizeToSet(String input) Return tokenized version of a string as a set of tokens.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.simmetrics.tokenizers.Tokenizer

tokenizeToList

Constructor Detail

AbstractTokenizer

public AbstractTokenizer()

Method Detail

tokenizeToSet

public Set<String> tokenizeToSet(String input)

Description copied from interface: Tokenizer

Return tokenized version of a string as a set of tokens.

Specified by:

tokenizeToSet in interface Tokenizer

Parameters:

input - input string to tokenize

Returns:

tokenized version of a string as a set

tokenizeToMultiset

public com.google.common.collect.Multiset<String> tokenizeToMultiset(String input)

Description copied from interface: Tokenizer

Return tokenized version of a string as a multiset of tokens.

Specified by:

tokenizeToMultiset in interface Tokenizer

Parameters:

input - input string to tokenize

Returns:

tokenized version of a string as a multiset