org.simmetrics.tokenizers

Interface Tokenizer

All Known Implementing Classes:

AbstractTokenizer

public interface Tokenizer

A tokenizer divides an input string into tokens. A tokenizer may not provide a collection containing null.

It is recommend that implementations extend AbstractTokenizer rather then implement this interface.

Method Summary

Modifier and Type	Method and Description
List<String>	tokenizeToList(String input) Return tokenized version of a string as a list of tokens.
com.google.common.collect.Multiset<String>	tokenizeToMultiset(String input) Return tokenized version of a string as a multiset of tokens.
Set<String>	tokenizeToSet(String input) Return tokenized version of a string as a set of tokens.

Method Detail

tokenizeToList

List<String> tokenizeToList(String input)

Return tokenized version of a string as a list of tokens.

Parameters:

input - input string to tokenize

Returns:

List tokenized version of a string

tokenizeToMultiset

com.google.common.collect.Multiset<String> tokenizeToMultiset(String input)

Return tokenized version of a string as a multiset of tokens.

Parameters:

input - input string to tokenize

Returns:

tokenized version of a string as a multiset

tokenizeToSet

Set<String> tokenizeToSet(String input)

Return tokenized version of a string as a set of tokens.

Parameters:

input - input string to tokenize

Returns:

tokenized version of a string as a set