Package org.simmetrics.tokenizers

Interface Summary

Interface	Description
Tokenizer	A tokenizer divides an input string into tokens.

Class Summary

Class	Description
AbstractTokenizer	Convenience tokenizer.
Tokenizers	Construct simple tokenizers, chains multiple tokenizers into a single tokenizer or creates a tokenizers that apply filters and transforms to tokens.