Splits given text to elements (tokens).
<tokenize delimiters="delimiters" trimtokens="trimtokens" allowemptytokens="allowemptytokens"> content to tokenize </tokenize>
Name | Required | Default | Description |
---|---|---|---|
delimiters | no | new line character | Tells which characters are used as token delimiters. |
trimtokens | no | yes | Tells whether to trim resulting tokens. |
allowemptytokens | no | no | Tells whether to include empty tokens in the resulting list (consisting only of whitespaces). |