TextTokenizer constructor Null safety

TextTokenizer(
  1. {required TextAnalyzer analyzer,
  2. TokenFilter? tokenFilter}
)

Instantiates a const TextTokenizerBase instance.

  • analyzer is used by the TextTokenizer to tokenize source text (default is English.analyzer); and
  • provide a custom tokenFilter if you want to manipulate tokens or restrict tokenization to tokens that meet specific criteria.

Implementation

factory TextTokenizer(
        {required TextAnalyzer analyzer, TokenFilter? tokenFilter}) =>
    _TextTokenizerImpl(analyzer, tokenFilter);