tokenize method Null safety

Future<List<Token>> tokenize(
  1. SourceText source,
  2. {NGramRange? nGramRange,
  3. Zone? zone,
  4. TokenizingStrategy strategy = TokenizingStrategy.terms}
)

Extracts one or more tokens from source for use in full-text search queries and indexes.

  • nGramRange is the range of N-gram lengths to generate; and
  • zone is the name of the zone in a document in which the term is located.

Returns a List<Token>.

Implementation

Future<List<Token>> tokenize(SourceText source,
    {NGramRange? nGramRange,
    Zone? zone,
    TokenizingStrategy strategy = TokenizingStrategy.terms});