TextTokenizer class Null safety

Interface for a text analyser class that extracts tokens from text for use in full-text search queries and indexes:

  • analyzer is a TextAnalyzer used by the TextTokenizer to tokenize source text; and
  • provide a tokenFilter if you want to manipulate tokens or restrict tokenization to tokens that meet criteria for either index or count; and
  • the tokenize function tokenizes source text using the analyzer and then manipulates the output by applying tokenFilter; and
  • the tokenizeJson function extracts tokens from the zones in a JSON document.
Implementers
Annotations
  • @Deprecated('Interface `TextTokenizer` is deprecated and will be removed ' 'from the next stable version. Use `TextAnalyzer.tokenize` and ' '`TextAnalyzer.tokenizeJson` in stead.')

Constructors

TextTokenizer({required TextAnalyzer analyzer, TokenFilter? tokenFilter})
Instantiates a const TextTokenizerBase instance.
factory

Properties

analyzer → TextAnalyzer
The TextAnalyzer used by the TextTokenizer.
read-only
hashCode → int
The hash code for this object.
read-only, inherited
runtimeType → Type
A representation of the runtime type of the object.
read-only, inherited
tokenFilter → TokenFilter?
A filter that returns a subset of tokens.
read-only

Methods

noSuchMethod(Invocation invocation) → dynamic
Invoked when a non-existent method or property is accessed.
inherited
tokenize(SourceText source, {NGramRange? nGramRange, Zone? zone, TokenizingStrategy strategy = TokenizingStrategy.terms}) → Future<List<Token>>
Extracts one or more tokens from source for use in full-text search queries and indexes.
tokenizeJson(Map<String, dynamic> document, {NGramRange? nGramRange, Iterable<Zone>? zones, TokenizingStrategy strategy = TokenizingStrategy.terms}) → Future<List<Token>>
Extracts tokens from the zones in a JSON document for use in full-text search queries and indexes.
toString() → String
A string representation of this object.
inherited

Operators

operator ==(Object other) → bool
The equality operator.
inherited

Constants

english → const _TextTokenizerImpl
Returns a static const TextTokenizer using the English analyzer.
_TextTokenizerImpl(English(), null)