tokenize method Null safety

  1. @override
Future<TextSource> tokenize(
  1. String source
)
override

Extracts tokens from text for use in full-text search queries and indexes.

Returns a TextSource with source and its component Sentences and Tokens

Implementation

@override
Future<TextSource> tokenize(String source) async {
  final sentenceStrings = configuration.sentenceSplitter(source);
  final sentences = <Sentence>[];
  // convert [sentenceStrings] into [Sentence]s
  for (final sentence in sentenceStrings) {
    final value =
        await Sentence.fromString(sentence, configuration, tokenFilter);
    sentences.add(value);
  }
  return TextSource(source, sentences);
}