Turkish Tokenization and Sentence Extraction
============

## Sentence Boundary Detection

### Usage

Zemberek provides a mechanism for extracting sentences from text. This is usually the first step
 for an NLP application. TurkishSentenceExtractor class is used for this purpose. This class uses 
 a combination of simple rules and a Binary Averaged Perceptron model for finding 
 sentence boundaries. For default behavior, a singleton instance is provided:
 
    TurkishSentenceExtractor extractor = TurkishSentenceExtractor.DEFAULT;

After that, three methods can be used for extracting sentences from a paragraph, a list
     of paragraphs or a String representing a document.
      For extracting sentences from a paragraph :

    String input = "Merhaba! Bugün 2. köprü Fsm.'de trafik vardı.değil mi?";     
    List<String> sentences = extractor.fromParagraph(input);

    sentences -> "Merhaba!", "Bugün 2. köprü Fsm.'de trafik vardı.", "değil mi?"
    
If input contains line breaks, fromDocument() method should be used.
fromParagraph() method will not split sentences from
 line breaks. fromDocument() method will first split input from line breaks to paragraphs
 then call fromParagraphs() internally.
 
    String input = "Merhaba\nNasılsınız?";
    List<String> sentences = extractor.fromDocument(input);        
    
    sentences -> "Merhaba", "Nasılsınız?"    
 

### Performance and speed

We compared our system with Maximum Entropy based OpenNlp 
[SentenceDetector](https://opennlp.apache.org/documentation/1.7.0/manual/opennlp.html#tools.sentdetect) for performance and speed.
 TurkishSentenceExtractor was trained with 8300 sentences.  OpenNLP model was
 trained with a very large but somewhat noisy data. We did try training it with our training data but results were worse.
 Test set contains 1916 sentences and many of them contains numbers with dot characters. 

Precision and Recall values are calculated for correct/incorrect boundaries. NIST error rate is calculated 
by dividing total amount of boundary errors to sentence count.

|  Library   | Precision| Recall   | NIST Error| Correct Sentences | Speed Sentences/s| Model Size |
|------------|----------|----------|-----------|-------------------|------------------|------------|
| Zemberek   |  0.9990  | 0.9974   |  0.3653%  |  99.37%           |  29030           |  10.1 KB   |
| OpenNLP    |  0.9871  | 0.9979   |  1.5136%  |  98.38%           |  23084           |  3.3 MB    |

Test platform: 2.3 Ghz AMD FX-8320, Ubuntu Linux 16.04 LTS

Both systems work quite well for this test set. Probably if OpenNlp system contained the rule based mechanism, it would give equivalent results. 

Open NLP model is provided by @sonerx

### Notes

- Extracting from paragraphs only splits from [.!?…] characters.
- fromDocument method also splits from line breaks.
- Class will not split if a sentence ends with an abbreviation or a number,
- Narration sentences such as [Ali "topu at." dedi.] will be split.

## Tokenization

### Usage

Zemberek offers a rule based tokenizer class called TurkishTokenizer. 
This tokenizer uses a [custom Antlr grammar based Lexer](https://github.com/ahmetaa/zemberek-nlp/blob/master/tokenization/src/main/resources/tokenization/TurkishLexer.g4#L75).
There are static instances provided for common use:
 
        TurkishTokenizer tokenizer = TurkishTokenizer.DEFAULT;
        TurkishTokenizer tokenizer = TurkishTokenizer.ALL;        

DEFAULT tokenizer ignores most white spaces (space, tab, line feed and carriage return). 
ALL tokenizer tokenizes everything. 
TurkishTokenizer instances are thread safe.
There are several helper methods in the tokenizer.
If detailed token information is required:

```java
    TurkishTokenizer tokenizer = TurkishTokenizer.DEFAULT;
    List<Token> tokens = tokenizer.tokenize("Saat 12:00.");
    for (Token token : tokens) {
        System.out.println("Content = " + token.getText());
        System.out.println("Type = " + TurkishLexer.VOCABULARY.getDisplayName(token.getType()));
        System.out.println("Start = " + token.getStartIndex());
        System.out.println("Stop  = " + token.getStopIndex());
        System.out.println();
    }
```

This will output 

    Content = Saat
    Type = Word
    Start = 0
    Stop  = 3
    
    Content = 12:00
    Type = Time
    Start = 5
    Stop  = 9
    
    Content = .
    Type = Punctuation
    Start = 10
    Stop  = 10

Note: until 0.17.0, Token class was and `org.antlr.v4.runtime.Token` implementation instance.
However as of 0.17.0 it is an internal Zemberek class `zemberek.tokenization.Token`. 
User can reach several information from this class  such as type, start and stop character indexes.

User can customize a TurkishTokenizer. 
For example if punctuations and white spaces are needed to be ignored:
 
    TurkishTokenizer tokenizer = TurkishTokenizer
            .builder()
            .ignoreTypes(Token.Type.Punctuation, Token.Type.NewLine, Token.Type.SpaceTab)
            .build();
    List<Token> tokens = tokenizer.tokenize("Saat, 12:00.");
    for (Token token : tokens) {
        System.out.println(token);
    } 

This will output 

    [Saat Word 0-3]
    [12:00 Time 6-10]

If user only interested in String values of the tokens, this method can be used:

    TurkishTokenizer tokenizer = TurkishTokenizer.DEFAULT;
    List<String> tokens = tokenizer.tokenizeToStrings("Saat, 12:00.");
    tokens.forEach(System.out::println);

Code will print out:

    Saat
    ,
    12:00
    .

### Token Types

Tokenizer currently tokenizes these types:

    public enum Type {
  
      // white space
      SpaceTab,
      NewLine,
  
      // words
      Word,
      WordAlphanumerical,
      WordWithSymbol,
      Abbreviation,
      AbbreviationWithDots,
  
      Punctuation,
  
      // numerals. May contain suffixes.
      RomanNumeral,
      Number,
      PercentNumeral,
  
      // temporal
      Time,
      Date,
  
      // web related
      URL,
      Email,
      HashTag,
      Mention,
      MetaTag,
  
      Emoji,
      Emoticon,
  
      UnknownWord,
      Unknown,
    }

### Speed

We tested the DEFAULT TurkishTokenizer with 100,000 lines of news sentences 
on an Intel Xeon E5-2680 @ 2.50GHz system. 
Tokenization speed is about 1,500,000 tokens per second. 

### Notes

- Tokenizer is not a very good classifier for some cases. There may be cases it splits meaningful words.
- It is suggested to use it for sentence tokenization after sentence extraction. 

