# Word Segmentation

By convention, Cantonese is not written with word boundaries (like spaces in English). However, in many natural language processing tasks, it is often necessary to work with a segmented form of Cantonese data. PyCantonese provides the function `segment()` that takes an unsegmented text string in Cantonese characters and returns the segmented version:

    >>> import pycantonese
    >>> pycantonese.segment("廣東話容唔容易學？")  # Is Cantonese easy to learn?
    ['廣東話', '容', '唔', '容易', '學', '？']

Currently, the underlying word segmentation model is a simple longest string matching algorithm, trained by (i) the HKCanCor corpus data included in this library and (ii) the rime-cantonese data (the 2021.05.16 release, CC BY 4.0 license). The segmentation is constrained such that the resulting words contain no more than five characters.

## Customizing Segmentation

Because the current implementation of word segmentation depends entirely on whether a potential word is found in the training data, there are situations where you would like to explicitly allow or disallow certain potential words. To this end, the `segment()` function has the `cls` keyword argument (think: the `cls` kwarg for `json.load`) that takes a `Segmenter` object for customizing in the following ways:

- To specify words to allow, pass an iterable of word strings to the `allow` keyword argument of `Segmenter`:

  >     >>> import pycantonese
  >     >>> from pycantonese.word_segmentation import Segmenter
  >     >>> segmenter = Segmenter(allow={"容唔容易"})
  >     >>> pycantonese.segment("廣東話容唔容易學？", cls=segmenter)
  >     ['廣東話', '容唔容易', '學', '？']

- To specify words to disallow, pass an iterable of word strings to the `disallow` keyword argument of `Segmenter`:

  >     >>> import pycantonese
  >     >>> from pycantonese.word_segmentation import Segmenter
  >     >>> segmenter = Segmenter(disallow={"廣東話"})
  >     >>> # 廣東 still exists as a word in the model, though 廣東話 is banned here.
  >     >>> pycantonese.segment("廣東話容唔容易學？", cls=segmenter)
  >     ['廣東', '話', '容', '唔', '容易', '學', '？']

- To control the maximum word length (default: 5), pass an integer to the `max_word_length` keyword argument of `Segmenter`:

  >     >>> import pycantonese
  >     >>> from pycantonese.word_segmentation import Segmenter
  >     >>> segmenter = Segmenter(max_word_length=2)
  >     >>> pycantonese.segment("廣東話容唔容易學？", cls=segmenter)
  >     ['廣東', '話', '容', '唔', '容易', '學', '？']

The keyword arguments `allow`, `disallow`, and `max_word_length` of the `Segmenter` class can be concurrently used in the same `Segmenter` instance.