# Part-of-Speech Tagging

A basic part-of-speech tagger is provided by `pos_tag()`, which takes a segmented phrase or sentence as the input:

    >>> import pycantonese
    >>> unsegmented = '我噚日買嗰對鞋。'  # I bought that pair of shoes yesterday.
    >>> segmented = pycantonese.segment(unsegmented)
    >>> segmented
    ['我', '噚日', '買', '嗰', '對', '鞋', '。']
    >>> pycantonese.pos_tag(segmented)
    [('我', 'PRON'), ('噚日', 'ADV'), ('買', 'VERB'), ('嗰', 'PRON'), ('對', 'NOUN'), ('鞋', 'NOUN'), ('。', 'PUNCT')]

The part-of-speech tagger uses the averaged perceptron model trained on HKCanCor data. HKCanCor has already been annotated for part-of-speech tags, with a tagset of over 100 tags (46 of which are described). By default, `pos_tag()` maps the HKCanCor tagset to the Universal Dependencies v2 tagset (with 17 tags), for cross-linguistic natural language processing work. If you would like the original HKCanCor tagset, `pos_tag()` accepts the keyword argument `tagset`:

    >>> pycantonese.pos_tag(segmented, tagset="hkcancor")
    [('我', 'R'), ('噚日', 'T'), ('買', 'V'), ('嗰', 'R'), ('對', 'Q'), ('鞋', 'N'), ('。', '。')]

The helper function `hkcancor_to_ud()` exposes the tagset mapping from HKCanCor to Universal Dependencies.

Due to the statistical nature of part-of-speech tagging, the quality of results from `pos_tag()` depends on (i) the training data, (ii) the quality of word segmentation, since the function expects a segmented input. Currently, a major limitation is the fact that HKCanCor is perhaps still the only Cantonese corpus with a permissive license that comes annotated with part-of-speech tags. Its relatively small size (about 150,000 tagged words) means that models more sophisticated than a standard averaged perceptron approach wouldn’t be worth it. If you think the results from `pos_tag()` are odd, it is potentially due to the HKCanCor training data (e.g., specific occurrences of word + tag combinations might have thrown off the tagger), or the quality of word segmentation, especially if your segmented input comes from `segment()` – please get in touch if you would like further investigation.