Hide navigation sidebar
Hide table of contents sidebar
Back to top
Edit this page
Toggle table of contents sidebar

# Other Operations

> This tutorial is based on the paper on stringi that was published in the Journal of Statistical Software; see [2]. To learn more about R, check out Marek’s open-access (free!) textbook Deep R Programming [3].

In the sequel, we cover the functions that deal with text boundary detection, random string generation, date/time formatting and parsing, amongst others.

## Analysing Text Boundaries

Text boundary analysis aims at locating linguistic delimiters for the purpose of splitting text into lines, word-wrapping, counting characters or words, locating particular text units (e.g., the 3rd sentence), etc.

Generally, text boundary analysis is a locale-sensitive operation (see the Unicode Standard Annex #29: Unicode Text Segmentation). For example, in Japanese and Chinese, spaces are not used for the separation of words – a line break can occur even in the middle of a word. Nevertheless, these languages have punctuation and diacritical marks that cannot start or end a line, so this must also be considered.

The ICU Break Iterator (see the ICU User Guide on Boundary Analysis) comes in four flavours (see the `type` option in `stri_opts_brkiter()`): `character`, `work`, `line_break`, and `sentence`.

We have access to functions such as `stri_count_boundaries()`, `stri_split_boundaries()`, `stri_extract__boundaries()`, and `stri_locate__boundaries()`, as well as their specialised versions: `stri_count_words()`, `stri_extract__words()`, and `stri_split_lines()`, amongst others. For example:

    x <- "The\u00a0above-mentioned    features are useful. " %s+%
      "My hovercraft is full of eels, eggs, and spam."

Number of sentences:

    stri_count_boundaries(x, type="sentence")
    ## [1] 2

The list of all the words:

    stri_extract_all_words(x)
    ## [[1]]
    ##  [1] "The"        "above"      "mentioned"  "features"   "are"
    ##  [6] "useful"     "My"         "hovercraft" "is"         "full"
    ## [11] "of"         "eels"       "eggs"       "and"        "spam"

## Trimming, Padding, and Other Formatting

The following functions can be used for pretty-printing character strings or text on the console, dynamically generating reports (e.g., with `Sweave()` or knitr [13]), or creating text files (e.g., with `stri_write_lines()`; see Reading and Writing Text Files).

### Padding

`stri_pad()` pads strings with some character so that they reach the desired widths (as in `stri_width()`). This can be used to centre, left-, or right-align a message when printed with, e.g., `cat()`.

    cat(stri_pad("SPAMITY SPAM", width=77, side="both", pad="."))
    ## ................................SPAMITY SPAM.................................

### Trimming

A dual operation is that of trimming from the left or right side of strings:

    x <- "      spam, eggs, and lovely spam.\n"
    stri_trim(x)  # side="both"
    ## [1] "spam, eggs, and lovely spam."

### Word Wrapping

The `stri_wrap()` function splits each (possibly long) string in a character vector into chunks of at most a given width. By default, the dynamic word wrap algorithm [8] that minimises the raggedness of the formatted text is used. However, there’s also an option (`cost_exponent=0`) to use the greedy alignment, for compatibility with the built-in `strwrap()`.

    x <- stri_rand_lipsum(1)  # random text paragraph
    cat(stri_wrap(x, width=74, indent=8, exdent=4, prefix="> "), sep="\n")
    ## >         Lorem ipsum dolor sit amet, quis donec pretium auctor, quis id.
    ## >     Mauris rhoncus donec amet egestas sagittis ipsum per. Sed, sociis
    ## >     amet. Aliquam fusce dictumst sed vehicula ultrices arcu. Eros,
    ## >     netus et. Amet amet mi vestibulum vitae dapibus ut felis. Magnis
    ## >     in vestibulum egestas massa curabitur a ut, eget in in facilisis.
    ## >     Etiam odio fermentum sit ante ridiculus sit elit. Sapien torquent
    ## >     fermentum tortor gravida ornare sapien consequat et sem turpis. Hac
    ## >     vel lacus habitasse et id non. Metus habitasse sed lacinia nibh ex
    ## >     metus. Amet nam vestibulum ornare tincidunt massa sed ullamcorper.

### Applying String Templates

`stri_sprintf()` is a Unicode-aware rewrite of the built-in `sprintf()` function. In particular, it enables formatting and padding based on character width, not just the number of code points. The function is also available as a binary operator `%s$%`, which is similar to Python’s `%` overloaded for objects of type `str`.

    cat(stri_sprintf("[%6s]", c("abcd", "\u200b\u200b\u200bąß²€")), sep="\n")
    ## [  abcd]
    ## [  ​​​ąß²€]

The above guarantees that the two output strings are of at least width of 6 (plus the square brackets).

    "value='%.4f'" %s$% pi  # equivalently: "value='%.4f'" %s$% list(pi)
    ## [1] "value='3.1416'"
    "%s='%.*3$f'" %s$% list("pi", pi, 1:4)
    ## [1] "pi='3.1'"    "pi='3.14'"   "pi='3.142'"  "pi='3.1416'"

## Generating Random Strings

Apart from `stri_rand_lipsum()`, which produces random-ish text paragraphs (“placeholders” for real text), we have access to a function that generates sequences of characters uniformly sampled (with replacement) from a given set.

For example, here are 5 random ACTG strings of lengths from 2 to 6:

    stri_rand_strings(5, 2:6, "[ACTG]")
    ## [1] "CT"     "CTT"    "AGTG"   "CTCGG"  "ATAACT"

Moreover:

    stri_rand_strings(1, 8, "[\\p{script=Katakana}&\\p{L}]")
    ## [1] "ﾐｮㇿ𛅧ｧｱヾㇷ"

See Matching Individual Characters and `help("stringi-search-charclass")` for different ways to specify character sets.

## Transliterating

Transliteration, in its broad sense, deals with the substitution of characters or their groups for different ones, according to some well-defined, possibly context-aware, rules. It may be utile, amongst others, when “normalising” pieces of strings or identifiers so that they can be more easily compared.

### Case Mapping

Mapping to upper, lower, or title case is a language- and context-sensitive operation that can change the total number of code points in a string.

    stri_trans_toupper("groß")
    ## [1] "GROSS"
    stri_trans_tolower("Iİ", locale="tr_TR")               # Turkish
    ## [1] "ıi"
    stri_trans_totitle("ijsvrij yoghurt", locale="nl_NL")  # Dutch
    ## [1] "IJsvrij Yoghurt"

### Mapping Between Specific Characters

When a fast 1-to-1 code point translation is required, we can call:

    stri_trans_char("GATAAATCTGGTCTTATTTCC", "ACGT", "tgca")
    ## [1] "ctatttagaccagaataaagg"

Here, “`A`”, “`C`”, “`G`”, and “`T`” is replaced with “`t`”, “`g`”, “`c`”, and “`a`”, respectively.

### General Transforms

`stri_trans_general()` gives access to a wide range of text transforms defined by ICU, whose catalogue can be browsed by calling `stri_trans_list()`.

    sample(stri_trans_list(), 9)  # a few random entries
    ## [1] "Any-und_FONIPA" "Any-FCD"        "Deva-Guru"      "yo-yo_BJ"
    ## [5] "Taml-Orya"      "Tamil-Arabic"   "hy-ar"          "de-ASCII"
    ## [9] "Any-Kana"

In the example below, we apply a transliteration chain: we first convert to upper case, and then convert characters in the Latin script to ASCII.

    stri_trans_general("groß© żółć La Niña köszönöm", "upper; latin-ascii")
    ## [1] "GROSS(C) ZOLC LA NINA KOSZONOM"

Custom rule-based transliteration is also supported (see the ICU User Guide on General Transforms). It can be used, for instance, to generate different romanisations of non-Latin alphabets.

## Parsing and Formatting Date and Time

In base R, dealing with temporal data in regional settings other than the current locale is somewhat difficult. For instance, many will find the task of parsing the following Polish date problematic:

    x <- "28 września 2021 r., godz. 17:17:32"

stringi connects to the ICU date and time services so that this becomes an easy exercise:

    stri_datetime_parse(x, "dd MMMM yyyy 'r., godz.' HH:mm:ss",
      locale="pl_PL", tz="Europe/Warsaw")
    ## [1] "2021-09-28 17:17:32 CEST"

This function returns an object of class `POSIXct`, for compatibility with base R. Note, however, that ICU uses its own format patterns (see the ICU User Guide on Formatting Dates and Times). For convenience, `strftime()`- and `strptime()`-like templates can be converted with `stri_datetime_fstr()`:

    stri_datetime_parse(x,
      stri_datetime_fstr("%d %B %Y r., godz. %H:%M:%S"),
      locale="pl_PL", tz="Europe/Warsaw")
    ## [1] "2021-09-28 17:17:32 CEST"

Relative dates are also supported:

    stri_datetime_format(
      stri_datetime_add(stri_datetime_now(), 1, "day"), # add 1 day to 'now'
      "datetime_relative_long",              # full format, relative to 'now'
      locale="en_NZ", tz="NZ")
    ## [1] "tomorrow at 5:14:20 pm NZDT"

For example, here’s how we can access different calendars:

    stri_datetime_format(
      stri_datetime_create(2020, 1:12, 1),
      "date_long",
      locale="@calendar=hebrew")
    ##  [1] "4 Tevet 5780"    "6 Shevat 5780"   "5 Adar 5780"     "7 Nisan 5780"
    ##  [5] "7 Iyar 5780"     "9 Sivan 5780"    "9 Tamuz 5780"    "11 Av 5780"
    ##  [9] "12 Elul 5780"    "13 Tishri 5781"  "14 Heshvan 5781" "15 Kislev 5781"
    stri_datetime_format(
      stri_datetime_create(2020, c(2, 8), c(4, 7)),
      "date_full",
      locale="ja_JP@calendar=japanese")
    ## [1] "令和2年2月4日火曜日" "令和2年8月7日金曜日"

We have selected the Hebrew calendar within the English locale and the Japanese calendar in the Japanese locale.