General Design Principles
=========================


```{r echo=FALSE, results="asis", message=FALSE}
source("common.R")
opts_chunk$set(cache.path="cache/design_principles_")
```


The API of the early releases of *stringi* has been designed to be
fairly compatible with that of the 0.6.2 version of the [*stringr*](https://stringr.tidyverse.org/)
package {cite}`Wickham2010:stringr` (dated 2012[^footstringr]),
with some fixes in the consistency of the
handling of missing values and zero-length vectors, amongst others.
However, instead of being merely thin wrappers around base R[^footstringx] functions,
which we have identified as not necessarily portable across platforms
and not really suitable for natural language processing tasks, all the
functionality has been implemented from the ground up, with the use of
[*ICU*](https://icu.unicode.org/) services wherever applicable.
Since the initial release, an
abundance of new features has been added, and the package can now be
considered a complete workhorse for text data processing.

Note that
the *stringi* API is stable. Future releases will aim for as much
backward compatibility as possible so that other software projects can
safely rely on it.

[^footstringr]: Interestingly, in 2015 the aforementioned
    *stringr* package was rewritten as a set of wrappers around some of
    the *stringi* functions instead of the base R ones. In Section 14.7 of
    *R for Data Science* {cite}`GrolemundWickham2017:rdatascience` we read:
    "*stringr* is useful when you're learning because it exposes a minimal
    set of functions, which have been carefully picked to handle the most
    common string manipulation functions. *stringi*, on the other hand, is
    designed to be comprehensive. It contains almost every function you
    might ever need: *stringi* has 250 functions to *stringr*'s 49".

[^footstringx]: See also the new
    [*stringx*](https://stringx.gagolewski.com) package
    which supplies a set of portable and efficient
    replacements for and enhancements of the base R functions
    (based on *stringi*).



<!--
Functions in (the historical) *stringr* 0.6.2 and their counterparts in base R 4.1.
:name: tab-oldstringr

| *stringr* 0.6.2                        | **Base R 4.1**                                  | **Purpose**                                                      |
| -------------------------------------- | ----------------------------------------------- | ---------------------------------------------------------------- |
| `str_c()`                              | `paste()`, `paste0()`, also `sprintf()`         | join strings                                                     |
| `str_count()`                          | `gregexpr()`                                    | count pattern matches                                            |
| `str_detect()`                         | `grepl()`                                       | detect pattern matches                                           |
| `str_dup()`                            | `strrep()`                                      | duplicate strings                                                |
| `str_extract()`, `str_extract_all()`   | `regmatches()` with `regexpr()`, `gregexpr()`   | extract (first, all) pattern matches                             |
| `str_length()`                         | `nchar()`                                       | compute string length                                            |
|                                        | `nchar(type="width")`                           | compute string width                                             |
| `str_locate()`, `str_locate_all()`     | `regexpr()`, `gregexpr()`                       | locate (first, all) pattern matches                              |
| `str_match()`, `str_match_all()`       | `regmatches()` with `regexec()`, `gregexec()`   | extract (first, all) matches to regex capture groups             |
| `str_pad()`                            |                                                 | add whitespaces at beginning or end                              |
| `str_replace()`, `str_replace_all()`   | `sub()`, `gsub()`                               | replace (first, all) pattern matches with a replacement string   |
| `str_split()`, `str_split_fixed()`     | `strsplit()`                                    | split up a string into pieces                                    |
| `str_sub()`                            | `substr()`, `substring()`                       | extract or replace substrings                                    |
| `str_trim()`                           | `trimws()`                                      | remove whitespaces from beginning or end                         |
| `str_wrap()`                           | `strwrap()`                                     | split strings into text lines                                    |
| `word()`                               |                                                 | extract words from a sentence                                    |
|                                        | `startsWith()`, `endsWith()`                    | determine if strings start or end with a pattern match           |
|                                        | `tolower()`, `toupper()`                        | case mapping and folding                                         |
|                                        | `chartr()`                                      | transliteration                                                  |
|                                        | `sprintf()`                                     | string formatting                                                |
|                                        | `strftime()`, `strptime()`                      | date-time formatting and parsing                                 |
-->


Naming
------

Function and argument names use a combination of lowercase letters and
underscores (and no dots). To avoid namespace clashes, all function
names feature the "`stri_`" prefix. Names are quite self-explanatory,
e.g., `stri_locate_first_regex` and `stri_locate_all_fixed` find,
respectively, the first match to a regular expression and all
occurrences of a substring as-is.

Vectorisation
-------------

Individual character (or code point) strings can be entered using double
quotes or apostrophes:

```{r }
"spam"  # or 'spam'
```

However, as the R language does not feature any classical scalar types,
strings are wrapped around atomic vectors of type "`character`":

```{r }
typeof("spam")  # object type; see also is.character() and is.vector()
length("spam")  # how many strings are in this vector?
```

Hence, we will be using the terms "string" and "character vector of
length 1" interchangeably.

Not having a separate scalar type is very convenient; the so-called
*vectorisation* strategy encourages writing code that processes whole
collections of objects, all at once, regardless of their size.

For instance, given the following character vector:

```{r }
pythons <- c("Graham Chapman", "John Cleese", "Terry Gilliam",
  "Eric Idle", "Terry Jones", "Michael Palin")
```

we can separate the first and the last names from each other (assuming
for simplicity that no middle names are given), using just a single
function call:

```{r }
(pythons <- stri_split_fixed(pythons, " ", simplify=TRUE))
```

Due to vectorisation, we can generally avoid using the `for`- and
`while`-loops ("for each string in a vector..."). This can make the code
much more readable, maintainable, and faster to execute.


Acting Elementwise with Recycling
---------------------------------

Binary and higher-arity operations in R are oftentimes vectorised with
respect to all arguments (or at least to the crucial, non-optional
ones). As a prototype, let's consider the binary arithmetic, logical,
or comparison operators (and, to some extent, `paste()`, `strrep()`, and
more generally `mapply()`), for example the multiplication:

```{r }
c(10, -1) * c(1, 2, 3, 4)  # == c(10, -1, 10, -1) * c(1, 2, 3, 4)
```

Calling "`x * y`" multiplies the corresponding components of the two
vectors elementwisely. As one operand happens to be shorter than
the other, the former is recycled as many times as necessary to match the
length of the latter (there would be a warning if partial recycling
occurred). Also, acting on a zero-length input always yields an empty
vector.

All functions in *stringi* follow this convention (with some obvious
exceptions, such as the `collapse` argument in `stri_join()`, `locale`
in `stri_datetime_parse()`, etc.). In particular, all string search
functions are vectorised with respect to both the `haystack` and the
`needle` arguments (and, e.g., the replacement string, if applicable).

Some users, unaware of this rule, might find this behaviour unintuitive
at the beginning and thus miss out on how powerful it is. Therefore, let's
enumerate the most noteworthy scenarios that are possible thanks to
the arguments' recycling, using the call to
`stri_count_fixed(haystack, needle)` (which looks for a needle in a
haystack) as an illustration:

* many strings -- one pattern:

    ```{r}
    stri_count_fixed(c("abcd", "abcabc", "abdc", "dab", NA), "abc")
    ```

    (there is 1 occurrence of `"abc"` in `"abcd"`,
    2 in `"abcabc"`, and so forth);

* one string -- many patterns:

    ```{r}
    stri_count_fixed("abcdeabc", c("def", "bc", "abc", NA))
    ```

    (`"def"` does not occur in `"abcdeabc"`,
    `"bc"` can be found therein twice, etc.);

* each string -- its own corresponding pattern:

    ```{r}
    stri_count_fixed(c("abca", "def", "ghi"), c("a", "z", "h"))
    ```

    (there are two `"a"`s in `"abca"`,
    no `"z"` in `"def"`, and one `"h"` in `"ghi"`);

* each row in a matrix -- its own corresponding pattern:

    ```{r}
    (haystack <- matrix(  # example input
    do.call(stri_join,
        expand.grid(
        c("a", "b", "c"), c("a", "b", "c"), c("a", "b", "c")
        )), nrow=3))
    needle <- c("a", "b", "c")
    matrix(stri_count_fixed(haystack, needle),  # call to stringi
    nrow=3, dimnames=list(needle, NULL))
    ```

    (this looks for `"a"` in the 1st row of
    `haystack`, `"b"` in the 2nd row, and `"c"` in the 3rd;
    in particular, there are 3 `"a"`s in `"aaa"`, 2 in `"aba"`,
    and 1 `"b"` in `"baa"`;
    this is possible because matrices are
    represented as "flat" vectors of length `nrow*ncol`,
    whose elements are read in a column-major (Fortran) order;
    therefore, here,
    the pattern `"a"` is being sought in the 1st, 4th, 7th, ...
    string in `haystack`, i.e., `"aaa"`, `"aba"`, `"aca"`, ...;
    pattern `"b"` in the 2nd, 5th, 8th, ... string;
    and `"c"` in the 3rd, 6th, 9th, ... one);

    On a side note, to match different patterns
    with respect to each column, we can (amongst others)
    apply matrix transpose twice (`t(stri_count_fixed(t(haystack), needle))`).

* all strings -- all patterns:

    ```{r}
    haystack <- c("aaa", "bbb", "ccc", "abc", "cba", "aab", "bab", "acc")
    needle <- c("a", "b", "c")
    structure(
    outer(haystack, needle, stri_count_fixed),
    dimnames=list(haystack, needle))  # add row and column names
    ```

    (which computes the counts over the Cartesian product
    of the two arguments)

    This is equivalent to:

    ```{r}
    matrix(
    stri_count_fixed(rep(haystack, each=length(needle)), needle),
    byrow=TRUE, ncol=length(needle), dimnames=list(haystack, needle))
    ```





Missing Values
--------------

Some base R string processing functions, e.g., `paste()`, treat missing
values as literal `"NA"` strings. *stringi*, however, *does* enforce the
consistent propagation of missing values (like arithmetic operations):

```{r }
paste(c(NA_character_, "b", "c"), "x", 1:2)  # base R
stri_join(c(NA_character_, "b", "c"), "x", 1:2)  # stringi
```

For dealing with missing values, we may rely on the convenience
functions such as `stri_omit_na()` or `stri_replace_na()`.

Data Flow
---------

All vector-like arguments (including factors and objects) in *stringi*
are treated in the same manner. For example, if a function expects a
character vector on input and an object of another type is provided,
`as.character()` is called first (we see that in the example above,
"`1:2`" is treated as `c("1", "2")`).

Following {cite}`Wickham2010:stringr`, *stringi* makes sure the output data
types are consistent and that different functions are interoperable.
This makes operation chaining easier and less error-prone.

For example, `stri_extract_first_regex()` finds the first occurrence of
a pattern in each string. Therefore, the output is a character of the
same length as the input (with the recycling rule in place if necessary).

```{r }
haystack <- c("bacon", "spam", "jam, spam, bacon, and spam")
stri_extract_first_regex(haystack, "\\b\\w{1,4}\\b")
```

Note that a no-match (here, we have been looking for words of at most 4
characters) is marked with a missing string. This makes the output
vector size consistent with the length of the inputs.

On the other hand, `stri_extract_all_regex()` identifies all occurrences
of a pattern, whose counts may differ from input to input, therefore it
yields a list of character vectors.

```{r }
stri_extract_all_regex(haystack, "\\b\\w{1,4}\\b", omit_no_match=TRUE)
```

If the 3rd argument were not specified, a no-match would be represented
by a missing value (for consistency with the previous function).

Also, care is taken so that the "data" or "`x`" argument is most often
listed as the first one (e.g., in base R we have
`grepl(needle, haystack)` vs `stri_detect(haystack, needle)` here). This
makes the functions more intuitive to use, but also more forward pipe
operator-friendly (either when using "`|>`" introduced in R 4.1 or
"`%>%`" from [*magrittr*](https://magrittr.tidyverse.org/)).

Furthermore, for increased convenience, some functions have been added
despite the fact that they can  trivially be reduced to a series of other
calls. In particular, writing:

```{r results="hide"}
stri_sub_all(haystack,
  stri_locate_all_regex(haystack, "\\b\\w{1,4}\\b", omit_no_match=TRUE))
```

yields the same result as in the previous example, but refers to
`haystack` twice.


Further Deviations from Base R
------------------------------

*stringi* can be used as a replacement for the existing string processing
functions. Also, it offers many facilities not available in base R.
Except for being fully vectorised with respect to all crucial arguments,
propagating missing values and empty vectors consistently, and following
coherent naming conventions, our functions deviate from their classic
counterparts even further.


**Following Unicode Standards.**
Thanks to the comprehensive coverage of the most important services
provided by *ICU*, its users gain access to collation, pattern
searching, normalisation, transliteration, etc., that follow the current
Unicode standards for text processing in any locale. Due to this, as we
state in {ref}`Sec:encoding`, all inputs are converted to Unicode.
Furthermore, all outputs are always in UTF-8.


**Portability Issues in Base R.**
As mentioned in the introduction, base R string operations have
traditionally been limited in scope. There also might be some issues
with regards to their portability, reasons for which may be plentiful.
For instance, varied versions of the [*PCRE*](https://www.pcre.org/)
(8.x or 10.x) pattern
matching libraries may be linked to during the compilation of R. On
Windows, there is a custom implementation of *iconv* that has a set of
character encoding IDs not fully compatible with that on GNU/Linux: to
select the Polish locale, we are required to pass `"Polish_Poland"` to
`Sys.setlocale()` on Windows whereas `"pl_PL"` on Linux. Interestingly,
R can be built against the system *ICU* so that it uses its Collator for
comparing strings (e.g., using the "`<=`" operator). However, this is
only optional and does not provide access to any other Unicode services.

For example, let us consider the matching of "all letters" by means of
the built-in `gregexpr()` function and the [*TRE*](https://github.com/laurikari/tre/)
(`perl=FALSE`) and
*PCRE* (`perl=TRUE`) libraries using a POSIX-like and Unicode-style
character set (see {ref}`Sec:regex` for more details):

```r
x <- "AEZaezĄĘŻąęż"  # "AEZaez\u0104\u0118\u017b\u0105\u0119\u017c"
stri_sub(x, gregexpr("[[:alpha:]]", x, perl=FALSE)[[1]], length=1)
stri_sub(x, gregexpr("[[:alpha:]]", x, perl=TRUE)[[1]],  length=1)
stri_sub(x, gregexpr("\\p{L}", x, perl=TRUE)[[1]],       length=1)
```

On Ubuntu Linux 20.04 (UTF-8 locale), the respective outputs are:

```r
## [1] "A" "E" "Z" "a" "e" "z" "Ą" "Ę" "Ż" "ą" "ę" "ż"
## [1] "A" "E" "Z" "a" "e" "z"
## [1] "A" "E" "Z" "a" "e" "z" "Ą" "Ę" "Ż" "ą" "ę" "ż"
```

On Windows, when `x` is marked as UTF-8
(see {ref}`Sec:encoding`), the current author obtained:

```r
## [1] "A" "E" "Z" "a" "e" "z"
## [1] "A" "E" "Z" "a" "e" "z"
## [1] "A" "E" "Z" "a" "e" "z" "Ą" "Ę" "Ż" "ą" "ę" "ż"
```

And again on Windows using the Polish locale but `x` marked as
natively-encoded (CP-1250 in this case):

```r
## [1] "A" "E" "Z" "a" "e" "z" "Ę" "ę"
## [1] "A" "E" "Z" "a" "e" "z" "Ą" "Ę" "Ż" "ą" "ę" "ż"
## [1] "A" "E" "Z" "a" "e" "z" "Ę" "ę"
```

As we mention in {ref}`Sec:collator`, when *stringi* links to
*ICU* built from sources
(`install.packages("stringi", configure.args="--disable-pkg-config")`),
we are always guaranteed to get the same results on every platform.


**High Performance of *stringi*.**
Because of the aforementioned reasons, functions in *stringi* do not
refer to their base R counterparts. The operations that do not rely on
*ICU* services have been rewritten from scratch with speed and
portability in mind. For example, here are some timings of string
concatenation:

```{r performance1,cache=TRUE}
x <- stri_rand_strings(length(LETTERS) * 1000, 1000)
microbenchmark::microbenchmark(
  join2=stri_join(LETTERS, x, sep="", collapse=", "),
  join3=stri_join(x, LETTERS, x, sep="", collapse=", "),
  r_paste2=paste(LETTERS, x, sep="", collapse=", "),
  r_paste3=paste(x, LETTERS, x, sep="", collapse=", ")
)
```

Another example -- timings of fixed pattern searching:

```{r performance2,echo=-1,cache=TRUE}
set.seed(123)
x <- stri_rand_strings(100, 100000, "[actg]")
y <- "acca"
microbenchmark::microbenchmark(
  fixed=stri_locate_all_fixed(x, y),
  regex=stri_locate_all_regex(x, y),
  coll=stri_locate_all_coll(x, y),
  r_tre=gregexpr(y, x),
  r_pcre=gregexpr(y, x, perl=TRUE),
  r_fixed=gregexpr(y, x, fixed=TRUE)
)
```


**Different Default Arguments and Greater Configurability.**
Some functions in *stringi* have different, more natural default
arguments, e.g., `paste()` has `sep=" "` but `stri_join()` has `sep=""`.
Also, as there is no one-fits-all solution to all problems, many
arguments have been introduced for more detailed tuning.



**Preserving Attributes.**
Generally, *stringi* preserves no object attributes whatsoever, but a
user can make sure themself that this is the case, e.g., by
calling "`x[] <- stri_...(x, ...)`" or
"\``attributes<-`\``(stri_...(x, ...), attributes(x))`".
