---
title: "Segmenting CJK text"
author: 'Youzhi Yu<br><span style="font-weight: 400; font-size: 0.85em;">University of Chicago</span>'
bibliography: tidycjk.bib
output:
  rmarkdown::html_vignette:
    toc: true
    toc_depth: 3
vignette: >
  %\VignetteIndexEntry{Segmenting CJK text}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE, message = FALSE, warning = FALSE, comment = "#>"
)
```

```{r setup}
library(tidycjk)
```

## The problem

English tokenisation is mostly a solved problem because English writes its
word boundaries down. You split on whitespace and you are close enough. In
Chinese and Japanese there is no such signal:

> 我今天很開心

Six characters, no spaces, four words: `我` (I), `今天` (today), `很` (very),
`開心` (happy). Nothing in the *encoding* tells you that `今` and `天` belong
together while `很` stands alone. That is a fact about Chinese, learned from a
corpus, and Unicode makes no claim about it. Unicode Annex #29 defines word
boundaries for scripts that mark them and explicitly defers to dictionaries
for those that do not [@uax29].

This is why `tidycjk` does not have a single `tokenize()`. It has an engine
slot.

## Choosing an engine

```{r engines}
cjk_segmenters()
```

### `"icu"`: dictionary-based, and already installed

ICU ships dictionary-based break iterators for Chinese and Japanese, and
**stringi** ships ICU [@icu; @gagolewski2022stringi]. Since **stringi** is
already a hard dependency of this package, real word segmentation costs
nothing extra:

```{r icu}
cjk_segment("我今天很開心", engine = "icu")

cjk_segment("今天天氣很好我們去公園散步", engine = "icu")
```

Japanese works out of the box. `locale` is accepted but does not change the
result; "`locale` does not pick the dictionary" below says why:

```{r icu-ja}
cjk_segment("今日は良い天気ですね", engine = "icu", locale = "ja")
```

Anything in `...` reaches the engine, which is how `locale` gets there.

What `"icu"` gives you is surface forms and nothing else: no part of speech,
no lemma, no user dictionary, and models lighter than the ones a dedicated
analyser carries. It is the right first choice and not the last word.

### `"character"`: the honest baseline

```{r character}
cjk_segment("我今天很開心", engine = "character")
```

One token per CJK character; runs of non-CJK text split on whitespace.
Whitespace is never a token, the ideographic space U+3000 included, even
though `has_cjk()` counts it as CJK, because it is a space and treating it as
a word would inflate every count.

This is *character* tokenisation, not word segmentation. For Chinese it cuts
two-character words in half. It is genuinely useful when a per-character unit
is what you want (character frequency, or a character-level model) and
misleading if you call its output "words".

```{r mixed}
cjk_segment("hello 中文 world", engine = "character")
```

### Why there is no default

`engine` is required. There is a real temptation to default it to
`"character"`, and the reason not to is that it answers a *different question*
from the one a caller asking for words is asking, while looking like an answer
to theirs. Silence there is the failure mode this package exists to prevent.

`"icu"` arriving does not change that. It would be a defensible default in a
way `"character"` never was, but a default is a decision made on the
caller's behalf about what a word is, and this package would rather that
decision were visible in the calling code.

### `locale` does not pick the dictionary

It is natural to assume it does, and it does not. ICU applies one combined
Chinese–Japanese word list to Han and kana runs, selected by the *script of
the text* rather than by the locale [@icu]. Segmenting seven CJK strings
under `"zh"`, `"ja"`, `"ko"`, `"en"` and the session default gives identical
output in every case.

`locale` is still accepted and forwarded, because ICU may use it elsewhere
and a locale ICU has no break data for is an error rather than a silent
fallback, but do not expect `locale = "ja"` to change how Japanese segments.

## In a pipeline

`cjk_tokens()` takes `(data, column)` and returns one row per token, in the
manner of **tidytext**'s `unnest_tokens()` [@silge2016tidytext]:

```{r tokens}
posts <- data.frame(
  id = 1:3,
  text = c("我今天很開心", "今天天氣很好", "hello 中文 world")
)

cjk_tokens(posts, text, engine = "icu")
```

Grouping is dropped, and other columns are carried along, so the usual
counting follows directly:

```{r counting}
tok <- cjk_tokens(posts, text, engine = "icu")
sort(table(tok$token), decreasing = TRUE)[1:5]
```

## Registering another engine

An engine is any function of `(x, ...)` returning a list the same length as
`x`, each element a character vector of tokens. `NA` in gives
`NA_character_`; the empty string gives `character(0)`. `cjk_segment()` checks
the shape and complains if an engine breaks the contract, rather than passing
a malformed result downstream: it checks the list itself, its length, and the
type of each element. An atomic element is coerced, so integers or a factor are fine;
a list element is refused, because `as.character()` deparses a list rather
than coercing it and `list(c(1, 2))` would otherwise arrive as the single
token `"c(1, 2)"`.

A leading byte-order mark survives both built-in engines, which takes work:
the stringi calls underneath them each read a leading U+FEFF as a byte-order
mark and drop it. An engine you register yourself gets no such help: if it
calls stringi, it needs the same guard.

### gibasa, for Japanese

[gibasa](https://CRAN.R-project.org/package=gibasa) binds MeCab
[@kudo2004mecab; @gibasa] and is on CRAN, though it needs MeCab itself and a
dictionary installed on the system. It gives part of speech and lemma, which
ICU does not:

```r
register_cjk_segmenter("mecab", function(x, ...) {
  lapply(x, function(s) {
    if (is.na(s)) return(NA_character_)
    if (!nzchar(s)) return(character(0))
    gibasa::tokenize(data.frame(doc_id = 1, text = s))$token
  })
})

cjk_segment("今日は良い天気ですね", engine = "mecab")
```

### jiebaR, for Chinese

jiebaR was archived from CRAN on 2025-05-01, so it cannot be a dependency of
a CRAN package; install it from source with
`remotes::install_github("qinwf/jiebaR")`. Then:

```r
register_cjk_segmenter("jiebar", function(x, ...) {
  worker <- jiebaR::worker(...)
  lapply(x, function(s) {
    if (is.na(s)) return(NA_character_)
    if (!nzchar(s)) return(character(0))
    as.character(jiebaR::segment(s, worker))
  })
})
```

### A caution about argument matching

`...` sits after `engine` in `cjk_segment()`, and after `data` and `col` in
`cjk_tokens()`, so R's partial matching claims a prefix of one of those before
the dots see it. `cjk_tokens(df, text, "mine", c = 1)` matches `c` to `col`,
pushes bare `text` into `engine`, resolves it to `graphics::text()`, which is
a function and so is accepted, and the error you get is about plotting. Name
engine arguments so they are not a prefix of `col`, `data` or `engine`, or
capture the setting in a closure:

```r
register_cjk_segmenter("mine", function(x, ...) my_segmenter(x, cutoff = 1))
```

## Comparing engines

Because engines share a contract, comparing them is a `lapply()`:

```{r compare}
x <- "今天天氣很好"
vapply(c("icu", "character"),
       function(e) paste(cjk_segment(x, engine = e)[[1]], collapse = " | "),
       character(1))
```

That is the intended way to decide: run the candidates over a sample of *your*
corpus and read the disagreements, rather than trusting a benchmark computed
on someone else's text.

## References