---
title: "Romanisation, script conversion and kana"
author: 'Youzhi Yu<br><span style="font-weight: 400; font-size: 0.85em;">University of Chicago</span>'
bibliography: tidycjk.bib
output:
  rmarkdown::html_vignette:
    toc: true
    toc_depth: 3
vignette: >
  %\VignetteIndexEntry{Romanisation, script conversion and kana}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE, message = FALSE, warning = FALSE, comment = "#>"
)
```

```{r setup}
library(tidycjk)
```

## Four conversions, four different levels of trust

This vignette covers the transforms that change *what characters are there*
rather than how wide they are. Three of them share an implementation, ICU's
transliterators via **stringi** [@icu; @gagolewski2022stringi], and the jamo
pair applies the Unicode arithmetic directly. They should not be trusted
equally.

| Conversion | Reversible? | Uses context? | Trust |
|---|---|---|---|
| `cjk_jamo()` / `cjk_compose_jamo()` | yes, exactly | no | complete |
| `to_hiragana()` / `to_katakana()` | within one syllabary | no | high |
| `cjk_simplify()` | no | yes | good |
| `cjk_traditionalize()` | no | yes | good on characters, blind to regional vocabulary |
| `cjk_romanize()` | no | no | a key for Chinese; wrong for Japanese kanji |

The rest of this vignette is why.

## Hangul jamo: exact

A modern Hangul syllable is built from a leading consonant, a vowel and an
optional trailing consonant, and Unicode encodes all 11,172 combinations
precomposed. The relationship is arithmetic,
`SIndex = (LIndex * 21 + VIndex) * 28 + TIndex`, so the decomposition is
exact rather than tabulated: no table can be out of date and no syllable is
missed.

```{r jamo}
cjk_jamo("한글")
```

`cjk_jamo()` returns a list, one vector of jamo per string, so join each
element before composing it back:

```{r jamo-roundtrip}
back <- vapply(cjk_jamo("한글"), paste, character(1), collapse = "")
cjk_compose_jamo(back)
```

The round trip is exact, because both verbs apply the Hangul arithmetic and
nothing else. Text that is not Hangul passes through untouched, which is the
difference from Unicode normalisation: NFD would split the accented letter
below, and NFD and NFC alike would replace the compatibility ideograph with
its unified form, which in a Korean column means rewriting Hanja.

```{r jamo-other}
cjk_jamo("café")
cjk_compose_jamo("豈") == "豈"
```

Jamo are the right unit for questions the syllable hides: which initial
consonants a corpus favours, or whether two spellings differ only in a final
consonant. The same syllable counts three different ways, each right for a
different question. `한` is one character to `nchar()`, two terminal columns
to `cjk_width()`, and three jamo here:

```{r jamo-count}
nchar("한")
cjk_width("한")
lengths(cjk_jamo("한"))
```

## Kana: safe, with two things to know

Hiragana and katakana encode the same sounds, so the mapping never depends on
the surrounding words.

```{r kana}
to_hiragana("カタカナ")
to_katakana("ひらがな")

to_hiragana(to_katakana("ひらがな"))
```

That round trip is exact because the input was in one syllabary. On mixed
text it is not, and cannot be: converting to a single syllabary erases the
distinction between the two, and that distinction carries meaning, because
katakana marks loanwords, onomatopoeia and emphasis.

```{r kana-mixed-roundtrip}
x <- "アか"                      # one katakana, one hiragana
to_hiragana(x)                    # both become hiragana
to_katakana(to_hiragana(x))       # and cannot be told apart again
```

The second thing is that ICU makes a few choices of its own. The small
katakana `ヵ` and `ヶ` become the full-size hiragana `か` and `け`, and the
digraph `ヿ` (*koto*) is spelled out as two kana, so even text that was all
katakana does not always survive the trip:

```{r kana-icu}
to_hiragana("ヵヶ")
to_katakana("ヿ")
```

So the conversion is safe in the sense that matters, which is that it never
guesses from context, but run it in one direction, as a normalisation,
rather than expecting to undo it.

Kanji, Latin and punctuation are untouched, so it is safe on a mixed column:

```{r kana-mixed}
to_katakana("日本語のtext です")
```

This is the normalisation you want before grouping Japanese text, where the
same word is often written either way for emphasis.

Halfwidth katakana is handled too, and composed on the way through. `ｶﾞ` is
two code points, the kana and the halfwidth voiced mark U+FF9E, and comes
back as the single character `ガ`:

```{r kana-width}
to_hiragana("ｶﾀｶﾅ")
to_katakana("ｶﾞ")
nchar(to_katakana("ｶﾞ"))
```

## Simplified and traditional Han: right on characters, blind to vocabulary

```{r han}
cjk_simplify("漢字")
cjk_traditionalize("汉字")
```

It is better than it looks. Simplified to traditional is genuinely
one-to-many (simplified `发` is `發`, "to send", or `髮`, "hair", and `干` is
`乾`, `幹` or `干`), and ICU resolves these **from context**, not character
by character:

```{r han-context}
cjk_traditionalize(c("头发", "发送"))   # hair, send
cjk_traditionalize(c("干净", "树干"))   # clean, tree trunk
cjk_traditionalize(c("后天", "皇后"))   # after, empress: 后 splits both ways
```

All six are right, and so were the fourteen other words checked for this
release, including `面条`/`面对` and `里面`/`公里`.

What ICU does *not* do is substitute regional vocabulary. The two standards
differ in the words they use, not only in glyph shape, and that is lexical:

```{r han-limit}
cjk_traditionalize(c("软件", "鼠标", "网络"))
```

Those are the correct characters and the wrong words in Taiwan, where the
terms are `軟體`, `滑鼠` and `網路`. No character mapping can reach that, and
it is what [OpenCC](https://github.com/BYVoid/OpenCC) [@opencc] and its
regional configurations exist for. Use these functions for script
conversion; reach for OpenCC when you need Taiwanese or Hong Kong idiom.

## Romanisation: depends entirely on the language

```{r romanize}
cjk_romanize("中文")
cjk_romanize("中文", ascii = TRUE)
```

For Chinese this is pinyin, with tone marks unless you ask for ASCII.
`ascii = TRUE` gives a sortable, greppable key for a CJK column, which is
often the real reason to romanise at all.

```{r romanize-scripts}
cjk_romanize(c("こんにちは", "안녕하세요"))
```

Three caveats, in increasing order of severity.

**Han readings are per character.** A character with several readings gets
ICU's preferred one regardless of the word it appears in, so common words
come back wrong:

```{r romanize-polyphone}
cjk_romanize(c("银行", "音乐"))   # bank, music: yín háng and yīn yuè
```

That is fine for a sort key and wrong for pinyin you show a reader.

**No boundary between scripts.** ICU spaces Han syllables but does not insert
a break where the script changes:

```{r romanize-mixed}
cjk_romanize("漢字カナ")
```

Segment first with `cjk_segment()` if you need word boundaries in the output.

**Japanese kanji come back as Chinese.** This is the severe one. ICU routes
every Han character through pinyin regardless of the surrounding language, so
`日本語` romanises to `rì běn yǔ` rather than `nihongo`, and `私` to `sī`
rather than `watashi`:

```{r romanize-ja}
cjk_romanize("私は日本語を話します")
```

That is not an approximation of the Japanese reading; it is the Chinese one.
The kana in the same string romanise correctly, which makes the output look
plausible at a glance and is exactly what makes it dangerous. Note too that
the particle `は` gives `ha`, which is how it is written rather than how it
is said.

For Japanese, use a morphological analyser that carries readings: **gibasa**
binds MeCab and returns them [@kudo2004mecab; @gibasa]. `cjk_romanize()` is
for Chinese, for kana, and for producing an ASCII sort key.

## Width forms are a separate concern

`to_halfwidth()` and `to_fullwidth()` change width and nothing else. The usual
advice for fullwidth text is an `NFKC` pass, which does fix width, and also
rewrites ligatures, Roman numerals and circled numbers, none of which a
change of width asked for. When that wider fold *is* what you want, it is
`cjk_normalize()` in the next section rather than a side effect here:

```{r width-forms}
as.numeric(to_halfwidth("１２３"))

to_halfwidth("½ Ⅸ ①")   # NFKC rewrites all three; this leaves them
```

The one composition it does perform is voiced halfwidth katakana, where a
single syllable is written as two code points. Composing them is the
difference between text that matches a literal and text that silently does
not:

```{r voiced}
x <- "ｶﾞ"
nchar(x)
nchar(to_halfwidth(x))
to_halfwidth(x) == "ガ"
```

## Normalisation is the fourth axis

Width, script and syllabary are three axes; the code points themselves are
the fourth. `cjk_normalize()` applies the Unicode normalisation forms
[@uax15], which is what decides whether two strings that render identically
compare equal:

```{r normalize}
both <- c("が", "が")   # composed, then KA + voiced mark
nchar(both)
nchar(cjk_normalize(both))
```

`"nfkc"` is the blunt instrument the width verbs exist to avoid, and it is the
right tool when matching rather than display is the goal:

```{r nfkc}
cjk_normalize("ＡＢ　①", form = "nfkc")
cjk_normalize("ＡＢ　①", form = "nfc")
```

**Canonical normalisation is not a no-op on Han.** Compatibility ideographs
have singleton canonical mappings, so `"nfc"` rewrites them just as `"nfkc"`
does. This is the part that surprises people who reach for NFC because it is
supposed to be the safe one:

```{r compat}
block <- vapply(c(0xF900:0xFA6D, 0xFA70:0xFAD9), intToUtf8, character(1))
sum(cjk_normalize(block, "nfc") != block)   # folded away by plain NFC
sum(cjk_normalize(block, "nfc") == block)   # and the survivors
```

The twelve survivors are unified ideographs that were encoded in the
compatibility block by accident and have no mapping to apply. So the block is
neither preserved nor uniformly folded: if the distinction between a
compatibility ideograph and its unified form means something in your data, as
it can in Korean and Japanese name records, keep the original column.

Variation selectors are the other thing that defeats an exact match, and four
of the five forms leave them alone:

```{r ivs}
ivs <- "辻\U000E0100"
nchar(cjk_normalize(ivs))
nchar(cjk_normalize(ivs, drop_variation_selectors = TRUE))
```

They are dropped *before* the form is applied, not after: a selector has
combining class zero and blocks canonical composition across itself, so
stripping one afterwards can leave text that is no longer in the form you
just asked for.

## References