---
title: "Getting started with icebergr"
author: "Youzhi Yu<br><span style='font-size:85%;'>University of Chicago</span>"
bibliography: vignette_reference.bib
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Getting started with icebergr}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)
# Handles and catalogs print their `location:` / `warehouse:`, which is a
# temporary directory unique to whichever machine built this page. Printing it
# would bake that path into the shipped HTML, so it is replaced here.
show <- function(x) {
  writeLines(gsub(tempdir(), "<tempdir>", capture.output(print(x)), fixed = TRUE))
}
```

Every example here runs offline against a table built on your own machine. No
catalog server, no network, no credentials.

```{r setup}
library(icebergr)
```

## What this package is for

Iceberg [@icebergspec] is the open table format that Snowflake, Databricks,
BigQuery, AWS and Dremio have all standardised on. Apache maintains clients for
Java, Python, Rust and Go, but not for R, where Iceberg has been reached through
a query engine such as DuckDB [@raasveldt2019duckdb]. A client is a different
thing: it hands you the table itself, with its snapshots, its schema as of each
of them and its scan plan, as R objects.

`icebergr` binds `iceberg-rust` [@icebergrust] directly through extendr
[@extendr]. Arrow is the interchange layer, so scan results arrive in R over the
C stream interface [@arrowcdata] without a serialisation round trip.

`vignette("table-format")` explains the format itself, and why a client rather
than an exporter is the thing worth having.

## A table to work with

`icebergr_example_table()` builds a real Iceberg table in a temporary warehouse
directory: two appends, so there is history to travel through and more than one
data file for a filter to prune.

```{r}
tbl <- icebergr_example_table(rows = 200)
show(tbl)
```

It is generated rather than shipped because Iceberg records absolute paths in its
metadata and its Avro manifests, so a table built on one machine does not
resolve on another.

## Inspecting a table

```{r}
icebergr_schema(tbl)
```

`type` is the Iceberg type, not the R type. The mapping to R happens on read.

An unpartitioned table returns zero partition fields:

```{r}
icebergr_partitions(tbl)
```

## Reading

`icebergr_scan()` describes a read; `icebergr_collect()` performs it.

```{r}
icebergr_collect(icebergr_scan(tbl, limit = 5))
```

Scanning the whole table is common enough to have a shorthand:

```{r}
nrow(icebergr_collect(tbl))
```

### Pushdown

`filter` and `select` are pushed into scan planning. This is the entire
performance argument for Iceberg over reading raw Parquet: manifests carry
per-file statistics, so whole files and row groups are eliminated before any
bytes are read.

```{r}
icebergr_collect(
  icebergr_scan(
    tbl,
    filter = id > 1000 & amount > 900,
    select = c("id", "event", "amount")
  )
)
```

Filters may use `==`, `!=`, `<`, `<=`, `>`, `>=`, `&`, `|`, `!`, `%in%`,
`is.na()`, `is.nan()` and `startsWith()` -- the last only against a `string`
column, since Iceberg defines a prefix comparison for no other type. A bare name
is read as a column when the table has one of that name, and otherwise evaluated
in the calling environment:

```{r}
# The second append holds ids 1001 to 1200, so this keeps the last fifty.
threshold <- 1150
icebergr_collect(icebergr_scan(tbl, filter = id > threshold, select = "id"))
```

Anything more elaborate is refused rather than quietly ignored, so you always
know whether a filter was pushed down:

```{r, error = TRUE}
icebergr_collect(icebergr_scan(tbl, filter = sqrt(amount) > 10))
```

### Verifying that pushdown happened

Comparing results proves nothing: a filter applied in R afterwards gives the same
rows. What distinguishes pushdown is how much was planned to be read.

```{r}
icebergr_scan_plan(icebergr_scan(tbl))[, c("record_count", "file_size_in_bytes")]
```

```{r}
icebergr_scan_plan(icebergr_scan(tbl, filter = id > 1000))[, c("record_count")]
```

Fewer files, and fewer records, than the table holds.

### One caveat about `limit`

`limit` is **not** pushdown. `iceberg-rust` has no row limit in its scan API, so
the same files are planned and rows are counted as batches arrive. It bounds how
much is decoded, not how much is planned. `print()` says so:

```{r}
icebergr_scan(tbl, filter = id > 1000, limit = 10)
```

## Time travel

Every write creates a snapshot.

```{r}
history <- icebergr_snapshots(tbl)
history[, c("snapshot_id", "operation", "added_records", "total_records")]
```

Snapshot ids are **character**, not numeric. Iceberg assigns them as random
64-bit integers, and an R numeric carries only 53 bits, so passing one through a
double would silently select the wrong snapshot.

Read an earlier state by id:

```{r}
nrow(icebergr_collect(icebergr_scan(tbl, snapshot_id = history$snapshot_id[[1]])))
```

Or by time, which is resolved against the history to the snapshot that was
current at that moment:

```{r}
nrow(icebergr_collect(icebergr_scan(tbl, as_of = history$timestamp[[1]])))
```

Iceberg records a schema per snapshot, so `filter` and `select` are resolved
against the schema of the snapshot actually being read rather than the current
one. A column another engine has since renamed or dropped is therefore still
nameable as of the snapshot that had it, and `icebergr_schema()` will show you
what those columns were:

```{r}
icebergr_schema(tbl, snapshot_id = history$snapshot_id[[1]])
```

## Writing

Writes are append-only. Nothing already in the table is rewritten or removed.

```{r}
new_rows <- data.frame(
  id = c(9001L, 9002L),
  event = c("purchase", "refund"),
  amount = c(42.5, -12.25),
  day = as.Date(c("2024-07-01", "2024-07-02")),
  recorded_at = as.POSIXct(c("2024-07-01 09:00:00", "2024-07-02 10:30:00"), tz = "UTC")
)

tbl <- icebergr_append(tbl, new_rows)
nrow(icebergr_collect(tbl))
```

`icebergr_append()` returns an *updated* handle rather than mutating the old one,
so reassign it. The old handle still reads the older snapshot, which is
occasionally useful and never surprising.

Columns are matched by name, not position, so order does not matter. A column the
table does not have is an error rather than a silent drop:

```{r, error = TRUE}
icebergr_append(tbl, transform(new_rows, unexpected = 1))
```

### Creating a table

A data frame is enough to define a schema. Only the column names and types are
used; no rows are written.

```{r}
warehouse <- tempfile("warehouse")
dir.create(warehouse)

catalog <- icebergr_catalog("memory", warehouse = warehouse)
icebergr_create_namespace(catalog, "analytics")

measurements <- data.frame(
  sensor = character(),
  reading = double(),
  taken_at = as.POSIXct(character(), tz = "UTC")
)

sensors <- icebergr_create_table(catalog, "analytics.measurements", measurements)
icebergr_schema(sensors)
```

## Type fidelity

Most R types survive unchanged. The exception worth knowing is `factor`: Iceberg
has no dictionary type, so levels cannot be carried and values come back as
`character`.

```{r}
types <- data.frame(
  i = 1L,
  d = 1.5,
  s = "text",
  b = TRUE,
  day = as.Date("2024-01-01"),
  ts = as.POSIXct("2024-01-01 12:00:00", tz = "UTC"),
  f = factor("a", levels = c("a", "b"))
)

type_tbl <- icebergr_create_table(catalog, "analytics.types", types)
type_tbl <- icebergr_append(type_tbl, types)

vapply(icebergr_collect(type_tbl), function(x) class(x)[[1]], character(1))
```

## Knowing what is not supported

`icebergr` is deliberately narrow. Rather than discovering a gap at
runtime, ask:

```{r}
support <- icebergr_spec_support()
support$spec_versions
```

```{r}
features <- support$features
unsupported <- !is.na(features$supported) & !features$supported
features[unsupported, c("feature", "reason")]
```

Some of those are absent from `iceberg-rust` itself, not just from this package;
the `reason` column says which.

## Where next

Six vignettes go deeper on one theme each:

* `vignette("table-format")`: what Iceberg is, and what it fixes
* `vignette("pushdown")`: how a filter prunes files and row groups
* `vignette("time-travel")`: snapshots, timestamps and rollbacks
* `vignette("writing")`: appends, table creation, and what is refused
* `vignette("types")`: the R, Arrow and Iceberg type correspondence
* `vignette("catalog-configuration")`: REST, AWS Glue, object storage,
  and how credentials are handled

# References