Package {tantivyr}

tantivyr: Fast Full-Text Search for R with 'Tantivy'

Description

Index data frames and document collections and run fast full-text search entirely on your machine. 'tantivyr' wraps the 'Tantivy' 'Rust' search engine (a 'Lucene'-inspired library) to provide 'BM25' ranking, structured filters, snippet highlighting and incremental updates over an on-disk or in-memory index. First-class support is provided for stemming and stop words in Portuguese and English, making it well suited to public documents, news clippings, extracted 'PDF' text, transcripts and legal acts.

Author(s)

Maintainer: Andre Leite leite@castlab.org

Authors:

• Andre Leite leite@castlab.org

• Marcos Wasilew marcos.wasilew@gmail.com

• Hugo Vasconcelos hugo.vasconcelos@ufpe.br

• Carlos Amorim carlos.agaf@ufpe.br

• Diogo Bezerra diogo.bezerra@ufpe.br

See Also

Useful links:

• https://strategicprojects.github.io/tantivyr/

• https://github.com/StrategicProjects/tantivyr

• Report bugs at https://github.com/StrategicProjects/tantivyr/issues

Version of the bundled 'Tantivy' engine

Description

Report the version string of the 'Tantivy' 'Rust' search engine that this package was built against.

Usage

tantivy_version()

Value

A length-one character vector giving the version of the bundled 'Tantivy' 'Rust' search engine (for example "0.26.0").

Add documents to an index

Description

Adds the rows of data as documents. Only columns whose names match a schema field are used; other columns are ignored. Additions become searchable after tnt_commit().

Usage

tnt_add(idx, data)

Arguments

Value

The tnt_index, invisibly (so calls can be piped).

See Also

tnt_commit(), tnt_update()

Examples

sch <- tnt_schema(id = tnt_i64(), body = tnt_text(stemmer = "english"))
idx <- tnt_index(schema = sch)
df <- data.frame(id = 1:2, body = c("the quick fox", "lazy dogs sleep"))
idx |> tnt_add(df) |> tnt_commit()
tnt_num_docs(idx)

Commit pending changes

Description

Flushes buffered additions and deletions to the index and refreshes the reader so they become visible to tnt_search().

Usage

tnt_commit(idx)

Arguments

Value

The tnt_index, invisibly.

Examples

sch <- tnt_schema(body = tnt_text(stemmer = "english"))
idx <- tnt_index(schema = sch)
tnt_add(idx, data.frame(body = "hello")) |> tnt_commit()

Count matching documents

Description

Returns the total number of documents matching a query and optional filter, ignoring any result limit.

Usage

tnt_count(idx, query = "", fields = NULL, filter = NULL)

Arguments

Value

A single numeric count.

See Also

tnt_search()

Examples

idx <- tnt_index_df(
  data.frame(t = c("apple pie", "apple tart", "banana bread")),
  text = t, stemmer = "english"
)
tnt_count(idx, "apple")

Delete documents by field value

Description

Marks for deletion every document whose field equals the given value(s), using a field == value expression. Deletions take effect after tnt_commit(). For reliable deletion the field should be an exact field: a numeric/date field, or a text field created with stemmer = "raw".

Usage

tnt_delete(idx, condition)

Arguments

Value

The tnt_index, invisibly.

See Also

tnt_update()

Examples

sch <- tnt_schema(id = tnt_i64(), body = tnt_text(stemmer = "english"))
idx <- tnt_index(schema = sch)
tnt_add(idx, data.frame(id = 1:3, body = c("a", "b", "c"))) |> tnt_commit()
tnt_delete(idx, id == 2) |> tnt_commit()
tnt_num_docs(idx)

Define schema fields

Description

These constructors describe a single field in a tnt_schema(). Each returns a lightweight tnt_field object.

Usage

tnt_text(
  stored = TRUE,
  indexed = TRUE,
  fast = FALSE,
  stemmer = "none",
  stopwords = FALSE
)

tnt_i64(stored = TRUE, indexed = TRUE, fast = FALSE)

tnt_u64(stored = TRUE, indexed = TRUE, fast = FALSE)

tnt_f64(stored = TRUE, indexed = TRUE, fast = FALSE)

tnt_bool(stored = TRUE, indexed = TRUE, fast = FALSE)

tnt_date(stored = TRUE, indexed = TRUE, fast = FALSE)

tnt_json(stored = TRUE, indexed = TRUE)

Arguments

Value

A tnt_field object.

See Also

tnt_schema()

Examples

tnt_schema(
  id    = tnt_i64(),
  title = tnt_text(stemmer = "portuguese", stopwords = TRUE),
  body  = tnt_text(stemmer = "portuguese"),
  date  = tnt_date()
)

Create or open a search index

Description

Opens an existing on-disk index, or creates a new one from a tnt_schema(). With path = NULL an in-memory index is created (useful for tests and transient work).

Usage

tnt_index(path = NULL, schema = NULL, overwrite = FALSE, heap_mb = 128)

Arguments

Value

A tnt_index object.

See Also

tnt_index_df(), tnt_add(), tnt_search()

Examples

sch <- tnt_schema(
  id    = tnt_i64(),
  title = tnt_text(stemmer = "english"),
  body  = tnt_text(stemmer = "english")
)
idx <- tnt_index(schema = sch) # in-memory
idx

Index a data frame in one call

Description

A convenience wrapper that infers a schema from data, creates an index, adds every row and commits. Text columns are made searchable; filter columns are indexed for filtering and ordering; all other columns are stored so they are returned by tnt_search().

Usage

tnt_index_df(
  data,
  text,
  filters = NULL,
  stemmer = "none",
  stopwords = FALSE,
  stored = TRUE,
  path = NULL,
  overwrite = FALSE,
  heap_mb = 128
)

Arguments

Value

A committed tnt_index object.

See Also

tnt_index(), tnt_search()

Examples

df <- data.frame(
  id = 1:2,
  title = c("Orçamento público aprovado", "Reforma tributária avança"),
  year = c(2023L, 2024L)
)
idx <- tnt_index_df(df, text = title, filters = year, stemmer = "portuguese")
tnt_search(idx, "orcamento")

Index schema as a tibble

Description

Index schema as a tibble

Usage

tnt_index_info(idx)

Arguments

Value

A tibble with one row per field: name, kind, stored, indexed, tokenizer.

Examples

idx <- tnt_index_df(data.frame(t = "hi"), text = t)
tnt_index_info(idx)

Number of searchable documents

Description

Number of searchable documents

Usage

tnt_num_docs(idx)

Arguments

Value

A single numeric value (committed document count).

Examples

idx <- tnt_index_df(data.frame(t = "hello world"), text = t)
tnt_num_docs(idx)

Create a search schema

Description

Combine named tnt_field definitions into a schema that can be passed to tnt_index().

Usage

tnt_schema(...)

Arguments

Value

A tnt_schema object (a named list of tnt_fields).

See Also

tnt_field, tnt_index()

Examples

tnt_schema(
  id    = tnt_i64(),
  title = tnt_text(stemmer = "english"),
  body  = tnt_text(stemmer = "english")
)

Search an index

Description

Runs a BM25 full-text search and returns the top matches as a tibble. Supports structured filters, result ordering and snippet highlighting.

Usage

tnt_search(
  idx,
  query = "",
  limit = 10L,
  fields = NULL,
  filter = NULL,
  highlight = NULL,
  snippet_chars = 150L,
  order_by = NULL,
  desc = TRUE
)

Arguments

Value

A tibble with a score column, every stored field, and any requested ⁠*_snippet⁠ columns.

See Also

tnt_count(), tnt_index_df()

Examples

df <- data.frame(
  id = 1:3,
  title = c("Quick brown fox", "Lazy dog", "Brown bear"),
  year = c(2019L, 2021L, 2023L)
)
idx <- tnt_index_df(df, text = title, filters = year, stemmer = "english")
tnt_search(idx, "brown")
tnt_search(idx, "brown", filter = year >= 2021)
tnt_search(idx, "fox", highlight = title)

List supported stemmer languages

Description

List supported stemmer languages

Usage

tnt_stemmers()

Value

A character vector of language names accepted by the stemmer argument of tnt_text().

Examples

tnt_stemmers()

Update documents (delete then re-add)

Description

Replaces documents by deleting any existing document that shares a key value (the by field) with a row of data, then adding the rows of data. The by field must be an exact field (numeric/date, or text with stemmer = "raw"). Call tnt_commit() afterwards.

Usage

tnt_update(idx, data, by)

Arguments

Value

The tnt_index, invisibly.

See Also

tnt_add(), tnt_delete()

Examples

sch <- tnt_schema(id = tnt_i64(), body = tnt_text(stemmer = "english"))
idx <- tnt_index(schema = sch)
tnt_add(idx, data.frame(id = 1:2, body = c("old one", "old two"))) |>
  tnt_commit()
tnt_update(idx, data.frame(id = 1L, body = "new one"), by = id) |>
  tnt_commit()