---
title: "Working with RCDF Files in R"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{rcdf}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)
```

This vignette will walk you through the key features of the package, including how to encrypt and save your data in RCDF format, how to decrypt and read RCDF files, and how to export data to other common formats.

## Writing data to RCDF format

The core function for writing data to the RCDF format is `write_rcdf()`. This function encrypts your data using AES encryption, generates encrypted metadata for version control using RSA encryption, and saves the data as encrypted Parquet files inside a zip archive. This ensures that the data is stored securely and can only be decrypted using the correct key.

<br>
**Usage**: 

```{r, eval=FALSE}
write_rcdf(
  data,
  path,
  pub_key,
  ...,
  metadata = list(),
  ignore_duplicates = TRUE
)
```

**Parameters**:

- `data`: A list of data frames or tables to be written to RCDF format. Each element of the list represents a record.
- `path`: The path where the RCDF file will be written. The file will be saved with a `.rcdf` extension if not already specified.
- `pub_key`: The public RSA key used to encrypt the AES encryption keys.
- `...`: Additional arguments passed to helper functions if needed.
- `metadata`: A list of metadata to be included in the RCDF file. Can contain system information or other relevant details.
- `ignore_duplicates`: A logical flag. If `TRUE`, a warning is issued when duplicates are found. If `FALSE`, the function stops with an error.


```{r, eval=FALSE}
library(rcdf)
# Sample data (list of data frames)
data <- rcdf_list()
data$table1 = data.frame(x = 1:10, y = letters[1:10])
data$table2 = data.frame(a = rnorm(10), b = rnorm(10))

# You can generate a new RSA key pair using the following function:
key <- generate_rsa_keys(
  path = "path/to/rsa-keys", 
  password = "1234" # this is optional
)

# Write the data to an RCDF file
write_rcdf(data = data, path = "path/to/rcdf_file.rcdf", pub_key = key)
```


In this example:

- `data` is a list containing two data frames. These will be encrypted and saved as separate Parquet files within the RCDF.
- `pub_key` is the RSA public key used to encrypt the AES keys. The AES keys are used for encrypting the data in a fast and secure manner.

The `write_rcdf()` function will create a zip archive containing the encrypted Parquet files and metadata, then save it to path.


## Reading and decrypting RCDF data

To read and decrypt an RCDF file, you can use the `read_rcdf()` function. This function extracts the encrypted Parquet files from the RCDF archive, decrypts them using the provided decryption key, and loads the data back into R as an RCDF object.

<br>
**Usage**: 

```{r, eval=FALSE}
read_rcdf(
  path, 
  decryption_key, 
  ..., 
  password = NULL,
  metadata = list(),
  ignore_duplicates = TRUE,
  recursive = FALSE,
  return_meta = FALSE
)
```

**Parameters**:

- `path`: A string specifying the path to the RCDF archive (zip file). If a directory is provided, all `.rcdf` files within that directory will be processed.
- `decryption_key`: The key used to decrypt the RCDF contents. This can be an RSA or AES key, depending on how the RCDF was encrypted.
- `...`: Additional parameters passed to other functions, if needed.
- `password`: A password used for RSA decryption (optional).
- `metadata`: An optional metadata object containing data dictionaries and value sets. This metadata is applied to the data if provided. If `NULL`, the metadata from the RCDF file (added during write), if available, is used.
- `ignore_duplicates`: A logical flag. If `TRUE`, a warning is issued when duplicates are found, based on the primary key/s defined during creation of RCDF file. If `FALSE`, the function stops with an error.
- `recursive`: A logical flag. If `TRUE`, the function will recursively look for RCDF files in subdirectories of the specified path.
- `return_meta`: Logical. If `TRUE`, the metadata will be returned as an attribute of the RCDF object.


```{r, eval=FALSE}
# Using sample RCDF data
dir <- system.file("extdata", package = "rcdf")

# mtcars.rcdf is a sample RCDF file included in the package for demonstration purposes only.  
rcdf_path <- file.path(dir, 'mtcars.rcdf')

# This is also a sample key built into the package.
private_key <- file.path(dir, 'sample-private-key.pem')

rcdf_data <- read_rcdf(path = rcdf_path, decryption_key = private_key)
rcdf_data

```

In this example:

- `path` is the path to the RCDF file that contains the encrypted data.
- `decryption_key` is the key used to decrypt the AES keys and Parquet files. If the RCDF was encrypted using RSA, you’ll need the private RSA key to decrypt it.

The `read_rcdf()` function returns an RCDF object, which is essentially a list of decrypted Parquet files (one for each data frame in the original data) along with metadata about the file.


When creating an RSA key pair, you can choose to protect the private key with a password. If your RCDF file was encrypted using RSA and the private key is password-protected, you will need to provide the password when reading the RCDF file.

```{r, eval=FALSE}
# Using encrypted/password protected private key
rcdf_path_pw <- file.path(dir, 'mtcars-pw.rcdf')
private_key_pw <- file.path(dir, 'sample-private-key-pw.pem')
pw <- '1234'

rcdf_data_with_pw <- read_rcdf(
  path = rcdf_path_pw,
  decryption_key = private_key_pw,
  password = pw
)
rcdf_data_with_pw

```

## Exporting data to other formats

Once the data has been decrypted and read into R, you can export it to other formats using the `write_rcdf_as()` or `write_rcdf_*()` family of functions. These functions support a wide variety of common formats, including CSV, TSV, JSON, Excel, Stata, SPSS, and SQLite.

### Exporting data to CSV format

The `write_rcdf_csv()` function allows you to export data stored in an RCDF object to CSV files. This is useful when you need to share or process the data in a non-encrypted, readable format.

<br>
**Usage**: 

```{r, eval=FALSE}
write_rcdf_csv(data, path, ..., parent_dir = NULL)
```

**Parameters**:

- `data`: The RCDF object that contains the decrypted data. This is the data you obtained from calling `read_rcdf()` or other decryption methods.
- `path`: The target directory or file where the CSV files will be saved.
- `...`: Additional arguments passed to the `write.csv()` function for customizing the CSV export (e.g., setting delimiters, row names, etc.).
- `parent_dir`: An optional parent directory to be included in the path where the files will be written.


```{r, eval=FALSE}
write_rcdf_csv(data = rcdf_data, path = "path/to/output", row.names = FALSE)
```

This will save each table in the RCDF object as a separate CSV file in the specified directory.


### Exporting data to TSV format

The `write_rcdf_tsv()` function is similar to the CSV export function but writes the data as tab-separated values (TSV) files.

<br>
**Usage**: 

```{r, eval=FALSE}
write_rcdf_tsv(data, path, ..., parent_dir = NULL)
```

**Parameters**:

- `data`: The decrypted RCDF object containing the data to export.
- `path`: The target directory or file for the output TSV files.
- `...`: Additional arguments for customizing the TSV export passed to the `write.table()` function (e.g., setting delimiters, handling row names).
- `parent_dir`: An optional parent directory to be included in the path where the files will be written.


```{r, eval=FALSE}
write_rcdf_tsv(data = rcdf_data, path = "path/to/output", row.names = FALSE)
```

This function will save each data frame in the RCDF object as a separate TSV file in the target location.


### Exporting data to JSON format

The `write_rcdf_json()` function allows you to export the decrypted RCDF data to JSON format. This is useful when working with APIs or other systems that require data in JSON.

<br>
**Usage**: 

```{r, eval=FALSE}
write_rcdf_json(data, path, ..., parent_dir = NULL)
```

**Parameters**:

- `data`: The decrypted RCDF object.
- `path`: The target directory or file for saving the JSON files.
- `...`: Additional arguments to customize the JSON export passed to `jsonlite::toJSON()` (such as specifying indentation or compactness of the JSON output).
- `parent_dir`: An optional parent directory to be included in the path where the files will be written.


```{r, eval=FALSE}
write_rcdf_json(data = rcdf_data, path = "path/to/output", pretty = TRUE)
```

This will convert each data frame in the RCDF object into a separate JSON file and save them in the specified directory. The `pretty = TRUE` option ensures that the output JSON files are human-readable with proper indentation.


### Exporting data to Parquet format

The `write_rcdf_parquet()` function exports the decrypted data back into the Parquet format. Parquet is a columnar storage format that is highly efficient for big data processing.

<br>
**Usage**: 

```{r, eval=FALSE}
write_rcdf_parquet(data, path, ..., parent_dir = NULL)
```

**Parameters**:

- `data`: The decrypted RCDF object.
- `path`: The directory or file path where the Parquet files will be saved.
- `...`: Additional arguments passed to the `write_parquet()` function for customization, such as specifying compression type.
- `parent_dir`: An optional parent directory to be included in the path where the files will be written.

```{r, eval=FALSE}
write_rcdf_parquet(data = rcdf_data, path = "path/to/output")
```

This function will write each data frame in the RCDF object into separate Parquet files, storing them in the specified directory.

### Exporting data to Excel format

The `write_rcdf_xlsx()` function is used to export the decrypted RCDF data to Excel (.xlsx) format. It’s helpful when sharing data with users who prefer spreadsheet software.

<br>
**Usage**: 

```{r, eval=FALSE}
write_rcdf_xlsx(data, path, ..., parent_dir = NULL)
```

**Parameters**:

- `data`: The decrypted RCDF object.
- `path`: The directory or file path where the Excel file will be saved.
- `...`: Additional arguments to customize the Excel file export in the `openxlsx` package.
- `parent_dir`: An optional parent directory to be included in the path where the files will be written.


```{r, eval=FALSE}
write_rcdf_excel(data = rcdf_data, path = "path/to/output.xlsx", sheetName = "Sheet1")
```


### Exporting data to Stata format

The `write_rcdf_dta()` function allows you to export the data to Stata’s .dta file format. This is useful for users who need to work with the data in Stata.

<br>
**Usage**: 

```{r, eval=FALSE}
write_rcdf_dta(data, path, ..., parent_dir = NULL)
```

**Parameters**:

- `data`: The decrypted RCDF object.
- `path`: The path where the Stata .dta file will be saved.
- `...`: Additional arguments passed to the `write.dta()` function (e.g., specifying version of Stata).
- `parent_dir`: An optional parent directory to be included in the path where the files will be written.


```{r, eval=FALSE}
write_rcdf_dta(data = rcdf_data, path = "path/to/output")
```

### Exporting data to SPSS format

The `write_rcdf_sav()` function is for exporting the decrypted RCDF data to SPSS’s .sav file format.

<br>
**Usage**: 

```{r, eval=FALSE}
write_rcdf_sav(data, path, ..., parent_dir = NULL)
```

**Parameters**:

- `data`: The decrypted RCDF object.
- `path`: The path where the .sav file will be saved.
- `...`: Additional arguments for customizing the SPSS file export.
- `parent_dir`: An optional parent directory to be included in the path where the files will be written.

```{r, eval=FALSE}
write_rcdf_sav(data = rcdf_data, path = "path/to/output")
```


### Exporting data to SQLite database format

The `write_rcdf_sqlite()` function allows you to export the decrypted RCDF data to an SQLite database (with .db extension). Each data frame is saved as a table within the SQLite database.

<br>
**Usage**: 

```{r, eval=FALSE}
write_rcdf_sqlite(data, path, ..., parent_dir = NULL)
```

**Parameters**:

- `data`: The decrypted RCDF object.
- `path`: The path where the SQLite database file will be created.
- `...`: Additional arguments for customizing the SQLite export.
- `parent_dir`: An optional parent directory to be included in the path where the files will be written.


```{r, eval=FALSE}
write_rcdf_sqlite(data = rcdf_data, path = "path/to/output")
```

### Exporting data to multiple formats simultaneously

The `write_rcdf_as()` function allows you to export decrypted RCDF data into multiple file formats simultaneously.

<br>
**Usage**: 

```{r, eval=F}
write_rcdf_as(data, path, formats, ...)
```

**Parameters**:

- `data`: A named list or RCDF object. Each element should be a table or tibble-like object (typically a `dbplyr` or `dplyr` table).
- `path`: The target directory where output files should be saved.
- `formats`: A character vector of file formats to export to. Supported formats include: `"csv"`, `"tsv"`, `"json"`, `"parquet"`, `"xlsx"`, `"dta"`, `"sav"`, and `"sqlite"`.
- `...`: Additional arguments passed to the respective writer functions.


