---
title: "Feature selection: crabs"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Feature selection: crabs}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include = FALSE}
knitr::opts_chunk$set (collapse = TRUE, comment = "#>", fig.width = 6, fig.height = 4.2,
                       fig.align = "center")
optional = c ("MASS", "e1071")
available = all (sapply (optional, requireNamespace, quietly = TRUE))
knitr::opts_chunk$set (eval = available)
```

```{r, echo = FALSE, eval = !available, results = "asis"}
cat ("**Note.** This vignette needs the following packages, some of which are missing:",
     paste (optional, collapse = ", "), "-- the code is shown but not run.\n")
```

One of the case studies of the *Fouille de données (M2 SID)* course, for which `fdm2id` was
written. Filters against a wrapper, on five measurements that are almost perfectly correlated
and two different targets.

The other case studies are listed by `vignette (package = "fdm2id")`; they use the same
handful of functions on other data, and can be read in any order.

```{r, message = FALSE, warning = FALSE}
library (fdm2id)
```

# The data

The `crabs` dataset of the `MASS` package, on *Leptograpsus variegatus* crabs collected at
Fremantle, Australia. Two colour forms, orange and blue, and both sexes. Five measurements were
taken on each individual (in mm): the size of the frontal lobe (`FL`), the rear width (`RW`),
the length of the carapace (`CL`), its width (`CW`) and the depth of the body (`BD`).

There are two targets rather than one: the species (`sp`, `O` or `B`) and the sex (`sex`, `F`
or `M`), with 50 males and 50 females of each species.

```{r}
data (crabs, package = "MASS")
summary (crabs)
```

```{r, fig.height = 6}
plotdata (crabs [, 4:8], crabs [, 1], type = "pairs")
```

```{r, fig.height = 6}
plotdata (crabs [, 4:8], crabs [, 2], type = "pairs")
```

The five measurements are almost perfectly correlated with each other -- a crab is simply
bigger or smaller -- so most of what any one of them says, the others say too. That is what
makes the *selection* the interesting question here rather than the classifier.

```{r}
round (cor (crabs [, 4:8]), 3)
```

# Question 1. Selecting for the species

Using a ranking algorithm and Fisher's index as the univariate criterion, which multivariate
criterion -- the F statistic, mRMR, or a wrapper -- gives the best predictions of the species
with a naive Bayes classifier?

```{r}
# The two filters are deterministic; the wrapper is not -- it judges each subset by fitting a
# naive Bayes classifier under a bootstrap, so without 'seed' it can stop at a different
# subset from one run to the next. That is the criterion's own variance, not the data's.
s.fstat1 = selectfeatures (crabs [, 4:8], crabs [, 1], algorithm = "ranking",
                           unieval = "fisher", multieval = "fstat", seed = 0)
s.mrmr1 = selectfeatures (crabs [, 4:8], crabs [, 1], algorithm = "ranking",
                          unieval = "fisher", multieval = "mrmr", seed = 0)
s.wrap1 = selectfeatures (crabs [, 4:8], crabs [, 1], algorithm = "ranking",
                          unieval = "fisher", multieval = "wrapper", wrapmethod = NB,
                          seed = 0)
s.fstat1
s.mrmr1
s.wrap1
```

```{r}
performance (NB, crabs [, 4:8], crabs [, 1], nruns = 100, seed = 0)
performance (NB, crabs [, 4:8] [, s.fstat1$selection], crabs [, 1], nruns = 100, seed = 0)
performance (NB, crabs [, 4:8] [, s.mrmr1$selection], crabs [, 1], nruns = 100, seed = 0)
performance (NB, crabs [, 4:8] [, s.wrap1$selection], crabs [, 1], nruns = 100, seed = 0)
```

**Answer.** *All three criteria stop at the same subset and therefore give the same
performance -- which is better than using all five variables. Adding correlated measurements
to a naive Bayes classifier, whose whole assumption is that they are independent, costs
accuracy.*

# Question 2. Selecting for the sex

The same question, for the sex of the crabs.

```{r}
s.fstat2 = selectfeatures (crabs [, 4:8], crabs [, 2], algorithm = "ranking",
                           unieval = "fisher", multieval = "fstat", seed = 0)
s.mrmr2 = selectfeatures (crabs [, 4:8], crabs [, 2], algorithm = "ranking",
                          unieval = "fisher", multieval = "mrmr", seed = 0)
s.wrap2 = selectfeatures (crabs [, 4:8], crabs [, 2], algorithm = "ranking",
                          unieval = "fisher", multieval = "wrapper", wrapmethod = NB,
                          seed = 0)
s.fstat2
s.mrmr2
s.wrap2
```

```{r}
performance (NB, crabs [, 4:8], crabs [, 2], nruns = 100, seed = 0)
performance (NB, crabs [, 4:8] [, s.fstat2$selection], crabs [, 2], nruns = 100, seed = 0)
performance (NB, crabs [, 4:8] [, s.mrmr2$selection], crabs [, 2], nruns = 100, seed = 0)
performance (NB, crabs [, 4:8] [, s.wrap2$selection], crabs [, 2], nruns = 100, seed = 0)
```

**Answer.** *Only the wrapper improves on the full set of variables. The two filter criteria
stop at a single variable and lose accuracy -- which is the trade-off of the family: a filter
judges a subset without ever fitting the model it is selecting for, and here that judgement is
wrong.*

# Question 3. Which variables, in the end?

```{r}
colnames (crabs) [4:8] [s.fstat1$selection]
colnames (crabs) [4:8] [s.wrap2$selection]
```

**Answer.** *The frontal lobe alone is what distinguishes the two species; distinguishing the
sexes takes three measurements -- the rear width, the length of the carapace and the depth of
the body. Two different questions asked of the same five measurements, and two different
answers.*