---
title: "From raw sample to final weights"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{From raw sample to final weights}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include = FALSE}
knitr::opts_chunk$set(collapse = TRUE, comment = "#>")
library(weightflow)
```

> **New here?** This is the 15-minute tour: what problem `weightflow` solves and
> the shortest recipe that produces analysis weights. For the statistical
> reasoning behind each adjustment, read
> [*Staged survey weighting: the adjustment logic*](weightflow.html).

## The problem is not the weights

Every survey organization has weighting scripts. They grow over time, absorb new
adjustments, and eventually become difficult to understand, maintain and
reproduce. Two analysts can implement the *same* weighting strategy in different
ways, which turns audits, updates and methodological reviews into archaeology.

The weights themselves are rarely the hard part. The hard part is that the
*strategy* (the sequence of methodological decisions that produced the weights)
is scattered across files, intermediate objects and the memory of whoever ran
them last. When the responsible methodologist moves on, the knowledge often
leaves with them.

A typical pipeline looks like this:

<svg width="100%" viewBox="0 0 680 470" role="img" xmlns="http://www.w3.org/2000/svg" style="font-family:-apple-system,'Inter',Segoe UI,Roboto,sans-serif;">
<title>Weighting spread across scripts and intermediate files</title>
<desc>Sampling feeds script1.R, which writes weights1.csv, read by script2.R, which writes weights2.csv, read by script_final_v7.R, which writes final_weights_NEW.csv. The steps zigzag between code files and intermediate CSVs.</desc>
<defs><marker id="qsred" viewBox="0 0 10 10" refX="8" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M2 1L8 5L2 9" fill="none" stroke="#9a8f8f" stroke-width="1.7"/></marker></defs>
<line x1="320" y1="62" x2="188" y2="90" stroke="#9a8f8f" stroke-width="1.5" marker-end="url(#qsred)"/>
<line x1="255" y1="120" x2="423" y2="168" stroke="#9a8f8f" stroke-width="1.5" marker-end="url(#qsred)"/>
<line x1="425" y1="182" x2="257" y2="220" stroke="#9a8f8f" stroke-width="1.5" marker-end="url(#qsred)"/>
<line x1="255" y1="238" x2="423" y2="284" stroke="#9a8f8f" stroke-width="1.5" marker-end="url(#qsred)"/>
<line x1="425" y1="298" x2="262" y2="336" stroke="#9a8f8f" stroke-width="1.5" marker-end="url(#qsred)"/>
<line x1="265" y1="356" x2="413" y2="390" stroke="#9a8f8f" stroke-width="1.5" marker-end="url(#qsred)"/>
<rect x="290" y="28" width="100" height="34" rx="17" fill="#ece9f6" stroke="#7a6ad0" stroke-width="1.3"/>
<text x="340" y="50" text-anchor="middle" font-size="13" fill="#20242e">sampling</text>
<rect x="95" y="90" width="160" height="40" rx="8" fill="#e8e7ef" stroke="#6b6a86" stroke-width="1.3"/>
<text x="175" y="115" text-anchor="middle" font-size="13" font-family="'JetBrains Mono',ui-monospace,monospace" fill="#33323f">script1.R</text>
<rect x="425" y="150" width="160" height="40" rx="8" fill="#f4e2b8" stroke="#c9902a" stroke-width="1.3"/>
<text x="505" y="175" text-anchor="middle" font-size="13" font-family="'JetBrains Mono',ui-monospace,monospace" fill="#5c4410">weights1.csv</text>
<rect x="95" y="208" width="160" height="40" rx="8" fill="#e8e7ef" stroke="#6b6a86" stroke-width="1.3"/>
<text x="175" y="233" text-anchor="middle" font-size="13" font-family="'JetBrains Mono',ui-monospace,monospace" fill="#33323f">script2.R</text>
<rect x="425" y="266" width="160" height="40" rx="8" fill="#f4e2b8" stroke="#c9902a" stroke-width="1.3"/>
<text x="505" y="291" text-anchor="middle" font-size="13" font-family="'JetBrains Mono',ui-monospace,monospace" fill="#5c4410">weights2.csv</text>
<rect x="90" y="324" width="175" height="40" rx="8" fill="#e8e7ef" stroke="#6b6a86" stroke-width="1.3"/>
<text x="177" y="349" text-anchor="middle" font-size="12" font-family="'JetBrains Mono',ui-monospace,monospace" fill="#33323f">script_final_v7.R</text>
<rect x="413" y="382" width="190" height="40" rx="8" fill="#f2cfc7" stroke="#c0492f" stroke-width="1.3"/>
<text x="508" y="407" text-anchor="middle" font-size="12" font-family="'JetBrains Mono',ui-monospace,monospace" fill="#7a2618">final_weights_NEW.csv</text>
<text x="340" y="452" text-anchor="middle" font-size="12" font-style="italic" fill="#8a7f7f">Logic scattered across files and run order, not kept in one place.</text>
</svg>

Anyone who has worked in a statistical office recognizes this immediately.
Along the way you typically reach for many different packages, spread the logic
across several scripts, and generate hundreds of lines of code. The methodology
is real, but it is *implicit*: hidden in file order, column names, intermediate
files and undocumented tweaks.

## A map of what the steps are for

Before we look at any code, one picture. Weights exist because the realized
sample differs from the population we want to describe, in specific, nameable
ways. Each region below is corrected by a specific step.

<svg width="100%" viewBox="0 0 680 560" role="img" xmlns="http://www.w3.org/2000/svg" style="font-family:-apple-system,'Inter',Segoe UI,Roboto,sans-serif;">
<title>From target population to respondents</title>
<desc>Two overlapping ellipses show the target population and the sampling frame, with undercoverage, an in-scope overlap, and out-of-scope units. The in-scope area is sampled; the sample splits into unknown eligibility, ineligible and eligible, and eligible splits into nonrespondents and respondents. A legend maps each region to a weightflow step.</desc>
<defs><marker id="qsarrow" viewBox="0 0 10 10" refX="8" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M2 1L8 5L2 9" fill="none" stroke="#7a7a86" stroke-width="1.6"/></marker></defs>
<ellipse cx="255" cy="112" rx="152" ry="72" fill="#7a6ad0" fill-opacity="0.16" stroke="#3d3580" stroke-width="1.5"/>
<ellipse cx="425" cy="112" rx="152" ry="72" fill="#8a8f9a" fill-opacity="0.16" stroke="#5b6472" stroke-width="1.5"/>
<text x="180" y="92" text-anchor="middle" font-size="14" font-weight="600" fill="#20242e">Target population</text>
<text x="500" y="92" text-anchor="middle" font-size="14" font-weight="600" fill="#20242e">Frame</text>
<text x="178" y="120" text-anchor="middle" font-size="12" fill="#3a3f4a">Undercoverage</text>
<text x="178" y="136" text-anchor="middle" font-size="12" fill="#3a3f4a">(missing from frame)</text>
<text x="340" y="116" text-anchor="middle" font-size="14" fill="#20242e">In scope</text>
<text x="500" y="120" text-anchor="middle" font-size="12" fill="#3a3f4a">Out of scope</text>
<text x="500" y="136" text-anchor="middle" font-size="12" fill="#3a3f4a">(ineligible on frame)</text>
<line x1="340" y1="184" x2="340" y2="230" stroke="#7a7a86" stroke-width="1.5" marker-end="url(#qsarrow)"/>
<text x="350" y="212" text-anchor="start" font-size="12" fill="#3a3f4a">draw sample (design weight 1/&#960;)</text>
<rect x="55" y="232" width="570" height="170" rx="10" fill="#f6f5fb" stroke="#3d3580" stroke-width="1.4"/>
<text x="70" y="256" text-anchor="start" font-size="14" font-weight="600" fill="#20242e">Sample (known inclusion probability &#960;)</text>
<rect x="75" y="284" width="135" height="98" rx="8" fill="#f4e2b8" stroke="#c9902a" stroke-width="1.3"/>
<text x="142" y="328" text-anchor="middle" font-size="14" fill="#20242e">Unknown</text>
<text x="142" y="346" text-anchor="middle" font-size="14" fill="#20242e">eligibility</text>
<rect x="225" y="284" width="120" height="98" rx="8" fill="#f2cfc7" stroke="#c0492f" stroke-width="1.3"/>
<text x="285" y="338" text-anchor="middle" font-size="14" fill="#20242e">Ineligible</text>
<rect x="360" y="270" width="252" height="115" rx="8" fill="#e7e3f5" stroke="#7a6ad0" stroke-width="1.3"/>
<text x="374" y="289" text-anchor="start" font-size="14" font-weight="600" fill="#20242e">Eligible</text>
<rect x="374" y="300" width="108" height="72" rx="7" fill="#e3e5e9" stroke="#8a8f9a" stroke-width="1.3"/>
<text x="428" y="332" text-anchor="middle" font-size="13" fill="#20242e">Non-</text>
<text x="428" y="350" text-anchor="middle" font-size="13" fill="#20242e">respondents</text>
<rect x="498" y="300" width="100" height="72" rx="7" fill="#cdeadd" stroke="#1d9e75" stroke-width="1.3"/>
<text x="548" y="341" text-anchor="middle" font-size="13" fill="#20242e">Respondents</text>
<circle cx="74" cy="432" r="6" fill="#c9902a"/>
<text x="90" y="436" text-anchor="start" font-size="12" fill="#3a3f4a">Unknown eligibility &#8594; step_unknown_eligibility(): redistribute their weight</text>
<circle cx="74" cy="456" r="6" fill="#c0492f"/>
<text x="90" y="460" text-anchor="start" font-size="12" fill="#3a3f4a">Ineligible &#8594; step_drop_ineligible(): remove from the sample</text>
<circle cx="74" cy="480" r="6" fill="#8a8f9a"/>
<text x="90" y="484" text-anchor="start" font-size="12" fill="#3a3f4a">Nonrespondents &#8594; step_nonresponse(): respondents absorb their weight</text>
<circle cx="74" cy="504" r="6" fill="#1d9e75"/>
<text x="90" y="508" text-anchor="start" font-size="12" fill="#3a3f4a">Respondents carry the final analysis weight</text>
<circle cx="74" cy="528" r="6" fill="#7a6ad0"/>
<text x="90" y="532" text-anchor="start" font-size="12" fill="#3a3f4a">Coverage &amp; known totals &#8594; step_calibrate(): align to the population</text>
</svg>

The frame and the population *overlap* rather than nest: some of the target
population is missing from the frame (undercoverage) and some frame units are
out of scope (overcoverage). From the sample downward it is pure nesting
(sample ⊃ eligible ⊃ respondents), and each split is a step. The weights are not
conjured; they are the consequence of this sequence of decisions.

## The same strategy as one object

`weightflow` expresses the whole strategy as a single, explicit recipe. Nothing
is hidden in a script; nothing lives in an intermediate CSV. Every
methodological decision is a `step_*()` you can read top to bottom:

<svg width="100%" viewBox="0 0 680 448" role="img" xmlns="http://www.w3.org/2000/svg" style="font-family:-apple-system,'Inter',Segoe UI,Roboto,sans-serif;">
<title>The whole strategy as one recipe object</title>
<desc>A single container holds weighting_spec() and a stack of step functions: unknown eligibility, drop ineligible, select within, nonresponse, calibrate and trim. An arrow leaves the container to prep(), which produces the final weights.</desc>
<defs><marker id="qspur" viewBox="0 0 10 10" refX="8" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M2 1L8 5L2 9" fill="none" stroke="#7a6ad0" stroke-width="1.7"/></marker></defs>
<text x="60" y="44" font-size="12" font-style="italic" fill="#6a6f7a">One object: the whole strategy, defined lazily, read top to bottom.</text>
<rect x="55" y="55" width="570" height="300" rx="12" fill="#f6f5fb" stroke="#3d3580" stroke-width="1.6"/>
<line x1="72" y1="98" x2="72" y2="318" stroke="#cbb8e0" stroke-width="1.6"/>
<text x="84" y="90" font-size="13" font-weight="600" font-family="'JetBrains Mono',ui-monospace,monospace" fill="#3d3580">weighting_spec(data, base_weights)</text>
<text x="372" y="90" font-size="12" font-family="'JetBrains Mono',ui-monospace,monospace" fill="#8a8f9a"># design weights 1/&#960;</text>
<rect x="84" y="110" width="212" height="26" rx="13" fill="#ece9f6" stroke="#7a6ad0" stroke-width="1.2"/>
<text x="95" y="128" font-size="12.5" font-family="'JetBrains Mono',ui-monospace,monospace" fill="#3d3580">step_unknown_eligibility()</text>
<text x="372" y="128" font-size="12" font-family="'JetBrains Mono',ui-monospace,monospace" fill="#8a8f9a"># resolve unknown eligibility</text>
<rect x="84" y="148" width="182" height="26" rx="13" fill="#ece9f6" stroke="#7a6ad0" stroke-width="1.2"/>
<text x="95" y="166" font-size="12.5" font-family="'JetBrains Mono',ui-monospace,monospace" fill="#3d3580">step_drop_ineligible()</text>
<text x="372" y="166" font-size="12" font-family="'JetBrains Mono',ui-monospace,monospace" fill="#8a8f9a"># remove out-of-scope units</text>
<rect x="84" y="186" width="168" height="26" rx="13" fill="#ece9f6" stroke="#7a6ad0" stroke-width="1.2"/>
<text x="95" y="204" font-size="12.5" font-family="'JetBrains Mono',ui-monospace,monospace" fill="#3d3580">step_select_within()</text>
<text x="372" y="204" font-size="12" font-family="'JetBrains Mono',ui-monospace,monospace" fill="#8a8f9a"># within-household selection</text>
<rect x="84" y="224" width="156" height="26" rx="13" fill="#ece9f6" stroke="#7a6ad0" stroke-width="1.2"/>
<text x="95" y="242" font-size="12.5" font-family="'JetBrains Mono',ui-monospace,monospace" fill="#3d3580">step_nonresponse()</text>
<text x="372" y="242" font-size="12" font-family="'JetBrains Mono',ui-monospace,monospace" fill="#8a8f9a"># correct for nonresponse</text>
<rect x="84" y="262" width="140" height="26" rx="13" fill="#ece9f6" stroke="#7a6ad0" stroke-width="1.2"/>
<text x="95" y="280" font-size="12.5" font-family="'JetBrains Mono',ui-monospace,monospace" fill="#3d3580">step_calibrate()</text>
<text x="372" y="280" font-size="12" font-family="'JetBrains Mono',ui-monospace,monospace" fill="#8a8f9a"># align to population totals</text>
<rect x="84" y="300" width="104" height="26" rx="13" fill="#ece9f6" stroke="#7a6ad0" stroke-width="1.2"/>
<text x="95" y="318" font-size="12.5" font-family="'JetBrains Mono',ui-monospace,monospace" fill="#3d3580">step_trim()</text>
<text x="372" y="318" font-size="12" font-family="'JetBrains Mono',ui-monospace,monospace" fill="#8a8f9a"># tame extreme weights</text>
<line x1="150" y1="355" x2="150" y2="382" stroke="#7a6ad0" stroke-width="1.5" marker-end="url(#qspur)"/>
<rect x="103" y="382" width="95" height="32" rx="16" fill="#ece9f6" stroke="#7a6ad0" stroke-width="1.4"/>
<text x="150" y="403" text-anchor="middle" font-size="13" font-family="'JetBrains Mono',ui-monospace,monospace" fill="#3d3580">prep()</text>
<line x1="198" y1="398" x2="243" y2="398" stroke="#7a6ad0" stroke-width="1.5" marker-end="url(#qspur)"/>
<rect x="245" y="382" width="170" height="32" rx="16" fill="#cdeadd" stroke="#1d9e75" stroke-width="1.4"/>
<text x="330" y="403" text-anchor="middle" font-size="13" fill="#14603f">final weights</text>
<text x="440" y="403" font-size="12" font-style="italic" fill="#6a6f7a">one output of the object</text>
</svg>

> **`weightflow` doesn't just compute weights: it documents how they are
> constructed.**
>
> It does compute the weights, of course. But a weighting strategy is a
> sequence of methodological decisions, and `weightflow` turns those decisions
> into an explicit object that can be inspected, reproduced, modified and
> reused. The final weights are only *one* output of that object, and usually
> the least valuable one. The recipe is the institutional asset.

If you know the `recipes`/`tidymodels` world, the analogy is deliberate: a
recipe describes how a dish is produced, not the dish itself. A `weightflow`
recipe describes how the weights are produced, not the weights themselves.

## The shortest real recipe

Enough theory. `weightflow` ships a small bundled example (a stratified
household sample, `sample_survey`, drawn from a known `population`), so every
line below actually runs.

```{r glimpse}
str(sample_survey[, c("pw", "region", "sex", "unknown_elig", "responded")])
```

A minimal but complete cascade: start from the design weights, resolve unknown
eligibility, correct for nonresponse, and calibrate to the population totals of
`region` and `sex`. We pass those totals in the **tidy** form: a small data
frame per margin with the categories and a counts column. `table()` already
produces exactly that shape (its counts column is named `Freq`):

```{r totals}
m_region <- as.data.frame(table(region = population$region))
m_sex    <- as.data.frame(table(sex    = population$sex))
m_region
```

```{r recipe}
fit <- weighting_spec(sample_survey, base_weights = pw) |>
  step_unknown_eligibility(unknown = unknown_elig, by = "region") |>
  step_nonresponse(respondent = responded, method = "weighting_class",
                   by = "region") |>
  step_calibrate(method = "raking",
                 totals = list(m_region, m_sex), count = "Freq") |>
  prep()
```

`prep()` is where the recipe is actually estimated. Everything before it only
*records* the strategy. `collect_weights()` returns the data with the final
weight attached as `.weight`:

```{r weights}
w <- collect_weights(fit)
summary(w$.weight)
```

## The object is the documentation

Because the strategy is an object, it explains itself. `summary()` walks the
cascade step by step, reporting how many units each stage touched, how weights
changed, and diagnostics such as the design effect:

```{r summary}
summary(fit)
```

That printout *is* the methodological record. You did not write a separate
memo describing what the scripts did; the recipe and its summary are the memo.
Re-running it next year, or handing it to a colleague, reproduces the exact
same weights and the exact same explanation.

## Where to go next

You now have runnable weights and a mental map. To go deeper:

- [*Staged survey weighting: the adjustment logic*](weightflow.html) covers the
  statistical reasoning behind each stage and why order matters.
- [*Preparing the sample*](preparing-the-sample.html) covers eligibility,
  rosters and within-household selection in detail.
- [*Nonresponse: weighting classes and propensities*](nonresponse-propensities.html)
  and [*Calibration*](calibration.html) cover the two workhorse adjustments.
- [*Variance estimation*](variance-estimation.html) covers the bootstrap and
  jackknife that re-apply the whole recipe, so uncertainty from nonresponse and
  calibration reaches your standard errors.
