Learn / Inside the pipeline / 03 · ingest

Counts are the truth; percentages are a courtesy

Exports usually carry both a %germination column and the raw counts. They disagree more often than you'd hope — typos, stale formulas, a percent computed against the wrong denominator. The rule: wherever valid counts exist, recompute the percentage from NumberGermin ÷ NumberSown and use that. The supplied percent is only a fallback for rows with no counts, with one rescue: values over 100 that are really proportions ×100² get divided back down.

Then the hard stop. Any row where germinated exceeds sown is not a warning — it's a data error that would silently corrupt every downstream number. Write those rows to an issues file and halt. A pipeline that keeps going past impossible data produces confident garbage.

Clamp the final percentage to [0, 100] and assert your invariants (max ≤ 100, no germ > sown) even after fixing, so a future edit that breaks the cleaning step fails loudly.

Build it yourself

library(dplyr)

df <- df %>%
  mutate(
    NumberSown   = as.numeric(NumberSown),
    NumberGermin = as.numeric(NumberGermin),
    pct_counts   = ifelse(NumberSown > 0 & NumberGermin >= 0 &
                          NumberGermin <= NumberSown,
                          100 * NumberGermin / NumberSown, NA_real_),
    PctGermin    = coalesce(pct_counts,
                            ifelse(PctGermin > 100, PctGermin / 100,
                                   as.numeric(PctGermin))),
    PctGermin    = pmin(pmax(PctGermin, 0), 100)
  ) %>% select(-pct_counts)

bad <- filter(df, NumberGermin > NumberSown)
if (nrow(bad) > 0) {
  readr::write_csv(bad, "_DATA_issues_germin_gt_sown.csv")
  stop(sprintf("%d row(s) with germinated > sown — fix the data first.", nrow(bad)))
}
stopifnot(max(df$PctGermin, na.rm = TRUE) <= 100)

The math behind this step: A germination test is binomial evidence

← Turn two dates into one clockMany tests, one year: max or pool →