Learn / Inside the pipeline / 02 · ingest

Turn two dates into one clock

The time axis of the whole analysis is YearsDifference: the accession's age, in whole years, at the moment each germination test started. Most databases don't store it — they store a collection date on the accession and a test-start date on each test. So the pipeline offers a preprocessing step: if YearsDifference is missing, compute it as (test date − collection date) ÷ 365.25, rounded half-up (0.5 → 1, 0.4 → 0).

The ugly part is date parsing. Real exports mix ISO dates, US slash dates, day-month-name strings, and — the classic — raw Excel serial numbers (days since 1899-12-30). A robust parser tries formats in order and only gives up when nothing matches. If YearsDifference is partially present, fill only the gaps; never overwrite values the bank computed themselves without being told to.

Whole-year rounding is a modeling decision, not an accident: monitoring tests cluster around anniversaries, and integer years make "latest year meeting the threshold" unambiguous. If your bank tests on a finer schedule, you can keep fractional years — but then define what "the same year" means before you collapse replicates.

Build it yourself
parse_seedbank_date <- function(x) {
  if (inherits(x, "Date")) return(x)
  if (is.numeric(x)) return(as.Date(x, origin = "1899-12-30"))  # Excel serials
  out <- suppressWarnings(as.Date(as.character(x)))             # ISO first
  fmts <- c("%m/%d/%Y", "%m/%d/%y", "%Y/%m/%d", "%m-%d-%Y", "%d-%b-%Y")
  for (fmt in fmts) {
    bad <- is.na(out)
    if (!any(bad)) break
    out[bad] <- suppressWarnings(as.Date(as.character(x)[bad], format = fmt))
  }
  out
}

if (!"YearsDifference" %in% names(df)) df$YearsDifference <- NA_real_
needs <- is.na(as.numeric(df$YearsDifference))
if (any(needs)) {
  yrs <- as.numeric(parse_seedbank_date(df$TestStartDate) -
                    parse_seedbank_date(df$CollectionDate)) / 365.25
  df$YearsDifference[needs] <- floor(yrs[needs] + 0.5)  # round half up
}
The math behind this step: The clock: an accession's age at test
Read messy exports, map columns by roleCounts are the truth; percentages are a courtesy