# Corpus, oracle and benchmark generator

Three tools measure the R runtime against GNU R, and all three need only a GNU R installation
(the repository was set up with 4.3.3, the version Ubuntu 24.04 ships). Each has a `just` recipe.

## The package corpus

```sh
just corpus
```

installs the benchmark dependency closure from `AGENTS.md` into `.corpus/library`, together with
the `Suggests` of each package in the list, which their own tests need. It prefers Linux binaries
from a dated [Posit Package Manager](https://packagemanager.posit.co) snapshot, takes `azidentity`,
`Rblpapi` and any other package that CRAN lacks from the finccam repository, and clones
`esgtoolkit` (tag `v1.0.0`) and `loadfast` from GitHub.

| Variable | Default | Meaning |
| --- | --- | --- |
| `RY_CORPUS_DIR` | `.corpus` | Where the library, logs and status file go. |
| `RY_CORPUS_SNAPSHOT` | `2025-03-01` | The Package Manager snapshot. An old date keeps versions compatible with R 4.3. |
| `RY_CORPUS_NCPUS` | `4` | Parallel package installs. |

The run writes `.corpus/status.csv` (one row per package with its version, or the reason it is
missing) and `.corpus/logs/install.log`. Packages that need system libraries fail until you
install them, and the log names the missing header or library.

### Install status

A run on Ubuntu 24.04 with R 4.3.3 installed 65 of the 67 closure packages and every `Suggests`
(the recommended packages `MASS`, `lattice` and `rpart` come from the system R). Two do not
install:

- `polars` is not available for R 4.3.3. Its CRAN release needs a newer R (the runtime design
  targets 4.6), so it needs a newer GNU R than the container's, for example one installed with
  [rig](https://github.com/r-lib/rig).
- `Rblpapi` links against the proprietary Bloomberg `blpapi` SDK (`-lblpapi3_64`), which is not
  redistributable, so it cannot be installed anywhere without a Bloomberg licence.

`FLSSS` is missing from the pinned snapshot, so the script installs it from the newest one. The
packages that load system libraries (`git2r`, `odbc`, `pdftools`, `Rglpk` and the compile
dependencies) need the Ubuntu packages in `tools/corpus/system-packages.txt` first; without them
they install but fail to load.

GNU R does not install a dependency its own `.Library` already holds, but ry's runtime searches
only the corpus library and its own base packages. The script therefore copies every
non-priority package of the running R's `.Library` (the reference R ships about 50, testthat and
rlang among them) into the corpus library, so both runtimes see the same packages.

Older R headers declared functions that libR exports but R 4.6's leave undeclared by default,
such as `ATTRIB`, `ENCLOS` and `Rf_findVarInFrame`. Snapshot C code that calls one compiles with
only a warning under gcc 13, and the implicit `int` return truncates a returned pointer, so the
package crashes at that call under GNU R and ry alike. After installing, the script lists each
package whose shared object imports such a function (unless its prototype under
`ENABLE_LEGACY_NONAPI` returns an `int` or nothing) and reinstalls it from the newest CRAN source.
`logs/install.log` names any package that imports one after the rebuild: either it declares the
function itself (S7 does, and is correct), or its newest source failed to install.

A package newer than the snapshot, from the retry against the newest snapshot or from that
rebuild, can need a newer version of a dependency the snapshot already installed, and
`install.packages()` never upgrades an installed dependency. Rdpack 2.6.6, for example, imports
rbibutils (> 2.4) while the snapshot holds 2.3, so every Rdpack user failed to load. The script
therefore checks the version constraints in every installed package's `Depends`, `Imports` and
`LinkingTo`, reinstalls each dependency that fails one from the newest CRAN source, and repeats
until all hold (at most three rounds). `logs/install.log` lists each unmet constraint it found and
any that still fails.

## The differential oracle

```sh
just oracle verify          # GNU R against the recorded .expected files
just oracle compare         # GNU R against ry-tier0 and ry-jit
just oracle show file.R     # one file's outcome under GNU R
RY_BLESS=1 cargo test -p oracle   # or: just oracle record
```

The `oracle` crate runs an R file the way `Rscript` does and records five things: the
script's stdout (including auto-printed values), its messages, its warnings, the error that
stopped it, and the deparsed last value. A warning or error carries its call, as in
`from f (in f())`. The cases live in `crates/oracle/cases/`, one `group__name.R` file with a
`.expected` file beside it holding GNU R's recorded outcome. They cover promises,
environments, S3 and S4, conditions, vectors, recycling, attributes, NA handling, numerics,
control flow, printing, delimited data I/O (`read.table`, `write.table`, `scan`, `type.convert`),
colours, contour lines and axis ticks, graphics devices and parameters (`par`, layouts, the
device list, text metrics), grid and the graphics engine (the `grid__*` cases, which need ragg and lattice installed and compare drawings with GNU R by pixel checksums of a ragg capture), object sizes, hash tables, and data.table queries (the `datatable__*` cases, which need data.table installed and check that data.table's own code prints and behaves under ry exactly as under GNU R). `cargo test -p oracle` replays them under GNU R when it is installed, and under ry's runtime when it is built (see below). The expectations are GNU R 4.6.2 output, the release ry bundles as its base R (see [The reference GNU R](#the-reference-gnu-r)).

The driver differs from `Rscript` in four ways that the cases avoid: it parses the whole file first, so a syntax error hides the output of the lines before it; its own frames show in `sys.nframe()` and `sys.calls()` (so data.table prints the table after a top-level `:=`, which it skips at a real prompt); `commandArgs(TRUE)` holds the two paths; and `quit()` ends the run without a report.

An engine compared with GNU R is anything that implements the `Engine` trait. `ProcessEngine` covers a
command line: it is invoked as `<command…> <script> <report>`, must print the script's output
to stdout, and must write the JSON report that `crates/oracle/src/driver.R` defines (`messages`,
`warnings`, `errors`, and `value` with `kind` and `deparsed`). ry's runtime is `--engine ry`:
it runs the driver through `ry run --runtime ry` with the workspace's debug build (or the binary in
`ORACLE_RY`, which must be an absolute path because each case runs in its own directory), so
build `ry` and `libr` first. `--engine ry-jit` is the same runtime with `RY_JIT=force`, which runs
every body that is called twice and every loop through the JIT, so comparing it against GNU R checks
the compiled tier on every case. A JIT helper or tier-0 step that releases handles compiled code
holds fails an assertion that is on in release builds too, so the case reports as `CRASHED`.
`ry-tier0` runs with `RY_JIT=off`, and `ry-gctorture` runs with
`RY_GC_STRESS=1`, which collects before every allocation once the script starts, and
`ry-gctorture-young` with `RY_GC_STRESS=young`, which also keeps the survivors collectable.
`ry-jit-gctorture` combines `RY_JIT=force` with `RY_GC_STRESS=1`, so it checks that compiled code
keeps every object it holds rooted. `ry-gctorture-seeded` runs with `RY_GC_STRESS=seed:0` and
`RY_GC_VERIFY=1`, which collects at random gaps and checks the heap's invariants before each
collection ([Robustness](/development/robustness.md) lists the modes).

`ry-repeat` evaluates each script twice in one process, and `ry-repeat-gctorture-seeded` does so
under the seeded stress. The second pass defines every function again over the same code and calls
it with the values the first pass left, so it runs what only a later call reaches: a compiled
body, a call site's plan, a reused frame. Only the first pass's outcome is compared, because the
second starts from another state. The second pass must finish: a crash or a failed runtime check
in it fails the case. The driver does this when `ORACLE_PASSES` is above 1, printing a marker line
before each later pass, which the oracle cuts standard output at. A case that cannot run twice in
one process (it leaves standard output diverted, or consumes something the first pass used up)
says so in a leading `# oracle: runs once` comment, and these engines run it once.

A case that reads or changes the collection stress setting itself (`gctorture`, `gctorture2`)
prints something else under a stressed engine by design. It says so in a leading
`# oracle: sets collection stress` comment, and the stress passes of `test_cases` leave it out.

`--engine` takes a comma-separated list, such as
`--engine ry,ry-jit-gctorture`, which runs GNU R once per case and prints a summary line per engine.
A failure listed in `crates/oracle/known_failures.toml` reports as known rather than failing the run
([Testing](/development/testing.md#known-oracle-failures) has the rule). Without `--engine`, the engines
are the ones `ORACLE_ENGINE` lists, else `ry-tier0,ry-jit`. Without GNU R, `compare` checks the
engines against the recorded `.expected` files instead.

`cargo test -p oracle` runs the same check whenever ry finds a base R. It first builds the `ry`
binary and `libR.so` (unless `ORACLE_RY` names a binary), so it never runs a stale runtime. Then
every case under `ry-tier0` and `ry-jit`, or under the engines `ORACLE_ENGINE` lists, must give its
recorded outcome, sorted against the known failures. It needs no GNU R: the recordings stand in for
it, and the GNU R replay checks them wherever GNU R is installed. ry loads its own base R here, not
the GNU R installation `compare` would point it at.

The test asks each engine which base R release and which BLAS and LAPACK it loaded. Another release
than the recordings' skips the test, or fails it when `ORACLE_ENGINE` is set. Other libraries turn a
difference in a `# oracle: library-dependent` case into a warning. ry's bundled base R links R's own
LAPACK (`libRlapack.so`), not the system LAPACK the recordings name, so under it those cases only
ever warn.

Every engine searches the same libraries: any `R_LIBS` you set, then the corpus library, then the
reference R's own. The corpus library is `.corpus/library` in the workspace (`$RY_CORPUS_DIR/library`
when that is set), where `just corpus` installs the packages many cases need, and the oracle adds it
whenever the directory exists. An `R_LIBS` set to the empty string leaves it out. The oracle asks
GNU R for its `.libPaths()` once with those libraries in `R_LIBS` (and `R_LIBS_SITE` and
`R_LIBS_USER` empty), then runs each engine, and the gallery and `oracle bench` runs too, with
`R_LIBS` set to the answer and `R_LIBS_SITE` and `R_LIBS_USER` set empty. Without that, ry would search its own
defaults: its bundled base R's library, which holds none of the reference R's packages, and any GNU
R site or user library it finds, such as Debian's `/usr/lib/R/site-library`, whose data.table 1.14.10,
rlang 1.1.3 and withr 2.5.0 would load in place of the reference R's. `just doctor` warns when
`ry run` by hand loads other packages than the reference R and prints the variables that make it
match:

```sh
R_LIBS=/opt/R/4.6.2/lib/R/library R_LIBS_SITE= R_LIBS_USER= ry run case.R
```

The packages themselves must be installed by GNU R. ry runs on GNU R's base R when the two releases
differ, so a package that ry installed (its `DESCRIPTION` says it was built under ry's release)
makes `library()` warn "package 'x' was built under R version 4.6.2". The driver therefore reports
the `R_LIBS` each engine saw, and a case whose engines saw different values fails with that message
instead of a diff. A wrapper given as `ORACLE_RY` must not set `R_LIBS` itself.

ry's native fast paths for base closures (`crates/interp/src/base_fast_paths.rs`) install only on
the GNU R release whose code they follow, recognised by a hash. Against an older GNU R, such as a
container's 4.3.3, ry runs that release's base R and most fast paths stay out, so only CI, which
pins the matching release, checks them against GNU R. Locally, `RY_FAST_PATHS=0` turns off the
paths that skip R frames (`message`, `signalCondition`, and the warning shortcuts for no handler,
an exiting handler and `suppressWarnings`), and running
a case with and without it on ry's own base R compares them with the R bodies they stand for.

## The plot gallery

```sh
cargo run -p oracle -- gallery                          # every figure, every device
cargo run -p oracle -- gallery --seed 3 --count 20      # a seeded sample
cargo run -p oracle -- gallery --figures base:persp_surface,lattice:lattice_xyplot_groups --devices agg
```

The oracle cases check graphics state by value, and the gallery checks what is drawn. Each
`crates/oracle/gallery/*.R` file (base graphics, the PerformanceAnalytics and xts charts in
`finance.R`, grid, lattice, ggplot2) declares figures with `figure("name", { ... })`, and
`oracle gallery` renders each one under GNU R and under ry's runtime on the `agg` (ragg), `png`,
`pdf` and `svg` devices, then compares the images and times every render. A figure's visible value is printed, as at the top level, because lattice,
ggplot2 and xts draw only when printed. A figure whose packages GNU R lacks is skipped, and the
gallery searches the corpus library like `compare` does, so `just corpus` is enough to run all of them.

ragg rasterizes the same way under both engines, so `agg` is the sharp check: at most 0.2% of
pixels may differ. `png` and `pdf` (rasterized with `pdftoppm` to `<figure>.pdf.png`) allow 5%, and there a pixel
counts as differing only when nothing within one pixel in the other image matches it, because
GNU R's cairo and ry's renderer antialias differently. `svg` is timed but not compared, because
cairo writes text as glyph paths. The report lists each figure's differing share per device and both
engines' times; `--repeats` keeps the fastest of several renders. A figure name used twice in the gallery, or a device outside `agg`, `png`, `pdf` and `svg`, is an error. `--seed` picks the sample and
seeds R's random numbers in every figure, so a run replays exactly.

`cargo test -p oracle` runs a bounded sample (`tests/test_gallery.rs`, four figures on `agg`, or
`png` without ragg), picked by `FUZZ_SEED` and sized by `FUZZ_ITERS`. It skips without GNU R or a
built runtime, and a failure prints the `oracle gallery` command that replays it.

## Package tests

```sh
just corpus-sources                   # source tarballs, which carry the tests binary installs lack
just package-tests                    # every closure package, under GNU R and ry's runtime
just package-tests --runtime ry vctrs # one package, ry only, reusing the recorded GNU R result
```

`oracle packages` runs each corpus package's own tests the way `R CMD check` runs `tests/`: every
top-level `tests/*.R` file runs as its own script with `tests/` as the working directory, and the
file passes when its process exits with status 0. A package passes when all of its files pass, so
a testthat suite passes when `testthat.R` finishes without a failed expectation. The harness also
reads testthat's summary line (`[ FAIL 0 | WARN 1 | SKIP 2 | PASS 30 ]`) to report expectation
counts beside the status.

`tools/corpus/sources.R` downloads the source tarball of every package in the closure at its
installed version into `.corpus/sources` and writes `.corpus/closure.txt`, the package list the
harness runs. Results go to `.corpus/tests/<package>/<runtime>/` (the copied `tests/` directory,
one `.Rout` per file, and `result.json`), and `.corpus/tests/summary.md` tables every package.
A GNU R result is reused while the package version is unchanged, unless you pass
`--rerun-gnu-r`. `--jobs` (default 4) runs packages in parallel, and `--timeout` (default 600
seconds) bounds one package's suite under one runtime, after which its whole process group is
killed. `ORACLE_RY` picks the ry binary; copy it, with `libR.so` beside it, to another directory
first if you rebuild while a long run is going.

The pass rate counts the packages whose tests pass under GNU R, since a suite that fails there
measures the container, not ry. With the packages ry installed itself, ry passes 140 of the 164
that GNU R passes (85.4%), and with the packages GNU R installed it passes 115 of 146 (78.8%).
`MEMORY.md` records the current rate.

### Top CRAN packages

The corpus is the benchmark's dependency closure, so ry is tuned to it. A second set measures
packages the corpus has not mined: `tools/corpus/top-packages.txt` lists the 100 most-downloaded
CRAN packages outside the closure, leaving out the few that need a system library the container
lacks or take hours to build.

```sh
just top-packages-setup     # GNU R installs them and their Suggests into .corpus-top, with sources
just top-packages --jobs 3  # ry builds them all from source, then both runtimes run their tests
```

`just top-packages` reinstalls every package into `.corpus-top/ry-library` with ry's `R CMD
INSTALL` and then runs `oracle packages --installed` against `.corpus-top`, so the results land
in `.corpus-top/tests/summary-installed.md`. To rerun one package's tests on ry without
reinstalling, call the oracle directly:

```sh
ORACLE_RY=$PWD/target/release/ry ./target/release/oracle packages --corpus .corpus-top \
  --installed --runtime ry ggplot2
```

Of the 100 packages, 85 have tests and GNU R passes 84; ry passes 82 of those 84 (97.6%).
`backlog.md` lists the two that fail and why.

## Installing on ry

```sh
just ry-home                       # build release libR and ry-rcmd, assemble .corpus/ry-home
just corpus-ry-install             # install the whole closure from source on ry
just corpus-ry-install vctrs tibble   # only these, reusing what the library already holds
just oracle packages --installed   # run the tests against ry's own installations
```

`oracle install` installs each package from its tarball in `.corpus/sources` with the R home's
`bin/R CMD INSTALL`, so configure, compilation and the R-level steps (lazy-load database, `Meta/`,
help) all run on ry, into `.corpus/ry-library`. A package starts once everything in its `Depends`,
`Imports` and `LinkingTo` has installed, or is already in the library when it is not part of the
run, and is skipped when one of them failed. After installing, it is loaded with `library()` in a
fresh `bin/Rscript`. Logs go to `.corpus/ry-install/logs/<package>.{install,load}.log`, and
`.corpus/ry-install/summary.tsv` keeps one row per package: a run replaces the rows of the
packages it names and keeps the others.

The R home holds a copy of `libR.so`, so after rebuilding run `just ry-home` again before
installing. `oracle packages --installed` runs ry's side through `.corpus/ry-home/bin/Rscript`
with `.corpus/ry-library` ahead of `.corpus/library` (which still supplies the `Suggests`), and
records results under `ry-installed`, beside the `ry` results of `ry run --runtime ry`, with the
table in `.corpus/tests/summary-installed.md`.

GNU R's binaries of graphics devices refuse ry's graphics engine, which is one version newer than
the one R 4.3.3 built them against, so a package whose tests plot through svglite needs svglite
installed on ry: `just corpus-ry-install svglite` builds it from source.

## Running against another GNU R

The harness uses whichever GNU R is first on `PATH`. Two environment variables point it elsewhere,
which is how to compare against a newer R installed beside the system one, for example with
[rig](https://github.com/r-lib/rig) (`rig add 4.6` installs under `/opt/R/<version>`):

```sh
ORACLE_R=/opt/R/4.6.2/bin/R just oracle verify            # the single-file cases
ORACLE_RSCRIPT=/opt/R/4.6.2/bin/Rscript just oracle packages   # the package corpus
```

Installing rig puts its `R` ahead of the system one on `PATH`; remove the `R` and `Rscript` links
in `/usr/local/bin` to keep the system R as the default.

`ORACLE_R` covers `verify`, `record` and `compare`. `ORACLE_RSCRIPT` covers `packages` and `bench`.
When `ORACLE_R` is set, the `ry` engine of `compare` also runs with `R_HOME` set to that R's home,
so ry's runtime loads the same base library and emulates the same version as the GNU R it is
compared with. That makes `compare` the check for a new R release:

```sh
ORACLE_R=/opt/R/4.6.1/bin/R just oracle compare --engine ry
```

Without `ORACLE_R`, the `ry` engine runs on its default base R only when that is the release of
the GNU R on `PATH`. Otherwise, including when ry's default release cannot be read because its
start fails or times out, it takes that R's home too, so a version-gated behavior never shows up
as a difference.

ry follows the version of the base R it loads. Each behavior GNU R changed between 4.3 and 4.6 is
gated on that version in `crates/interp` (`system::r_at_least`), so ry with `R_HOME` set to GNU R
4.3.3 behaves as 4.3.3 does, and ry on its bundled base behaves as 4.6.2 does.

Without any GNU R, `compare` checks the engine against each case's recorded outcome instead and says
so on stderr. The engine then runs on ry's bundled base R 4.6.2, the release the recordings come
from. A case that needs a package the engine cannot load is listed as `skip ... needs <package>`,
and the run fails unless you pass `--allow-skips` ([Testing](/development/testing.md#known-oracle-failures)).

## The reference GNU R

The `.expected` files are recorded with GNU R 4.6.2 built from source with

```sh
./configure --enable-R-shlib --enable-memory-profiling --with-x --without-recommended-packages --disable-java
```

with the pango development headers installed (`libpango1.0-dev`), and with `data.table`, `bit64`,
`rlang`, `knitr`, `rmarkdown`, `ragg`, `lattice`, `R6`, `withr` and `xts` installed from CRAN into
its library. Some
cases depend on that build:

- `utils__rprofmem` and `environments__sys_call_shares_arguments` call `Rprofmem()`, so they need
  memory profiling. Without it GNU R stops with "memory profiling is not available on this system".
- The graphics cases need `png()` to draw through cairo, which R uses only when pango is present.
  Without it the bitmap type falls back to Xlib, which needs an X11 display, even though
  `capabilities("cairo")` reads `TRUE`. `tools/doctor.py` and `tools/reference-r.sh` therefore check `getOption("bitmapType")`.
- Collation and a few string cases follow ICU, which R uses when its headers are installed.
- The `datatable__*`, `grid__*` and knitr cases print what those packages' current CRAN releases
  print, and each declares the packages it needs in an `# oracle: requires` line.
- `strings__riconv_native_encoding` switches to `en_US.ISO-8859-1`, which a stock Ubuntu container
  lacks. It declares the locale in a `# oracle: requires locale` line, and
  `tools/oracle-locales.sh`, which dev-setup and CI run, generates it.

`sudo tools/dev-setup.sh` builds exactly this into `/opt/R/4.6.2` and installs the packages, and
`just doctor` checks it. `--enable-R-shlib` changes no output: it builds the `libR.so` that ry's
console loads for GNU R and that binary packages link.

Other builds differ in the last digits too: OpenBLAS changes `lm` and `arima` results, and the sign
of `NA^NaN` depends on the compiler. So does `NaN + NA_integer_` (and `*`) in a closure's first,
uncompiled call, where GNU R's AST interpreter gives NA or NaN depending on how the C compiler
ordered the operands, so a case about NaN and NA makes those calls with `NA_real_`. glibc's `mktime` starts its search from the offset its last call
found, so a repeated wall-clock time with unknown daylight saving depends on what ran before; the
cases that print one convert a summer time first.

`crates/oracle/cases/gnu-r.txt` names the build the suite was recorded with: its release, and its
BLAS and LAPACK libraries by file name. Every `verify`, `record` and `compare` report starts with the
build of the GNU R it ran, and a comparison against recorded outcomes names the recorded build.
`verify`, `record` and the `recorded_outcomes_match_gnu_r` test check the build file before they run
a case:

| GNU R at hand | `verify` | `record`, `RY_BLESS=1` | `recorded_outcomes_match_gnu_r` |
|---|---|---|---|
| same release and libraries | runs | runs | runs |
| other BLAS or LAPACK | warns, runs | warns, runs | warns, runs all but the library-dependent cases |
| other release | refuses (exit 2) | refuses (exit 2) | skips, naming both builds |
| no build file | warns, runs | writes the file | fails (bless writes it) |

A case whose expected digits are those of the numerical libraries declares it in a leading
`# oracle: library-dependent` line: matrix products, `lm`, `optim` and the other cases that print
BLAS or LAPACK results to the last bit, where another library also moves an optimizer's iteration
count and warnings. Only a GNU R on the recorded libraries can confirm those expectations, so
`recorded_outcomes_match_gnu_r` skips them on other BLAS or LAPACK libraries and names each one
it skips. The C math library and the compiler move results too, but the build file does not name
them, so the cases that follow them carry the marker as well and are compared only where BLAS and
LAPACK match too:

- glibc's x86-64-v3 build, which CI's Ubuntu runner uses, changes `pgamma`, `fft` and `nlm` results.
- A GNU R configured against glibc 2.41 or later calls the C library's `tanpi`, `sinpi` and
  `cospi` instead of its own, which moves the last bit of the Cauchy quantiles
  (`distributions__cauchy_quantiles`).
- A complex sum of an NA part and a NaN part is NA or NaN depending on how the compiler ordered
  the operands (`kernels__complex_na_nan_sums`).

Keep such a computation in a small case of its own, so the larger case it came from is still
compared everywhere.

On a machine whose `PATH` has another release, set `ORACLE_R` to a 4.6.2 build. To move the suite
to a new release, delete `gnu-r.txt` and record every case. A GNU R killed by a signal before it
writes its report skips the case with `skip ... this GNU R crashed`, because some hardened builds
abort with "buffer overflow detected" in graphics cases that the reference build runs; a crash of
ry's engines is a failure.

Record with `ORACLE_R=<that R> oracle record crates/oracle/cases` and review every changed
`.expected` file.

## The benchmark generator

```sh
just bench-gen --out .bench --lines 10000    # three packages of about 10,000 lines
just bench-gen --out .bench --lines 300000 --files 500
just bench-test .bench/finbench
```

`tools/bench/tooling/generate.py` writes `finbench`, `dsbench` and `biobench` under `--out`. Each is
an R package with a `DESCRIPTION` that imports exactly the benchmark dependency set, a
`NAMESPACE`, R files, and a testthat suite. The generator is deterministic for a given `--seed`,
and its output is not committed. `--domain` generates one package, and `--files` fixes the file
count (by default one file per 600 lines). The templates are written with inline annotations.
`--typed` keeps them and turns on strict mode in the generated `ry.toml`, so the package doubles as
a strict-mode dogfood corpus and ry's runtime checks and compiles its typed functions. Without it,
the generator erases the package with `ry debug erase` (so `target/release/ry` must exist, or pass
`--ry`), which gives plain R. Both runs count the same text toward the line target, so a typed and
an untyped run with the same seed generate the same program. Where a record or a tag union models the data better than S4, R6 or a named list, a template keeps both versions in `#@ typed` and `#@ untyped` blocks, so the typed program uses ry's own data types while the untyped one stays plain R (the finance `ledger`, data-science `readings` and biostatistics `followup` templates model their data as tags, tuples and records this way); the templates' `README.md` has the details. `--only` limits
generation to the named templates, which is the quick way to iterate on one of them.

The code comes from the R templates in `tools/bench/tooling/templates/` (see its `README.md`): shared
infrastructure plus domain code, instantiated with random parameters until the line target is
reached. Services that cannot run in a test (Azure, Bloomberg, ODBC) take an injectable client,
so the tests never touch a network. `just bench-test` runs a package's tests against the corpus library, so run `just corpus` first.
Because `polars` and `Rblpapi` cannot be installed, it tests a copy whose `Imports` omit the
missing packages; the code reaches them only inside guarded or injectable branches. The three
10,000-line packages pass all of their roughly 340 tests each under GNU R 4.3.3.

## Benchmark pass rate

```sh
just bench-compare .bench/finbench .bench/dsbench .bench/biobench
just bench-compare .bench/finbench --filter portfolio --verbose
```

`oracle bench` runs a generated package's testthat suite under GNU R and under ry's runtime and
compares the two test by test. Both runs go through `ry load <package> --file bench.R`, once with
`--runtime gnu-r` and once with `--runtime ry`, so the same loader plan loads the package on both
sides and only evaluation differs. The driver (`crates/oracle/src/bench.R`) records every
`test_that()` block: its status, the failed expectations, errors, warnings and skips it reported,
a deparse of its value, and its run time. A test matches when everything but the time agrees.

GNU R runs the suite twice. A test whose value or problems differ between those two runs (a
fresh UUID, a timestamp in a message) is compared by status alone, so only what GNU R reproduces
counts against ry. The table prints,
per package, the matching tests, the rate, the tests that differ or that ry never reported, the
tests ry passes, and both suites' run times; process wall time, which adds startup and loading
the package, follows it, and then the time each runtime spent collecting during the suite, as
`gc.time()` reports it, with its share of the suite time. For pause distributions, set
`RY_GC_LOG=<file>`: ry's runtime then appends one line per collection, giving its kind (`minor`,
`intermediate` or `major`), its elapsed nanoseconds, the bytes of small objects and of large
vectors allocated since the previous collection, the old generation's bytes after it, and the
process id. Every process of a run appends to the same file, so the process id tells a test
suite's worker processes apart. Collections are deterministic: two runs of the same program in one
process make the same sequence. A testthat suite that runs in parallel (`Config/testthat/parallel`)
hands files to its workers as they finish, so its collections vary from run to run; set
`TESTTHAT_PARALLEL=false` to compare them. `--json` writes every comparison, and `--keep` keeps each runtime's
report and stderr. Pass `--ry target/release/ry` for timings, with `libR.so` built beside it
(`cargo build --release -p ry-lang -p libr`). `MEMORY.md` records the current rates.

The two time ratios answer different questions. Suite time adds up the `test_that()` blocks, so
it measures evaluation. Process wall time also counts what happens between blocks, and at the full
benchmark size (`--lines 300000 --files 500`) that is dominated by testthat calling `gc()` after
every test file that registered a teardown. A full collection with the whole package loaded takes
about 600 ms on GNU R and about 100 ms on ry, so ry can trail on suite time and still beat GNU R
end to end.
