Skip to content
ry

Corpus, oracle and benchmark generator

Three tools measure the R runtime against GNU R, and all three need only a GNU R installation (the repository was set up with 4.3.3, the version Ubuntu 24.04 ships). Each has a just recipe.

Terminal window
just corpus

installs the benchmark dependency closure from AGENTS.md into .corpus/library, together with the Suggests of each package in the list, which their own tests need. It prefers Linux binaries from a dated Posit Package Manager snapshot, takes azidentity, Rblpapi and any other package that CRAN lacks from the finccam repository, and clones esgtoolkit (tag v1.0.0) and loadfast from GitHub.

Variable Default Meaning
RY_CORPUS_DIR .corpus Where the library, logs and status file go.
RY_CORPUS_SNAPSHOT 2025-03-01 The Package Manager snapshot. An old date keeps versions compatible with R 4.3.
RY_CORPUS_NCPUS 4 Parallel package installs.

The run writes .corpus/status.csv (one row per package with its version, or the reason it is missing) and .corpus/logs/install.log. Packages that need system libraries fail until you install them, and the log names the missing header or library.

A run on Ubuntu 24.04 with R 4.3.3 installed 65 of the 67 closure packages and every Suggests (the recommended packages MASS, lattice and rpart come from the system R). Two do not install:

  • polars is not available for R 4.3.3. Its CRAN release needs a newer R (the runtime design targets 4.6), so it needs a newer GNU R than the container’s, for example one installed with rig.
  • Rblpapi links against the proprietary Bloomberg blpapi SDK (-lblpapi3_64), which is not redistributable, so it cannot be installed anywhere without a Bloomberg licence.

FLSSS is missing from the pinned snapshot, so the script installs it from the newest one. The packages that load system libraries (git2r, odbc, pdftools, Rglpk and the compile dependencies) need the Ubuntu packages in tools/corpus/system-packages.txt first; without them they install but fail to load.

GNU R does not install a dependency its own .Library already holds, but ry’s runtime searches only the corpus library and its own base packages. The script therefore copies every non-priority package of the running R’s .Library (the reference R ships about 50, testthat and rlang among them) into the corpus library, so both runtimes see the same packages.

Older R headers declared functions that libR exports but R 4.6’s leave undeclared by default, such as ATTRIB, ENCLOS and Rf_findVarInFrame. Snapshot C code that calls one compiles with only a warning under gcc 13, and the implicit int return truncates a returned pointer, so the package crashes at that call under GNU R and ry alike. After installing, the script lists each package whose shared object imports such a function (unless its prototype under ENABLE_LEGACY_NONAPI returns an int or nothing) and reinstalls it from the newest CRAN source. logs/install.log names any package that imports one after the rebuild: either it declares the function itself (S7 does, and is correct), or its newest source failed to install.

A package newer than the snapshot, from the retry against the newest snapshot or from that rebuild, can need a newer version of a dependency the snapshot already installed, and install.packages() never upgrades an installed dependency. Rdpack 2.6.6, for example, imports rbibutils (> 2.4) while the snapshot holds 2.3, so every Rdpack user failed to load. The script therefore checks the version constraints in every installed package’s Depends, Imports and LinkingTo, reinstalls each dependency that fails one from the newest CRAN source, and repeats until all hold (at most three rounds). logs/install.log lists each unmet constraint it found and any that still fails.

Terminal window
just oracle verify # GNU R against the recorded .expected files
just oracle compare # GNU R against ry-tier0 and ry-jit
just oracle show file.R # one file's outcome under GNU R
RY_BLESS=1 cargo test -p oracle # or: just oracle record

The oracle crate runs an R file the way Rscript does and records five things: the script’s stdout (including auto-printed values), its messages, its warnings, the error that stopped it, and the deparsed last value. A warning or error carries its call, as in from f (in f()). The cases live in crates/oracle/cases/, one group__name.R file with a .expected file beside it holding GNU R’s recorded outcome. They cover promises, environments, S3 and S4, conditions, vectors, recycling, attributes, NA handling, numerics, control flow, printing, delimited data I/O (read.table, write.table, scan, type.convert), colours, contour lines and axis ticks, graphics devices and parameters (par, layouts, the device list, text metrics), grid and the graphics engine (the grid__* cases, which need ragg and lattice installed and compare drawings with GNU R by pixel checksums of a ragg capture), object sizes, hash tables, and data.table queries (the datatable__* cases, which need data.table installed and check that data.table’s own code prints and behaves under ry exactly as under GNU R). cargo test -p oracle replays them under GNU R when it is installed, and under ry’s runtime when it is built (see below). The expectations are GNU R 4.6.2 output, the release ry bundles as its base R (see The reference GNU R).

The driver differs from Rscript in four ways that the cases avoid: it parses the whole file first, so a syntax error hides the output of the lines before it; its own frames show in sys.nframe() and sys.calls() (so data.table prints the table after a top-level :=, which it skips at a real prompt); commandArgs(TRUE) holds the two paths; and quit() ends the run without a report.

An engine compared with GNU R is anything that implements the Engine trait. ProcessEngine covers a command line: it is invoked as <command…> <script> <report>, must print the script’s output to stdout, and must write the JSON report that crates/oracle/src/driver.R defines (messages, warnings, errors, and value with kind and deparsed). ry’s runtime is --engine ry: it runs the driver through ry run --runtime ry with the workspace’s debug build (or the binary in ORACLE_RY, which must be an absolute path because each case runs in its own directory), so build ry and libr first. --engine ry-jit is the same runtime with RY_JIT=force, which runs every body that is called twice and every loop through the JIT, so comparing it against GNU R checks the compiled tier on every case. A JIT helper or tier-0 step that releases handles compiled code holds fails an assertion that is on in release builds too, so the case reports as CRASHED. ry-tier0 runs with RY_JIT=off, and ry-gctorture runs with RY_GC_STRESS=1, which collects before every allocation once the script starts, and ry-gctorture-young with RY_GC_STRESS=young, which also keeps the survivors collectable. ry-jit-gctorture combines RY_JIT=force with RY_GC_STRESS=1, so it checks that compiled code keeps every object it holds rooted. ry-gctorture-seeded runs with RY_GC_STRESS=seed:0 and RY_GC_VERIFY=1, which collects at random gaps and checks the heap’s invariants before each collection (Robustness lists the modes).

ry-repeat evaluates each script twice in one process, and ry-repeat-gctorture-seeded does so under the seeded stress. The second pass defines every function again over the same code and calls it with the values the first pass left, so it runs what only a later call reaches: a compiled body, a call site’s plan, a reused frame. Only the first pass’s outcome is compared, because the second starts from another state. The second pass must finish: a crash or a failed runtime check in it fails the case. The driver does this when ORACLE_PASSES is above 1, printing a marker line before each later pass, which the oracle cuts standard output at. A case that cannot run twice in one process (it leaves standard output diverted, or consumes something the first pass used up) says so in a leading # oracle: runs once comment, and these engines run it once.

A case that reads or changes the collection stress setting itself (gctorture, gctorture2) prints something else under a stressed engine by design. It says so in a leading # oracle: sets collection stress comment, and the stress passes of test_cases leave it out.

--engine takes a comma-separated list, such as --engine ry,ry-jit-gctorture, which runs GNU R once per case and prints a summary line per engine. A failure listed in crates/oracle/known_failures.toml reports as known rather than failing the run (Testing has the rule). Without --engine, the engines are the ones ORACLE_ENGINE lists, else ry-tier0,ry-jit. Without GNU R, compare checks the engines against the recorded .expected files instead.

cargo test -p oracle runs the same check whenever ry finds a base R. It first builds the ry binary and libR.so (unless ORACLE_RY names a binary), so it never runs a stale runtime. Then every case under ry-tier0 and ry-jit, or under the engines ORACLE_ENGINE lists, must give its recorded outcome, sorted against the known failures. It needs no GNU R: the recordings stand in for it, and the GNU R replay checks them wherever GNU R is installed. ry loads its own base R here, not the GNU R installation compare would point it at.

The test asks each engine which base R release and which BLAS and LAPACK it loaded. Another release than the recordings’ skips the test, or fails it when ORACLE_ENGINE is set. Other libraries turn a difference in a # oracle: library-dependent case into a warning. ry’s bundled base R links R’s own LAPACK (libRlapack.so), not the system LAPACK the recordings name, so under it those cases only ever warn.

Every engine searches the same libraries: any R_LIBS you set, then the corpus library, then the reference R’s own. The corpus library is .corpus/library in the workspace ($RY_CORPUS_DIR/library when that is set), where just corpus installs the packages many cases need, and the oracle adds it whenever the directory exists. An R_LIBS set to the empty string leaves it out. The oracle asks GNU R for its .libPaths() once with those libraries in R_LIBS (and R_LIBS_SITE and R_LIBS_USER empty), then runs each engine, and the gallery and oracle bench runs too, with R_LIBS set to the answer and R_LIBS_SITE and R_LIBS_USER set empty. Without that, ry would search its own defaults: its bundled base R’s library, which holds none of the reference R’s packages, and any GNU R site or user library it finds, such as Debian’s /usr/lib/R/site-library, whose data.table 1.14.10, rlang 1.1.3 and withr 2.5.0 would load in place of the reference R’s. just doctor warns when ry run by hand loads other packages than the reference R and prints the variables that make it match:

Terminal window
R_LIBS=/opt/R/4.6.2/lib/R/library R_LIBS_SITE= R_LIBS_USER= ry run case.R

The packages themselves must be installed by GNU R. ry runs on GNU R’s base R when the two releases differ, so a package that ry installed (its DESCRIPTION says it was built under ry’s release) makes library() warn “package ‘x’ was built under R version 4.6.2”. The driver therefore reports the R_LIBS each engine saw, and a case whose engines saw different values fails with that message instead of a diff. A wrapper given as ORACLE_RY must not set R_LIBS itself.

ry’s native fast paths for base closures (crates/interp/src/base_fast_paths.rs) install only on the GNU R release whose code they follow, recognised by a hash. Against an older GNU R, such as a container’s 4.3.3, ry runs that release’s base R and most fast paths stay out, so only CI, which pins the matching release, checks them against GNU R. Locally, RY_FAST_PATHS=0 turns off the paths that skip R frames (message, signalCondition, and the warning shortcuts for no handler, an exiting handler and suppressWarnings), and running a case with and without it on ry’s own base R compares them with the R bodies they stand for.

Terminal window
cargo run -p oracle -- gallery # every figure, every device
cargo run -p oracle -- gallery --seed 3 --count 20 # a seeded sample
cargo run -p oracle -- gallery --figures base:persp_surface,lattice:lattice_xyplot_groups --devices agg

The oracle cases check graphics state by value, and the gallery checks what is drawn. Each crates/oracle/gallery/*.R file (base graphics, the PerformanceAnalytics and xts charts in finance.R, grid, lattice, ggplot2) declares figures with figure("name", { ... }), and oracle gallery renders each one under GNU R and under ry’s runtime on the agg (ragg), png, pdf and svg devices, then compares the images and times every render. A figure’s visible value is printed, as at the top level, because lattice, ggplot2 and xts draw only when printed. A figure whose packages GNU R lacks is skipped, and the gallery searches the corpus library like compare does, so just corpus is enough to run all of them.

ragg rasterizes the same way under both engines, so agg is the sharp check: at most 0.2% of pixels may differ. png and pdf (rasterized with pdftoppm to <figure>.pdf.png) allow 5%, and there a pixel counts as differing only when nothing within one pixel in the other image matches it, because GNU R’s cairo and ry’s renderer antialias differently. svg is timed but not compared, because cairo writes text as glyph paths. The report lists each figure’s differing share per device and both engines’ times; --repeats keeps the fastest of several renders. A figure name used twice in the gallery, or a device outside agg, png, pdf and svg, is an error. --seed picks the sample and seeds R’s random numbers in every figure, so a run replays exactly.

cargo test -p oracle runs a bounded sample (tests/test_gallery.rs, four figures on agg, or png without ragg), picked by FUZZ_SEED and sized by FUZZ_ITERS. It skips without GNU R or a built runtime, and a failure prints the oracle gallery command that replays it.

Terminal window
just corpus-sources # source tarballs, which carry the tests binary installs lack
just package-tests # every closure package, under GNU R and ry's runtime
just package-tests --runtime ry vctrs # one package, ry only, reusing the recorded GNU R result

oracle packages runs each corpus package’s own tests the way R CMD check runs tests/: every top-level tests/*.R file runs as its own script with tests/ as the working directory, and the file passes when its process exits with status 0. A package passes when all of its files pass, so a testthat suite passes when testthat.R finishes without a failed expectation. The harness also reads testthat’s summary line ([ FAIL 0 | WARN 1 | SKIP 2 | PASS 30 ]) to report expectation counts beside the status.

tools/corpus/sources.R downloads the source tarball of every package in the closure at its installed version into .corpus/sources and writes .corpus/closure.txt, the package list the harness runs. Results go to .corpus/tests/<package>/<runtime>/ (the copied tests/ directory, one .Rout per file, and result.json), and .corpus/tests/summary.md tables every package. A GNU R result is reused while the package version is unchanged, unless you pass --rerun-gnu-r. --jobs (default 4) runs packages in parallel, and --timeout (default 600 seconds) bounds one package’s suite under one runtime, after which its whole process group is killed. ORACLE_RY picks the ry binary; copy it, with libR.so beside it, to another directory first if you rebuild while a long run is going.

The pass rate counts the packages whose tests pass under GNU R, since a suite that fails there measures the container, not ry. With the packages ry installed itself, ry passes 140 of the 164 that GNU R passes (85.4%), and with the packages GNU R installed it passes 115 of 146 (78.8%). MEMORY.md records the current rate.

The corpus is the benchmark’s dependency closure, so ry is tuned to it. A second set measures packages the corpus has not mined: tools/corpus/top-packages.txt lists the 100 most-downloaded CRAN packages outside the closure, leaving out the few that need a system library the container lacks or take hours to build.

Terminal window
just top-packages-setup # GNU R installs them and their Suggests into .corpus-top, with sources
just top-packages --jobs 3 # ry builds them all from source, then both runtimes run their tests

just top-packages reinstalls every package into .corpus-top/ry-library with ry’s R CMD INSTALL and then runs oracle packages --installed against .corpus-top, so the results land in .corpus-top/tests/summary-installed.md. To rerun one package’s tests on ry without reinstalling, call the oracle directly:

Terminal window
ORACLE_RY=$PWD/target/release/ry ./target/release/oracle packages --corpus .corpus-top \
--installed --runtime ry ggplot2

Of the 100 packages, 85 have tests and GNU R passes 84; ry passes 82 of those 84 (97.6%). backlog.md lists the two that fail and why.

Terminal window
just ry-home # build release libR and ry-rcmd, assemble .corpus/ry-home
just corpus-ry-install # install the whole closure from source on ry
just corpus-ry-install vctrs tibble # only these, reusing what the library already holds
just oracle packages --installed # run the tests against ry's own installations

oracle install installs each package from its tarball in .corpus/sources with the R home’s bin/R CMD INSTALL, so configure, compilation and the R-level steps (lazy-load database, Meta/, help) all run on ry, into .corpus/ry-library. A package starts once everything in its Depends, Imports and LinkingTo has installed, or is already in the library when it is not part of the run, and is skipped when one of them failed. After installing, it is loaded with library() in a fresh bin/Rscript. Logs go to .corpus/ry-install/logs/<package>.{install,load}.log, and .corpus/ry-install/summary.tsv keeps one row per package: a run replaces the rows of the packages it names and keeps the others.

The R home holds a copy of libR.so, so after rebuilding run just ry-home again before installing. oracle packages --installed runs ry’s side through .corpus/ry-home/bin/Rscript with .corpus/ry-library ahead of .corpus/library (which still supplies the Suggests), and records results under ry-installed, beside the ry results of ry run --runtime ry, with the table in .corpus/tests/summary-installed.md.

GNU R’s binaries of graphics devices refuse ry’s graphics engine, which is one version newer than the one R 4.3.3 built them against, so a package whose tests plot through svglite needs svglite installed on ry: just corpus-ry-install svglite builds it from source.

The harness uses whichever GNU R is first on PATH. Two environment variables point it elsewhere, which is how to compare against a newer R installed beside the system one, for example with rig (rig add 4.6 installs under /opt/R/<version>):

Terminal window
ORACLE_R=/opt/R/4.6.2/bin/R just oracle verify # the single-file cases
ORACLE_RSCRIPT=/opt/R/4.6.2/bin/Rscript just oracle packages # the package corpus

Installing rig puts its R ahead of the system one on PATH; remove the R and Rscript links in /usr/local/bin to keep the system R as the default.

ORACLE_R covers verify, record and compare. ORACLE_RSCRIPT covers packages and bench. When ORACLE_R is set, the ry engine of compare also runs with R_HOME set to that R’s home, so ry’s runtime loads the same base library and emulates the same version as the GNU R it is compared with. That makes compare the check for a new R release:

Terminal window
ORACLE_R=/opt/R/4.6.1/bin/R just oracle compare --engine ry

Without ORACLE_R, the ry engine runs on its default base R only when that is the release of the GNU R on PATH. Otherwise, including when ry’s default release cannot be read because its start fails or times out, it takes that R’s home too, so a version-gated behavior never shows up as a difference.

ry follows the version of the base R it loads. Each behavior GNU R changed between 4.3 and 4.6 is gated on that version in crates/interp (system::r_at_least), so ry with R_HOME set to GNU R 4.3.3 behaves as 4.3.3 does, and ry on its bundled base behaves as 4.6.2 does.

Without any GNU R, compare checks the engine against each case’s recorded outcome instead and says so on stderr. The engine then runs on ry’s bundled base R 4.6.2, the release the recordings come from. A case that needs a package the engine cannot load is listed as skip ... needs <package>, and the run fails unless you pass --allow-skips (Testing).

The .expected files are recorded with GNU R 4.6.2 built from source with

Terminal window
./configure --enable-R-shlib --enable-memory-profiling --with-x --without-recommended-packages --disable-java

with the pango development headers installed (libpango1.0-dev), and with data.table, bit64, rlang, knitr, rmarkdown, ragg, lattice, R6, withr and xts installed from CRAN into its library. Some cases depend on that build:

  • utils__rprofmem and environments__sys_call_shares_arguments call Rprofmem(), so they need memory profiling. Without it GNU R stops with “memory profiling is not available on this system”.
  • The graphics cases need png() to draw through cairo, which R uses only when pango is present. Without it the bitmap type falls back to Xlib, which needs an X11 display, even though capabilities("cairo") reads TRUE. tools/doctor.py and tools/reference-r.sh therefore check getOption("bitmapType").
  • Collation and a few string cases follow ICU, which R uses when its headers are installed.
  • The datatable__*, grid__* and knitr cases print what those packages’ current CRAN releases print, and each declares the packages it needs in an # oracle: requires line.
  • strings__riconv_native_encoding switches to en_US.ISO-8859-1, which a stock Ubuntu container lacks. It declares the locale in a # oracle: requires locale line, and tools/oracle-locales.sh, which dev-setup and CI run, generates it.

sudo tools/dev-setup.sh builds exactly this into /opt/R/4.6.2 and installs the packages, and just doctor checks it. --enable-R-shlib changes no output: it builds the libR.so that ry’s console loads for GNU R and that binary packages link.

Other builds differ in the last digits too: OpenBLAS changes lm and arima results, and the sign of NA^NaN depends on the compiler. So does NaN + NA_integer_ (and *) in a closure’s first, uncompiled call, where GNU R’s AST interpreter gives NA or NaN depending on how the C compiler ordered the operands, so a case about NaN and NA makes those calls with NA_real_. glibc’s mktime starts its search from the offset its last call found, so a repeated wall-clock time with unknown daylight saving depends on what ran before; the cases that print one convert a summer time first.

crates/oracle/cases/gnu-r.txt names the build the suite was recorded with: its release, and its BLAS and LAPACK libraries by file name. Every verify, record and compare report starts with the build of the GNU R it ran, and a comparison against recorded outcomes names the recorded build. verify, record and the recorded_outcomes_match_gnu_r test check the build file before they run a case:

GNU R at hand verify record, RY_BLESS=1 recorded_outcomes_match_gnu_r
same release and libraries runs runs runs
other BLAS or LAPACK warns, runs warns, runs warns, runs all but the library-dependent cases
other release refuses (exit 2) refuses (exit 2) skips, naming both builds
no build file warns, runs writes the file fails (bless writes it)

A case whose expected digits are those of the numerical libraries declares it in a leading # oracle: library-dependent line: matrix products, lm, optim and the other cases that print BLAS or LAPACK results to the last bit, where another library also moves an optimizer’s iteration count and warnings. Only a GNU R on the recorded libraries can confirm those expectations, so recorded_outcomes_match_gnu_r skips them on other BLAS or LAPACK libraries and names each one it skips. The C math library and the compiler move results too, but the build file does not name them, so the cases that follow them carry the marker as well and are compared only where BLAS and LAPACK match too:

  • glibc’s x86-64-v3 build, which CI’s Ubuntu runner uses, changes pgamma, fft and nlm results.
  • A GNU R configured against glibc 2.41 or later calls the C library’s tanpi, sinpi and cospi instead of its own, which moves the last bit of the Cauchy quantiles (distributions__cauchy_quantiles).
  • A complex sum of an NA part and a NaN part is NA or NaN depending on how the compiler ordered the operands (kernels__complex_na_nan_sums).

Keep such a computation in a small case of its own, so the larger case it came from is still compared everywhere.

On a machine whose PATH has another release, set ORACLE_R to a 4.6.2 build. To move the suite to a new release, delete gnu-r.txt and record every case. A GNU R killed by a signal before it writes its report skips the case with skip ... this GNU R crashed, because some hardened builds abort with “buffer overflow detected” in graphics cases that the reference build runs; a crash of ry’s engines is a failure.

Record with ORACLE_R=<that R> oracle record crates/oracle/cases and review every changed .expected file.

Terminal window
just bench-gen --out .bench --lines 10000 # three packages of about 10,000 lines
just bench-gen --out .bench --lines 300000 --files 500
just bench-test .bench/finbench

tools/bench/tooling/generate.py writes finbench, dsbench and biobench under --out. Each is an R package with a DESCRIPTION that imports exactly the benchmark dependency set, a NAMESPACE, R files, and a testthat suite. The generator is deterministic for a given --seed, and its output is not committed. --domain generates one package, and --files fixes the file count (by default one file per 600 lines). The templates are written with inline annotations. --typed keeps them and turns on strict mode in the generated ry.toml, so the package doubles as a strict-mode dogfood corpus and ry’s runtime checks and compiles its typed functions. Without it, the generator erases the package with ry debug erase (so target/release/ry must exist, or pass --ry), which gives plain R. Both runs count the same text toward the line target, so a typed and an untyped run with the same seed generate the same program. Where a record or a tag union models the data better than S4, R6 or a named list, a template keeps both versions in #@ typed and #@ untyped blocks, so the typed program uses ry’s own data types while the untyped one stays plain R (the finance ledger, data-science readings and biostatistics followup templates model their data as tags, tuples and records this way); the templates’ README.md has the details. --only limits generation to the named templates, which is the quick way to iterate on one of them.

The code comes from the R templates in tools/bench/tooling/templates/ (see its README.md): shared infrastructure plus domain code, instantiated with random parameters until the line target is reached. Services that cannot run in a test (Azure, Bloomberg, ODBC) take an injectable client, so the tests never touch a network. just bench-test runs a package’s tests against the corpus library, so run just corpus first. Because polars and Rblpapi cannot be installed, it tests a copy whose Imports omit the missing packages; the code reaches them only inside guarded or injectable branches. The three 10,000-line packages pass all of their roughly 340 tests each under GNU R 4.3.3.

Terminal window
just bench-compare .bench/finbench .bench/dsbench .bench/biobench
just bench-compare .bench/finbench --filter portfolio --verbose

oracle bench runs a generated package’s testthat suite under GNU R and under ry’s runtime and compares the two test by test. Both runs go through ry load <package> --file bench.R, once with --runtime gnu-r and once with --runtime ry, so the same loader plan loads the package on both sides and only evaluation differs. The driver (crates/oracle/src/bench.R) records every test_that() block: its status, the failed expectations, errors, warnings and skips it reported, a deparse of its value, and its run time. A test matches when everything but the time agrees.

GNU R runs the suite twice. A test whose value or problems differ between those two runs (a fresh UUID, a timestamp in a message) is compared by status alone, so only what GNU R reproduces counts against ry. The table prints, per package, the matching tests, the rate, the tests that differ or that ry never reported, the tests ry passes, and both suites’ run times; process wall time, which adds startup and loading the package, follows it, and then the time each runtime spent collecting during the suite, as gc.time() reports it, with its share of the suite time. For pause distributions, set RY_GC_LOG=<file>: ry’s runtime then appends one line per collection, giving its kind (minor, intermediate or major), its elapsed nanoseconds, the bytes of small objects and of large vectors allocated since the previous collection, the old generation’s bytes after it, and the process id. Every process of a run appends to the same file, so the process id tells a test suite’s worker processes apart. Collections are deterministic: two runs of the same program in one process make the same sequence. A testthat suite that runs in parallel (Config/testthat/parallel) hands files to its workers as they finish, so its collections vary from run to run; set TESTTHAT_PARALLEL=false to compare them. --json writes every comparison, and --keep keeps each runtime’s report and stderr. Pass --ry target/release/ry for timings, with libR.so built beside it (cargo build --release -p ry-lang -p libr). MEMORY.md records the current rates.

The two time ratios answer different questions. Suite time adds up the test_that() blocks, so it measures evaluation. Process wall time also counts what happens between blocks, and at the full benchmark size (--lines 300000 --files 500) that is dominated by testthat calling gc() after every test file that registered a teardown. A full collection with the whole package loaded takes about 600 ms on GNU R and about 100 ms on ry, so ry can trail on suite time and still beat GNU R end to end.