Skip to content
ry

Testing

Most of ry is tested with fixtures: plain-text files that pair a piece of R with the output ry should produce for it. A fixture reads well in a diff, and a new case takes seconds to add. Reach for an ordinary Rust test only when the behavior is awkward to express as rendered output.

A fixture suite describes the desired semantics, not what the implementation happens to pass. Record a known gap explicitly, in .agents/memory/backlog.md or as a case pinned to the wrong-but-current output with a comment that says so, because an unrecorded gap looks intentional to the next reader.

Start every command with source tools/env.sh &&. Where nix is installed, which includes every Claude cloud container, it loads the flake’s default dev shell: the Rust toolchain, the reference GNU R, just and mold, the same versions CI tests with. It then selects mold for Linux links (through RUSTC_WRAPPER=tools/mold-rustc, which cargo leaves out of its fingerprints) and reports whether ry’s base R (target/base-r) is built:

$ source tools/env.sh
ry env: nix dev shell; GNU R 4.6.2 at /nix/store/...-gnu-r-4.6.2/bin; base R built; mold linker
Step Time on a 4-core cloud container
First source tools/env.sh in a fresh container 98 s, 8.8 GB downloaded into /nix/store, nothing built
Every later source tools/env.sh 0.06 s
nix develop -c <command>, for comparison 4.7 s per command
First debug build of ry, libR and ry-rcmd in the shell 5 min 20 s

The script saves nix print-dev-env in target/nix-dev-env.sh and sources that file, which is why later runs are fast; it rewrites the file when flake.nix, flake.lock, rust-toolchain.toml or the GNU R pin in tools/base-r is newer. Binaries built in the shell link nix’s glibc, so do not mix shell and host builds in one target directory: cargo rebuilds everything when you switch.

RY_ENV=system source tools/env.sh skips nix and uses the host’s tools, with the reference GNU R from /opt/R/<version>/bin. Use it for the package corpus (just bench-packages), whose packages link the Ubuntu libraries that sudo tools/dev-setup.sh installs and the shell does not carry. Without nix the script behaves the same way.

Every command below is plain cargo, so it works without just. A passing run prints only cargo’s summary, and a failure prints a rerun command for just the failing case.

The time is from touching one source file of the crate to the result, on 4 cores in a debug build with mold, GNU R on PATH and base R built.

You edited Run Time
the parser (syntax) cargo test -p syntax --test test_syntax_fixtures; the whole crate, cargo test -p syntax, takes 18 s, 15 of them in fuzz_edit_stream 1 s
the checker (semantics) cargo test -p semantics --test test_typing_fixtures 10 s
the formatter (format) cargo test -p format --test test_format_fixtures 1 s
an editor feature (ide) cargo test -p ide --test test_ide_fixtures 4 s
the interpreter, to run R cargo build -p ry-lang -p libr && target/debug/ry run --runtime ry x.R 13 s
the interpreter, its tests cargo test -p interp --test test_<area> 4 s
the interpreter, against GNU R cargo run -p oracle -- compare crates/oracle/cases/<case>.R 10 s
the interpreter, every oracle case cargo test -p oracle --test test_cases ry_matches_recorded_outcomes 10 min
the typed core (typed) cargo test -p typed 4 s
the CLI or the server (ry) cargo test -p ry-lang --test test_cli, or one journey: SCENARIO=<name> cargo test -p ry-lang --test test_scenarios scenarios_hold 6 s

cargo build -p ry-lang does not build the runtime: ry loads libR.so at run time, so an interpreter edit needs -p libr too. Tests and oracle compare that run ry’s runtime build both themselves (a no-op build takes 0.4 s), so they never run a stale libR.so.

Variable Read by Effect
FIXTURE_FILTER every fixture suite runs the cases whose group__case id contains the value (pipes__ runs a group)
RY_BLESS=1 fixture suites, test_docs, oracle test_cases rewrites the expected output from the actual output
FUZZ_SEED, FUZZ_ITERS, FUZZ_DEPTH every seeded fuzzer and simulator replays one seed; sets the iteration count; multiplies the default count
SCENARIO test_scenarios scenarios_hold runs the one scenario of that name, and fails if it skips
RY_TEST_WORKERS fixture suites, oracle, scenarios, test_format_range the worker count; failures print the count the run used
DOCS_PAGE test_docs runs the examples of pages whose path contains the value
ORACLE_ENGINE oracle test_cases compares those engines (comma separated: ry, ry-jit, ry-gctorture, …) instead of ry-tier0,ry-jit, and fails if base R is not built or is another release than the recordings’
ORACLE_RY the oracle the ry binary to run, which the oracle then does not build
ORACLE_R, ORACLE_RSCRIPT the oracle the GNU R to compare with (default: R and Rscript on PATH)
RY_BASE_R ry the base R tree; a debug ry otherwise finds target/base-r
R_HOME ry, scenarios a GNU R installation whose base R ry emulates
RY_LIBR ry the libR.so to load instead of the one beside the binary
R_LIBS, RY_CORPUS_DIR oracle, scenarios the package library cases and journeys load from (default: .corpus/library)
RY_SKIP_GNU_R=1 libr tests skips the parts that need GNU R even when it is installed
RUSTC_WRAPPER cargo tools/mold-rustc links Linux targets with mold; tools/env.sh sets it on Linux

A test that cannot run here (no GNU R, no base R, a missing package) passes and says why on standard error, which cargo shows only for a failing test or with -- --nocapture. Stable Rust has no way for a test to report itself skipped at run time, so a green run of the default oracle test or of the scenarios may have checked less than it looks. A run you ask for by name does not pass that way: SCENARIO=<name> fails when the scenario skips, and ORACLE_ENGINE=<engine> fails when the engine is ry and base R is not built.

The parallel harnesses deal items out statically, so worker w of n takes items w, w + n, and so on. A fixture suite reuses one analysis database per worker, so a failure that depends on what rendered before it on the same worker replays with the whole suite and the same count, which the failure message prints as RY_TEST_WORKERS=<n>.

Whole test binaries, run only (already built), 4 cores, debug, GNU R on PATH:

Binary Time Why
jit test_tiers 160 s four seeded fuzzers (150 to 400 seeds each) run 115 to 125 s apiece in parallel; the binary is CPU-bound
ry-lang test_scenarios 120 s 80 journeys and the typed benchmark programs, each spawning ry processes; parallel inside each test, CPU-bound
oracle test_cases 180 s with GNU R, 25 s without every case under GNU R against its recording, and the collection-stress replays; without GNU R, 5 s of it is the 24-case ry sample and the rest the three stress replays
frames test_standard_library 82 s standard_library_at_runtime evaluates every case in one process: the interpreter is a process-wide singleton, so it cannot run in parallel
table test_table 42 s engine model fuzzers
format test_format_range 35 s the range-format battery over every fixture and corpus source, in parallel

cargo run -p oracle -- compare runs the whole oracle in 118 s against the recorded outcomes and 184 s against a live GNU R, one case per core, printing only the cases that need attention.

Every suite in the shipping crates runs through one harness, syntax::testing::run_fixture_suite, which syntax, semantics, format, and ide all share. A .test file holds groups of cases:

#==== group_name
#---- case_name
<input source>
#++++
<expected rendered output>
  • group__case is the case’s stable identity. It must be unique across the suite, and a duplicate is rejected rather than allowed to shadow the other case silently.
  • What the expected output shows (schemes, diagnostics, a tree dump, hover text) is up to the suite’s runner. Behavior specific to one suite belongs in its runner, never in the fixture syntax.
  • A typing or IDE case may hold several files. Each #~~~~ <path> line starts one (see syntax::testing::split_files), and a path under R/ is a package file while anything else is a script. A typing case renders each file’s output under an == <path> line.
  • The runner loads only .test files, so anything else in a suite directory is ignored.

To run a single case, set FIXTURE_FILTER and pick the suite’s test target. The filter selects every case whose group__case id contains it, so pipes__ runs a whole group:

Terminal window
FIXTURE_FILTER=group__case cargo test -p semantics --test test_typing_fixtures
FIXTURE_FILTER=group__case cargo test -p format --test test_format_fixtures
FIXTURE_FILTER=group__case cargo test -p ide --test test_ide_fixtures

One test target can drive several suites, so a suite that holds no matching case skips quietly. But a filter that matches no case in any suite fails: that is almost certainly a typo, and a run of zero cases would otherwise report a pass.

A suite renders its cases on one thread per core and reports the failures in file order, each ending with the command that reruns it:

rerun: FIXTURE_FILTER=annotation_compact__atomic cargo test -p format --test test_format_fixtures format_fixtures -- --exact

While iterating, cargo test -p semantics is the default command, since it covers analysis behavior. just gate runs every test plus clippy and a formatting check.

Tests run locally, with just gate for everything and the two recipes below for the part that needs no GNU R. CI runs tests only when a maintainer starts a workflow by hand or tags a release, never on a push or a pull request. The workflow for the part without GNU R, test.yml, is staged in .github/pending/ and runs nothing until a maintainer accepts it (Releasing). It has two jobs:

Job Recipe What it runs
tests just test-fast Every test of the crates that build without the interpreter, libR and Polars (syntax, format, semantics, ide, sniff, loader, heap, numeric, ssa, rdb, crash, oracle, rcmd, repl and ry-lang): every fixture suite, the property fuzzers’ bounded passes, the simulator and the scenarios. A test that compares with GNU R or runs ry’s runtime skips, because the runner has neither
fuzz just fuzz-all 60 runtime Every libFuzzer target that builds without Polars (15 of 17, all but csv and unserialize) for 60 seconds of wall clock each. Every target runs, and the job fails if any finds a failure. A failing input is uploaded with the run

The staged release.yml calls test.yml, and its publish job needs it, so once both files are accepted a release with a failing test publishes nothing.

The full gate is ci.yml. It runs just ci, which builds base R and runs rustfmt, clippy and the tests with the CI runner’s variables, inside the flake’s ci dev shell. That shell pins everything the tests touch (the Rust toolchain, GNU R 4.6.2 built as the oracle recorded it, GCC 13.3, glibc and the oracle’s locales), so this command reproduces a CI run on any machine with nix:

Terminal window
nix develop .#ci -c just ci

To change what a workflow runs, edit its recipe or the shell in flake.nix; the workflow files only call them. The full gate runs only when started by hand: the orchestrating agent starts it on main at its hourly check when main has new commits. A pull request gets no run of its own.

With RY_BLESS=1, a mismatch does not fail. Instead, the harness rewrites the #++++ blocks in the .test files in place from the runner’s actual output:

Terminal window
RY_BLESS=1 cargo test -p semantics --test test_typing_fixtures

Only the body of each #++++ block is rewritten. Directive lines and the spacing around them are kept, so a blessed file is byte-identical to what a person would have written, and a second run without bless passes. FIXTURE_FILTER still applies, so you can bless a single case, and blessing a suite that is already correct changes no bytes.

Review every blessed change before you commit it. Bless records whatever the runner produces, wrong outcomes included, so accept a changed expectation only when the behavior or the wording improved on purpose.

  • crates/syntax/tests/syntax: golden lossless trees and syntax errors, rendered by debug_dump.
  • crates/syntax/tests/tsr: tree-sitter-r’s parser corpus, converted to the same format.
  • crates/syntax/tests/errors: golden Elm-style error messages, organized by area (the lexer, delimiters, control flow, functions and calls, the #: type and directive grammar, and recovery locality). Its coverage contract has three parts: every distinct error message the lexer or parser can emit has at least one case here, valid but tricky shapes are pinned as no errors, and recovery cases assert that one broken construct reports once and leaves the rest of the file clean.
  • crates/semantics/tests/naming: the name-resolution suite. It records which binding every name in an item resolves to, rendered flat in source order. This is the only suite that tests naming directly. Everywhere else naming is tested through its consequences, a type that comes out right or a diagnostic that does or does not fire, and consequences cannot see a read that resolves to the wrong binding while still producing the same type.
  • crates/semantics/tests/typing: the typing suite. Each case runs the full semantic pipeline on one package file, or on several, with the shipped stubs installed. It renders the exported scheme of every top-level definition (name: <T: numeric> fn(x: T) -> T), the type of every other top-level statement as a bare line, and then the file’s diagnostics (start..end severity[code] message, in byte offsets). Each test thread reuses one database across cases (semantics::testing::with_fixture_project), because parsing the stub corpus costs far more than checking a case. Cases are grouped by feature, with one exception: programs.R.test holds whole small programs written the way real projects are written. A case earns its place there by exercising several features at once, and a clean case with an empty diagnostics block is a promise of zero false positives for that style of code.
  • crates/semantics/tests/lowering: the HIR each item lowers to, dumped as an indented tree under the item’s name (or under # <statement> for an unnamed item). Argument tags are rendered inline, because matching arguments is what a call’s lowering has to get right. The suite exists because rewrites, such as the native pipe becoming a call, a replacement form, or local, would otherwise be tested only through the type that falls out at the far end, which cannot tell “lowered correctly” from “lowered to the wrong shape but typed the same”. A case whose comment says WRONG-BUT-CURRENT pins a known defect next to the shape it should have, so fixing the defect turns the case red.
  • crates/semantics/tests/typing-scripts: the same pipeline over script documents, which have one sequential, top-down scope. Besides script semantics, this suite holds the rules for single expressions (literals, operators, indexing, lists, and calls to stubbed functions), rendered one line per statement.
  • crates/semantics/tests/annotations: one #: block per case, where the case’s lines are the block. It renders what the block lowers to (the kind of coercion, the declared type, the @new target, and each @type or @alias definition), followed by its grammar and lowering errors. This is the compact contract for the type notation, while the syntax suite pins the trees and the errors suite the wording.
  • crates/semantics/tests/stubs: .Rtypes declarations. Each case renders the declarations it loads, with overload candidates in order, and the problems that make the loader drop a line, in the wording the editor and the override report show.
  • crates/semantics/tests/typing-inline-off: the ordinary rendering with [check] inline-annotations = false, which pins the ry-syntax warning on records, tuples, tags and match, and shows that literals are still typed while match coverage is not checked.
  • crates/semantics/tests/typing-strict: the strict stream. After the ordinary rendering, each case appends the file’s typing mode and its strict diagnostics.
  • crates/semantics/tests/top-level: what a package file’s top level runs under strict mode. A case lists files with #~~~~ separators, so R/ files are library files and tests/ or root files are entries. Each case loads the shipped stubs and renders, per file, every top-level item as definition, constant or effect, with (pure) when the item declares its function pure, followed by the file’s top-level strict diagnostics. Run it with cargo test -p semantics --test test_typing_fixtures top_level_fixtures.
  • crates/semantics/tests/typing-imports: package metadata. A leading #namespace line supplies the case’s NAMESPACE source and a #description line its DESCRIPTION source, and both are installed as the PackageMetadata input before rendering. The directive lines stay in the analyzed text as ordinary comments, so expected offsets are honest. Attaching a package needs no directive: a library(data.table) statement in the source activates the conditional stub namespace through the same scan the hosts run.
  • crates/semantics/tests/lints, lints-style and lints-object-systems-off: lints::lint_file, first under the default configuration, then under naming-style = "snake_case", unused-parameter = "warn", shadows-builtin = "warn", and shadows-namespace = "warn", all of which are off by default, and last under [check] object-systems = false. Each case renders every finding as start..end severity[code] message, or clean. Run all three with cargo test -p semantics --test test_lint_fixtures.
  • crates/semantics/tests/load_index: the package-load index. A case lists R/ files, and optionally NAMESPACE and DESCRIPTION, with #~~~~ separators, and renders the package’s name, the NAMESPACE directives a load honors, and, per file, each item’s position and whether it is bound lazily (as a function or, in a strict file, a constant) or runs eagerly, with what it writes, what it reads while running, and the registrations it makes. Run it with cargo test -p semantics --test test_load_index_fixtures.
  • crates/loader/tests/reload: reload plans. A file headed #~~~~ before <path> or #~~~~ after <path> exists on that side only, so the same path on both sides is an edit and one side alone is an addition or a deletion, while a plain #~~~~ <path> is unchanged. The case loads the before side, then renders the plan that takes it to the after side: what is bound, run, removed, unregistered, re-registered, and which datasets are read again. R/ files run in the order DESCRIPTION’s collation field gives them, and a data/ file is a dataset whose text stands in for its size and modification time. Run it with cargo test -p loader --test test_reload_fixtures.
  • crates/format/tests/format: the formatter’s golden suite. Each case’s source must format to the expected block, and the runner formats that output again to assert idempotence on every case. When the expected result is a refusal, the case renders the structured FormatError.
  • crates/format/tests/format-range: range formatting. The source marks the selection with $0 markers (two for a selection, one for a bare caret, none for the whole file), and the expectation lists the edits by one-based position and shows the document they leave behind, with each replacement between « and ». Run it with cargo test -p format --test test_format_range_fixtures.
  • crates/ide/tests/ide: the editor features. The source carries one $0 cursor marker, which is stripped before analysis, and the expectation renders each feature’s answer at that position: the hover line with its absolute range, the definition’s target range, and the reference ranges. A target inside the stub corpus renders the token its range covers, as in declared in stub source 0 at `print` , rather than a byte offset, because an offset shifts whenever an unrelated declaration is added, while the token proves the range is right.
  • crates/ide/tests/structure: whole-file editor features. Each case renders the outline (nested by indentation), the folds as zero-based inclusive line ranges, the semantic tokens for R names, the quick fixes the diagnostics and default lints offer, and, when the source has a $0 marker, every step of selection expansion from it. Run it with cargo test -p ide --test test_ide_fixtures structure.
  • crates/ide/tests/migrate: the data.table to DataFrame rewrites behind the code actions and ry migrate data-table. Each case is one script whose attached packages are the ones it attaches, and the expectation lists every rewrite as its title, the original lines (- ) and the replacement (+ ), or no rewrite. A $0 marker also lists the code actions offered there. Run it with cargo test -p ide --test test_ide_fixtures migrate.

Fixtures check the answers someone wrote down. The harnesses in this section check properties that must hold for every input, over a great many inputs. Every pipeline stage gets fuzz coverage from its first commit.

The batteries of invariants are exported, so that the in-tree harnesses and the coverage-guided targets share exactly the same contracts: syntax::testing::check_parse_invariants for the parser, format::check_format_invariants for the formatter, and semantics::testing::check_semantics_input for the semantic pipeline. The last one folds every lint, under a configuration that turns them all on, into its rendering, so the lint layer inherits the same invariants. The format and semantics harnesses, and those of syntax and ide, also carry a fuzz_regressions_hold_invariants battery that pins every input the coverage-guided targets have ever broken.

crates/semantics/tests/test_fuzz.rs runs generated programs, token soup, and two-file projects through the full semantic pipeline, with the generator biased toward reference cycles, closures, annotations, and typing-mode directives. On every input it checks four properties:

  • Nothing panics, which among other things means every salsa fixpoint converges.
  • The output is deterministic across fresh databases.
  • Diagnostic ranges are well formed.
  • Incremental equals fresh. Editing a file through the salsa setter gives the same output as a fresh database on the edited text, and editing it back restores the original.

FUZZ_ITERS scales the budgets. A bounded default runs inside cargo test -p semantics, the ignored fuzz_deep test carries the long runs, and on a panic the harness prints the inputs that produced it.

crates/format/tests/test_fuzz.rs feeds the formatter valid-program seeds, byte-level mutations of those seeds, token soup, random bytes, every fixture case source in the repository, and real corpus files once they are fetched. It checks four properties: nothing panics (the formatter either succeeds or refuses with a structured error), the output is deterministic, the code is preserved, and formatting is idempotent, meaning that formatting a successful output again succeeds and reproduces it byte for byte.

The refusal path is part of the contract. A file with a syntax error in its R is refused, while an error in a #: annotation, which the parser marks in_annotation, only sends the affected block down the verbatim path.

Preservation is the one property that can catch the formatter changing your code. Determinism, idempotence, and “the output formats again” all hold for a formatter that silently drops a statement or misspells a token. Preservation compares the sequence of non-trivia token kinds and spellings between input and output, with two allowances:

  • Four kinds are ignored entirely ({, }, ;, and #:), because the formatter may add or move them: it braces single-statement bodies, splits ; chains onto separate lines, and re-lays out annotation blocks.
  • A string is compared by its content rather than its text, because the formatter chooses the quote character. Raw strings are copied byte for byte, so they keep their full spelling.

Comparing kinds alone would miss a miscompile: a stale or off-by-one source range makes the formatter emit the wrong bytes for an identifier, which renames a variable while preserving every kind. An injected bug of exactly that shape passed a kind-only comparison and fails the spelling comparison.

Every generated input also runs the range battery, check_range_format_invariants, over a bounded spread of selections (the whole file, carets, whole lines, part-line spans, a range past the end). Range edits must be deterministic, refuse exactly when the whole file refuses, cover whole lines in order without overlapping, preserve the code, leave a document that still formats to the same result, be a subset of the edits formatting the whole file in place makes, reproduce that result when the selection is everything, and be stable when re-run on what they produced. On inputs of up to 24 lines it also checks additivity (a span of lines makes exactly the union of its lines’ edits) and composition (formatting line by line until nothing changes reaches the whole-file result, and every step still formats to it). Composition is the strongest of these: it found whole-file idempotence bugs that no single-pass check sees. crates/format/tests/test_format_range.rs runs the battery over every fixture source, the program corpus, and re-indented copies of the fixtures (four spaces, tabs, mixed steps), and checks every byte selection of a few short sources against an independent line oracle.

crates/ide/tests/test_fuzz.rs sweeps every feature over a sample of byte offsets in each input: every token boundary plus a coarse stride, so positions off token boundaries and in the middle of multi-byte characters are covered too. At each offset nothing may panic, and every range handed to the editor must lie inside the text it indexes.

Containment alone is a weak check, though: it says a range is somewhere in the file, not that it is on the right thing. Wherever the cursor sits on a name, two stronger properties hold as well:

  • Name identity: every rename edit, every reference, and the definition target cover text that spells the same name (with backtick quoting normalized, since `x` and x are one binding).
  • Round trip: if the cursor navigates to a declaration, asking for references from that declaration leads back to the cursor’s own token.

An off-by-one injected where item-relative ranges convert back to absolute offsets passes the containment check and fails name identity on the first seed.

The data.table rewrites are checked once per input: they never overlap, every range lies inside the text, and when the input parses cleanly, so does the text with every rewrite applied.

Completion is sampled once per completion context (the kind of token the cursor sits in or after) rather than at every offset, because it costs far more per call than any other feature and its answer depends on the syntactic context rather than the exact byte.

crates/syntax/tests/programs/*.R.corpus holds 1,967 distinct R programs, read through syntax::testing::program_corpus_sources. The syntax, format, and semantics batteries each run the whole set. The corpus holds inputs only, with no expected output.

The inputs are usable on their own because the invariants need no expected output: nothing panics, syntax checks the lossless reprint and the tree geometry, format checks preservation, and semantics checks that every diagnostic range lies inside its file. The semantics arm shares one database across the whole corpus instead of using the per-input battery, because a fresh database re-parses the stubs every time. It therefore asserts only what a shared database can, and leaves determinism and incremental equivalence to the generated inputs. Every arm also asserts that the corpus is non-empty, so deleting it fails loudly instead of passing on zero cases.

crates/semantics/tests/test_roundtrip.rs asserts that every type the checker prints is a type you could write back. A #: TYPE annotation says the annotated value is compatible with TYPE, and the checker has just proved the value has the type it rendered, so pasting an inferred scheme above its own definition must add no finding.

No other test compares the renderer with the annotation grammar: every other suite reads a rendered type as a string, so a type that prints well but parses back as something else is invisible to them.

The test runs over every fixture case source. It skips cases that do not already check clean, because “re-declaring adds no finding” needs a baseline with no findings, and it skips definitions that already carry a #: block, because those render the user’s own text rather than a type the checker chose. It re-points one database with the salsa setter instead of building a new one per probe, because a fresh database re-parses the whole stub corpus.

KNOWN_UNWRITABLE lists schemes that are allowed not to round-trip. It is empty, and worth keeping that way. Its entries name case ids rather than skipping by shape, so a new unwritable rendering of any kind fails the test instead of blending into a known category.

crates/syntax/tests/test_corpus.rs compares our verdict on “does this file have errors?” with tree-sitter-r’s, in both directions and over two sets of inputs:

  • corpus_acceptance runs over the fetched corpus/ and gates the direction where we reject what tree-sitter accepts, which means a gap in our grammar.
  • in_tree_acceptance runs over inputs that are always present (the program corpus plus every fixture case source) and gates the other direction, where we accept what tree-sitter rejects.

That second direction is the only check that bounds the number of parse errors from below. Every other parser invariant bounds it from above, so without it an over-suppressing parser, the regression most likely to slip in given how much deduplication the parser does, would pass every fuzz battery while broken files went silently clean.

Each direction is gated only where it means something. Rejecting what tree-sitter accepts is noise on in-tree input, which is full of #: annotations (comments, to tree-sitter) and of deliberately broken sources. Accepting what tree-sitter rejects is gated there against crates/syntax/tests/in-tree-acceptance-allowlist.txt. Each allowlist entry is adjudicated by running R itself, and must say which of two things it is: tree-sitter being wrong, or a known gap in this parser.

Inline type annotations are ry syntax that R rejects by design, so tree-sitter judges the plain R program a source erases to (syntax::erase_inline_annotations), not the source itself. That makes the test a check on erasure too: an annotated fixture whose erased text tree-sitter rejects fails it. check_parse_invariants pins erasure from the other side, for every fuzz input that parses cleanly: the erased text parses cleanly, holds no inline annotation, and erases to itself.

The runtime is tested against models and against GNU R rather than with fixtures. The oracle’s cases (crates/oracle/cases/, Corpus and oracle) hold the R-level behavior, and crates/interp/tests/test_fuzz.rs fuzzes lowering and checks that the tier-0 bytecode agrees with the evaluator. crates/jit/tests/test_tiers.rs does the same one tier up: it runs random loop, branch and vector programs as closure bodies under tier 0 and under the JIT forced on (jit::Mode::Force, which compiles a body on its second call and a loop on its first backward jump), and every call must give the same value or error. Most generated programs must run to a value, so a generator that drifts into erroring early fails the test instead of passing without comparing anything. The test interpreter has no base R closures (numeric, as.list, withCallingHandlers), so programs use builtins, and each hand-written seed states whether it is meant to fail. Behavior that needs base R, such as a warning handler watching a compiled loop, goes in the oracle case bytecode__hot_loops.R. crates/interp/tests/test_linalg.rs pins base R’s linear algebra to GNU R’s results bit for bit. The matrix products are ry’s own kernels, so their bits hold on every machine; the LAPACK cases hold only on the reference libraries, so on another BLAS (such as OpenBLAS) the test compares them with a GNU R on the same libraries, and skips them without one. crates/table/tests/test_table/main.rs checks the table engine against values read from data.table, lazy.rs beside it checks the Polars-style verbs, and crates/libr/tests/test_frame_altrep.rs checks the columns as ALTREP vectors. crates/interp/tests/test_records.rs and test_arrays.rs and crates/frames/tests/test_standard_library.rs cover ry’s own values at run time, the last one also holding the runtime’s function table against the declarations in types/std.Rtypes. crates/libr/tests/test_rmath.rs, test_numerics.rs and test_linpack.rs compare the C API’s numerics with answers recorded from GNU R, so they run without R. The optimizer cases test_numerics.rs holds bit for bit are skipped, with a message, where they cannot match: against a fresh GNU R that does not call the reference BLAS, and, for the trust-region case, where the C library’s hypot rounds differently from the recording’s (its hypot case fingerprints it). test_fuzz_rmath.rs checks properties of the distribution functions on random and extreme arguments. crates/heap/tests/test_gc.rs builds random object graphs (vectors, lists, interned strings, environments), drops roots, stores young objects into old ones and collects at random points, with and without collection stress, and after every collection checks each object the model still reaches. It also re-interns every string ever made, so an intern-table entry that outlived its string shows up. A disabled write barrier, a frame whose cells are not traced, and a minor collection that keeps dead strings in the table each turn it red.

Tests that load base R take it from the GNU R installation R_HOME names, and the runtime then behaves as that version; without R_HOME they use ry’s own base R (just base-r), and the scenarios skip when it is not built. The oracle’s expectations are GNU R 4.6.2 output, so just ci sets R_HOME to 4.6.2. Run them once per supported version, for example R_HOME=/opt/R/4.6.1/lib/R cargo test -p interp, and the oracle the same way with ORACLE_R (Corpus and oracle).

crates/libr/tests/test_fuzz_native.rs fuzzes the native boundary itself. It nests every non-local exit native code can set up (Rf_eval, R_tryEval, R_ToplevelExec, R_tryCatchError, R_UnwindProtect, R_ExecWithCleanup and a cpp11-style C++ unwind) around values and errors, and checks the outcome, which cleanups ran, and that every C context, protection and R_alloc block was released. A second battery drives random API sequences under collection stress against a model.

oracle fuzz runs whole R programs under GNU R and ry’s engines and reports every program on which an engine differs. A seed and a kind name a program, so oracle fuzz --kind K --seed N --print prints it and --engines replays it. The five kinds find different bugs:

Kind A program is It finds
grammar (oracle::fuzz::program) several blocks from a grammar of language features: loops shaped for the JIT, closures, S3 dispatch, conditions and on.exit, environments, attributes, NA and overflow, and element stores into a function’s argument copy on both sides of 64 elements, in a function called three times so its body runs interpreted and compiled; about one in eight ends on a deliberate error interpreter and JIT bugs in control flow and scoping
metamorphic (oracle::metamorphic::program) six to twelve trials; each builds one value, runs one body of updates and reads on it, and runs the same body in other forms that must give an identical result a result that depends on how often or where code runs, or on how a value is stored
builtins (oracle::metamorphic::builtins_program) builtins called on a zoo of values, each call caught a builtin that accepts, rejects or computes differently from GNU R
api (oracle::metamorphic::api_program) calls of functions drawn from everything base, stats and utils export, on two zoo values and constants of other types, each call caught a function nobody listed: one that crashes, or accepts, rejects or computes differently from GNU R
corpus (oracle::corpus) a snippet of real R code with one to four mutations what no grammar author thought of

A metamorphic trial draws its value from a zoo: every atomic type and lists, with NA, NaN, Inf and -0; zero-length vectors and lengths on both sides of the sizes where storage changes (the inline capacities, the 512-byte small-vector limit, the 64 KiB arena chunk, a compact 1:n); names that are empty, NA, duplicated or non-ASCII, attached through names<-, setNames, structure, argument tags and do.call; strings marked UTF-8, latin1 and bytes; matrices, arrays, data frames, factors and dates. The body is a few statements over x: element, slice, matrix and name assignments, attribute and replacement-function calls, <<-, loops, and reads by position and by name. The trial computes the reference result once, then compares it by identical() with the body’s result in each other form and prints same or DIFFERENT:

Form The body runs
again xK K times in a row in the same closure, K from 1 to 300
warmed, cold in the closure after it ran on values of other types and sizes, and in a copy of it that never ran
local, inline, evaluated, on.exit inside local(), in an anonymous closure, through eval(quote()) in a new environment, or in a closure with an on.exit handler
top level as top-level statements on a global variable
renamed NAME with x under another name, among them a non-syntactic one, a non-ASCII one, T and c
caller called through do.call, lapply, Map, Reduce, replicate or tryCatch
shared on a value that a second variable, a list, an environment, a closure or an attribute also holds, whose copy must come out unchanged
collected with a gc() after every statement
serialized v2, serialized v3 on the value after unserialize(serialize())

A DIFFERENT line is a bug in the engine that printed it, whatever GNU R says, so the kind is also a test without GNU R. Each seed turns on a random subset of 33 features (the value types, the name and encoding sources, the statement families, the forms), so a rare pair of features gets programs of its own instead of drowning in the common ones.

Printed output hides much of a value, so every engine runs with the deep observer on (crates/oracle/src/observe.R, ProcessEngine::observing, off with --no-observe). After each top-level expression the driver records a canonical dump of its value and whether it was visible: the type, length and object bits, every element (a double by its bits, a string by its encoding mark and bytes), the attributes in the order attributes() gives them, and the structure of calls, closures and environments. A vector of more than 96 elements shows its ends and, for the middle, checksums that depend on the position of every element. Each condition is recorded with its classes, message and call. A non-ASCII string shows the same mark whether it is flagged UTF-8 or unflagged, because the two are the same text in a UTF-8 session; ORACLE_OBSERVE=strict tells them apart.

The builtins kind calls a hand-written list of functions with arguments that suit them. The api kind takes its functions from a table instead, crates/oracle/src/api_functions.txt, which api_functions.R writes from GNU R’s namespaces (regenerate it after a GNU R upgrade: Rscript crates/oracle/src/api_functions.R > crates/oracle/src/api_functions.txt). The script leaves out what cannot be compared: functions that read or write files, connections, the terminal, the clock, the session or the graphics device, most dot-prefixed internals, and one function on which GNU R corrupts its own memory. A formal that names a file or a package is never passed. A call gets up to three positional arguments (the first is usually the zoo value v, the second w, the third a small count) and sometimes one named argument, and a replacement function is often called through replacement syntax (x <- v; names(x) <- w). A third of the calls print their result, so print methods run too. Most calls are errors in both engines, and then the messages are compared.

The report of a diverging seed names each top-level expression that differs and how:

Difference Meaning
crash the engine died of a signal or a panic
value the expression’s value or visibility differs
accepts, rejects one engine signals an error where the other returns
message both signal an error, with another message, call or class
conditions the values agree; a warning’s or message’s text, call or classes differ
printed the values and conditions agree and the printed text does not

--jobs N runs N programs at once, --minutes M keeps taking seeds until the time is up, --findings DIR writes each diverging program and its report to a directory, and the summary counts the differences by kind. For the builtins and api kinds it also counts them by the function called and writes DIR/functions.txt, one count, kind and function per line, most frequent first, which ranks the gaps:

14 accepts type.convert
10 rejects aregexec
7 value nargs

oracle fuzz --kind all --count 500 --coverage (just fuzz-coverage) runs no engine. It generates the programs and reports how many of the table’s callable functions each kind names in a call, so a generator change shows what it added to the reach:

kind all base stats utils
(the table) 1479 1000 427 52
grammar 144 143 0 1
metamorphic 131 129 1 1
builtins 297 285 7 5
api 1473 995 426 52
corpus 649 458 178 13
together 1474 995 427 52

The count is a close upper bound, because a name inside a string literal counts as a call.

Every fuzzed engine runs with at most 2 GiB of writable memory (oracle::fuzz::MEMORY_LIMIT, RLIMIT_DATA), so a call that asks for a huge vector gets R’s “cannot allocate” error in GNU R and in ry instead of taking the machine’s memory. just fuzz-differential runs the grammar, just fuzz-kind KIND any kind, and just fuzz-long HOURS BASE all five for a time budget, under tier 0, the forced JIT and seeded collection stress.

crates/oracle/tests/test_differential.rs holds the bounded passes. Grammar programs run under GNU R, tier 0 and the JIT, and under collection stress against ry unstressed. Metamorphic, builtin and api programs run under tier 0 and the forced JIT, which must agree with each other, print no DIFFERENT and never crash (fuzz_metamorphic_forms_agree_across_tiers, fuzz_builtins_agree_across_tiers, fuzz_api_calls_agree_across_tiers); their comparison with GNU R is the long run’s, because ry differs from GNU R on many builtin and api programs and a default test must pass.

A divergent program is 50 to 300 lines, so a finding is reduced before it becomes an oracle case. oracle reduce FILE --engine NAME (just fuzz-reduce FILE ENGINE, or oracle fuzz --reduce for every diverging seed) shrinks it by delta debugging over the syntax tree: it removes statements, block items, arguments and parameters in halving chunks, replaces a construct by one of its parts (an if by a branch, a call by an argument, a loop by its body), replaces compound expressions by NULL, 0, 1L, "" or TRUE, and shrinks string and number literals. A candidate counts only when GNU R parses it, GNU R runs it to completion if it ran the original to completion, and the engine still diverges with the same signature: the same crash signal, a timeout, or the first of errors, warnings, messages, stdout, value and the deep observer’s dump that differs. A difference in errors or warnings also keeps both engines’ messages, and a difference in the dump keeps the way its first expression differs (the table above), so a reduction seldom slides to another bug; a difference in printed output or the last value keeps only its channel, so read the reduced program against the original report. Both engines run with the observer on. For example, a segfault of ry-tier0 in a replacement on a matrix, buried in a 130-line fuzz program, reduces in about 300 candidates (a few minutes in a debug build) to:

f <- function(m) { m[1, 1] <- 0; m }
f(matrix())
f(matrix(0, 8, 8))

The command prints the reduced program, its signature and a replay command (with ORACLE_OBSERVE=1 when only the observer’s dump differs), and --output writes it. --budget caps the candidates (2000 by default), and the reduction is deterministic unless a candidate runs into the timeout. oracle fuzz --reduce starts no candidate after the time budget and leaves a timeout unreduced, because each candidate that still hangs costs the whole timeout. oracle::reduce::reduce takes any predicate, so its unit tests run without an engine.

A grammar only covers what its author thought of, so a second source of programs is real R code, mutated (oracle::corpus, oracle::mutate). The corpus is extracted at run time into target/fuzz-corpus and never committed: the examples of every base package’s help pages (tools::Rd2ex), the tests/*.R of the R source for the installed release (downloaded from CRAN once by oracle fuzz --kind corpus), the oracle cases, and the tests of corpus packages that ship them. Each file splits into snippets, runs of top-level expressions cut at blank lines, without the paragraphs that touch graphics, files, the clock, the environment, addresses or unseeded random numbers. A seed picks a snippet and applies one to four mutations on the syntax tree: boundary values for literals, materialized a:b, named, reversed or emptied c(), a statement wrapped in a closure, local, identity, a serialization round trip or eval(quote()), repeated calls, swapped, dropped or duplicated arguments, and neighboring operators. Every engine runs under the deep observer. The unmutated snippet runs first: an engine that already differs on it is a plain compatibility gap, listed in target/fuzz-corpus/base-differs.txt, and only a mutant that diverges while its snippet agrees is a finding. A diverging mutant runs under GNU R a second time and is dropped when GNU R disagrees with itself. just fuzz-corpus SEED COUNT ENGINES runs a range of seeds. The bounded pass, fuzz_corpus_mutants_never_crash, fails only when a mutant crashes ry and prints the other divergences, because the snippet a seed picks depends on the corpus on disk: it changes whenever an oracle case is added, and with the R source and the packages a machine has. A corpus seed therefore replays only against the same corpus, and a report carries the mutant’s text for that reason.

crates/oracle/tests/test_gallery.rs renders a seeded sample of the plot gallery under both engines and compares the images (Corpus and oracle). just gc-torture runs every oracle case on every core and takes about ten minutes on four cores in a debug build, and just gc-torture young runs them under ry-gctorture-young. just gc-torture repeat runs them twice in one process under seeded stress with the heap checked (ry-repeat-gctorture-seeded), takes about forty minutes on four cores in a debug build, and prints each case that takes over a minute. A case is killed after fifteen minutes for each pass it makes. The generated programs run under both torture engines. RY_GC_STRESS=1 stresses from the script’s start and RY_GC_STRESS=startup from the first allocation, which takes about seven seconds on an empty script in a debug build (Robustness lists the modes).

crates/frames/tests/test_debugger.rs drives the debug adapter (interp::debugger) in-process. Scripted sessions pin breakpoints, stepping, the stack, variables, evaluate, browser() and uncaught errors as transcripts, and a seeded simulator plays a client that sends random, sometimes invalid, requests at every stop. Every request must get exactly one response, the session must end, and the interpreter must be left clean. FUZZ_SEED=<seed> replays a failing session.

crates/oracle/known_failures.toml lists the oracle cases that fail on main under one engine, so a branch can tell its own regressions from failures it inherited. Each entry names the case (its file name in cases/ without .R), the engine as --engine names it, the area of ry the bug lives in (such as jit or gc), and a one-line reason; provisional = true marks a failure seen on a branch whose runtime matches main but not yet confirmed on main itself.

[[failure]]
case = "base__match_arg_native"
engine = "ry-jit"
area = "jit"
reason = "the compiled body warns twice where GNU R warns once"

oracle compare and cargo test -p oracle (with ry’s runtime built) sort every failure against the file and fail the run on two things: a failure the file does not list, and a listed case that passes. The second rule means the change that fixes a case also deletes its entry. Add an entry only for a failure main already has, never for one your own change introduces. A run prints each listed failure as known without its diff and ends with one line per engine:

ry-jit: 409 passed, 16 failed (16 known, 0 new), 0 skipped

A crash is never a known failure. When an engine is killed by a signal, the run prints the case as CRASHED and counts it as new even if the file lists it, because a crash is a memory-safety bug and blocks until it is fixed. Where it shows can depend on the host, since it depends on where the collector happens to run, which is why the gctorture engines (ry-gctorture, ry-gctorture-young, ry-gctorture-seeded, ry-repeat-gctorture-seeded and ry-jit-gctorture) exist. The file has no mark that lets a case pass on some runs and fail on others. A beta release requires the file to be empty.

A count is only trustworthy if it covers every case, so oracle compare also refuses to start when a case needs a package that is not installed or a locale that is not generated. A case names them in a leading # oracle: requires comment: packages by name (# oracle: requires knitr, rmarkdown) and a locale it switches to as # oracle: requires locale en_US.ISO-8859-1. The run lists each such case and exits 1. Install the packages into the corpus library with just corpus, which the oracle searches by itself (Corpus and oracle), and generate the locales with sudo tools/dev-setup.sh (or tools/oracle-locales.sh alone). For a deliberately partial local run, --allow-skips skips those cases and marks every summary line PARTIAL:

skip cases/datatable__fread.R: needs bit64
skip cases/packages__xts_keeps_source_rownames.R: needs xts
ok cases/arith__complex_math.R
ok cases/base__match_arg_native.R
PARTIAL ry: 2 passed, 0 failed (0 known, 0 new), 2 skipped (2 missing a package or locale)
PARTIAL: --allow-skips left out 2 cases, so these counts do not cover the suite

The oracle refuses a file whose entry names an unknown engine, a missing case, an unknown key, or the same case and engine twice. known_failures_are_valid in test_cases.rs checks the file without running R.

crates/table/tests/test_table.rs drives the engine from Rust. Every expected value that encodes data.table behavior was read from data.table under GNU R. Its differential fuzz generates random tables and queries and checks the native kernels against the Polars plans (query::execute_with_plans_only), which serve as the reference. FUZZ_ITERS raises the iteration count; run it in release, as FUZZ_ITERS=3000 cargo test-release -p table fuzz. The fuzz tables are small, so the kernels’ parallel paths get large-table tests of their own.

just bench-table <dir> [--rows 1e8 --groupby-only] runs the db-benchmark questions under data.table, as R code under ry run and on the engine from Rust, fails if any checksum differs from data.table’s, and rewrites the matching results table in development/design/r-runtime.md. --groupby-only, --joins-only and --updates-only limit a run to the group-by questions, the join questions, or the two := loops that write existing columns.

just bench-base [--rows 1e5,1e6 --only REGEX --memory] times base R’s own kernels (sorting, string formatting and regex, grouping, merge, match, matrix margins, cumulative sums, xts and zoo rolling operations) under ry run and GNU R, and prints the ratio ry / GNU R per kernel. A kernel whose checksum differs from GNU R’s is marked MISMATCH. With --memory every kernel runs in its own process, and the table adds both engines’ peak RSS. zoo and xts load from RY_BENCH_RLIB (default ~/R/library).

just bench-ci [--only k1,k3 --runs 2 --scale 1] runs fifteen kernels cut from a user’s profiles of a large package’s test suite (tools/bench/internal/ci/kernels.R) under GNU R and ry run. Each kernel is one code shape a profile ranked high: creating an S4 object with a validity method (k1), as.Date on a character scalar (k2), scalar calls of ifelse, isTRUE and unname (k3), an argument checker over list(...) (k4), %in% and which on a small vector (k5), an xts merge (k6), a data.table grown by rbind (k7), pmin/pmax on matrices (k8), and parse with source references (k9), readLines (k10) and a recursive list.files (k11) over forty generated source files, new() on a class of two plain slots (k12) and on a class whose prototype holds a million doubles (k13), forcing every object of three installed packages’ lazy-load databases (k14), and a grouped data.table j that calls a function (k15). The table shows the fastest of --runs runs on each engine and the speed-up, and a kernel whose checksum differs from GNU R’s is marked DIFFERS and fails the run. The packages (xts, zoo, data.table, purrr) load from R_LIBS, by default the corpus library that just bench-packages install fills. The file says which kernels follow code a user sent and which are reconstructions.

Compare peak RSS between two builds only from the same page-cache state. A freshly built .ryc or libR.so sits in the page cache in large folios, so one fault maps a whole block, and the same build reads back in small pages after eviction: two identical builds differed by 5 MB this way. python3 tools/bench/evict.py RY... evicts every file each build reads at run time, recursively: the executable’s directory, its base R tree with every package’s .ryc, ry’s cache directory, the homes R_HOME lists (a path list, so one call covers a separate home per build) and the R_LIBS libraries. tools/bench/bench.py and bench-base --memory run it before measuring. Run it before every peak-RSS run you script yourself, because evicting only the top of base-r leaves the .ryc images cached.

Judge a change’s peak RSS in three parts, because they behave differently. Anonymous memory and the start image’s resident pages must not grow. The resident pages of libR.so’s code are not judged per change. A plain release build has no function ordering, so where the linker happens to place hot functions decides which neighbouring code readahead and fault-around map in with them, and how far that moves depends on the change rather than its size. On the validation workload, a no-op exported function added to an otherwise unchanged build moved resident libR.so text by under 0.1 MB, while a change that grew .text by 5 KB moved it by +2.4 MB, with identical anonymous and image RSS. That change’s extra resident pages held code the workload never runs (the analysis core, polars, graphics): they were mapped because they sit next to hot code. Judge the size of libR.so’s .text section instead (size -A), which is deterministic, and expect it to grow only by about what the change adds. Resident text is checked on the shipped layout (tools/layout/optimize.sh) when a release is cut. tools/bench/bench.py prints each program’s peak in those parts (whole process, anonymous, start image and libR.so resident, sampled from /proc/<pid>/smaps every 20 ms) and the .text size of each build’s libR.so. With --baseline, it also prints the paired ratio baseline / ry for each program and its geometric mean with a 95% interval, so a value above 1 means the build under test is faster.

A change that allocates less can still raise anonymous peak RSS, because fewer collections change which heap pages fill before the next one. Such a rise is exempt only when all three of these hold, and the change’s description shows each:

  • RY_GC_LOG gives the same peak old-generation bytes (the last column) on both builds;
  • valgrind --tool=massif gives the same malloc peak, and with --pages-as-heap=yes the same mapped size;
  • a control that puts the missing allocations back, such as one throwaway c(1, 2) per loop iteration, closes the gap.

formatC()’s direct kernel call met all three: report’s VmHWM rose by about 1 MB, all of it in one GC arena, and the control closed it to about 0.1 MB.

Layout moves single-program wall time too, and with it callgrind’s simulated branch mispredictions and cache misses. Only the instruction count (Ir) holds still. So judge a regression on one program by Ir from valgrind --tool=callgrind --cache-sim=yes --branch-sim=yes on both builds: a rise in Ir is real. A wall regression, or a rise in simulated mispredictions or cache misses, without a rise in Ir blocks a change only when it reproduces on two layouts: the change’s build, and the change’s build plus a no-op exported function (the control), which shifts the code placed after it. On the public suite’s lapply program, one source built several ways gave:

libR layout Ir Indirect mispredictions Conditional mispredictions I1 misses
plain release 6,650.6M 8.6M 16.6M 69.4M
plain release + control 6,650.6M 10.6M 12.2M 68.3M
functions and blocks aligned 6,645.4M 10.6M 14.5M 23.4M
aligned + control 6,645.4M 8.6M 16.1M 27.8M
PGO and BOLT 5,256.3M 0.54M 29.2M 2.75M
PGO and BOLT + control, same profile 5,256.3M 0.55M 28.7M 2.74M
PGO and BOLT + control, own profile 5,256.3M 5.45M 24.2M 2.76M

Wall time can follow these counts: a build with 8.6M indirect mispredictions ran lapply about 25% slower than one with 0.6M at the same Ir. Two layouts were tried as a way to take the noise out of benchmark builds, and neither does. Aligning functions to 64 bytes and blocks to 32 (-C llvm-args=-align-all-functions=6 -C llvm-args=-align-all-nofallthru-blocks=5) grows libR.so’s .text by 24 MB (27%), and the control swings the counts as far as before. The shipped layout (tools/layout/optimize.sh, applied to libR for this table) holds still under the control only when both builds reuse one training profile. A build that trains its own profile, as every release does, gets a different function order and the swing returns. The builds that judge a change therefore stay plain release builds, and the control is the test for a layout effect. The published suites measure the shipped layout itself: they are recorded on the release archive (bench.py --release, see Benchmarks).

Two builds can share target/base-r in a comparison, because each loads only images its own name carries (ry-start.<build>.image, <package>.<build>.ryc, see the architecture page). A build’s first start writes its images into the base R tree, so tools/bench/bench.py starts each build once before timing anything. A package library gets code images only when a package is installed, so bench.py stops when a package in the corpus library or R_LIBS has only another build’s code image, and names the packages to reinstall. RY_IMAGE_LOG=1 makes ry print ry: image unused: <image> was written by another build of the runtime (or why else it skipped an image) on standard error, and ry: image written: ... when it writes a build’s images. bench.py sets it on every timed process and fails one that prints either, because that process timed a slow start. It refuses a runtime that predates the flag. Under --release, the archive extracts its runtime and base R into a cache directory of its own, so it loads the images it shipped, and bench.py reads its runtime from the R home it reports. two_builds_alternate_starts_without_displacing_each_others_images in crates/ry/tests/test_perf.rs pins both halves.

just bench-stdlib [filter] [n] times every List, Str, Dict and Result function at n = 1e6 by default against its fastest GNU R equivalent, and prints one row per function with the ratio ry / GNU R. The cases live in tools/bench/internal/stdlib: ry.R holds ry’s side and gnu.R the R alternatives in base R, purrr, stringr and stringi under the same case names, and the fastest alternative sets the bar. A callback function is compared with R’s callback idioms (vapply, Filter, Reduce, a for loop, purrr), and a function on one string with R’s vectorized string functions. Where ry’s input is a vector, the case also runs it on a List, ry’s unboxed vector, and the faster of the two counts. Each case runs in its own process, so a case slower than --timeout shows as timeout without stopping the rest, and a check column compares checksums of both results. Packages load from R_LIBS_USER, and an alternative whose package is missing is skipped and named in the row. --sort ratio puts the slowest functions first, and --write saves the table, as in tools/bench/internal/stdlib/results.md.

fuzz/ is a cargo-fuzz crate, in its own workspace and excluded from the main one. Each target is a thin wrapper around a battery its crate exports, which the crate’s in-tree harness runs too:

Target Battery
parse syntax::testing::check_parse_invariants
format format::check_format_invariants
semantics semantics::testing::check_semantics_input
ide ide::testing::check_ide_invariants
position ry::position::check_position_invariants
config ry::config::check_config_invariants
lsp the language-server simulator, run_session, reading its choices from the input
load_index, reload loader::testing::check_load_index_input, check_random_reload
sniff sniff::check_sniff_invariants
csv table::io::check_csv_bytes
interp_lower, interp_unserialize, bytecode interp::testing::check_lowering_input, check_unserialize_input, check_bytecode_agreement
interp_code_image, interp_code_image_file interp::testing::check_code_image_round_trip, check_code_image_file
unserialize R::serialize::check_unserialize_input

The semantics target derives its document kind and an incremental edit from the input bytes, so a single byte stream drives the full battery, the spliced reparse of the edit included. The generator-driven targets (lsp, reload, bytecode) read the generator’s choices from the input through syntax::testing::ByteStream, so a byte mutation is a mutation of the program or session.

Two features of fuzz/Cargo.toml select the targets. runtime adds the interpreter’s targets, and polars adds csv and unserialize, whose crates link Polars. Building Polars with coverage instrumentation needs more memory than everything else in the workspace: in a 15 GB container with four build jobs, the compiler is killed while it builds polars-core. On such a machine, cargo fuzz build --no-default-features --features runtime builds the other 15 targets, and --no-default-features alone builds the analysis targets.

A target that links the runtime’s heap runs without leak detection. LeakSanitizer does not scan mapped memory, which is where the heap keeps its objects, so memory that a live object owns would be reported as leaked. The heap crate turns the check off for a fuzz build (__lsan_default_options in crates/heap/src/space.rs); the analysis targets keep it, except reload and load_index. The loader parses ahead on a detached thread, and a leak check that runs while that thread exits reports the thread’s own bookkeeping, so those two targets turn the check off themselves.

interp_unserialize runs with a ceiling of 16 MB on one allocation (Heap::set_allocation_ceiling, set by check_unserialize_input). A valid stream can name more memory than it holds, because a compact sequence is three numbers and reads as a vector of any length. Under the ceiling such a stream fails with the error for a vector that does not fit, so the target’s memory is bounded by the size of its input.

The reader treats its input as hostile, because unserialize() and readRDS() take any file. It refuses a length that the rest of the input cannot hold, a byte-code reference outside its table, and an item of the wrong kind in a position that implies one: a symbol’s name, a string’s element, an attribute list or an attribute’s name, an environment, the name of a binding, of an argument or of a formal argument. Every such stream is pinned in crates/interp/tests/test_fuzz.rs, where one test asserts that each is refused and that every stream GNU R wrote for libR’s conformance cases still reads. That test runs under collection stress, since the damage a wrong kind did showed only at a later collection.

It needs nightly, which the repository’s pinned toolchain already is. Run cargo install cargo-fuzz, then just fuzz-run format runs one target, and just fuzz-all 120 seeds the corpora, builds every target and runs each for two minutes. A second argument names the features, so just fuzz-all 60 runtime leaves out the two Polars targets, and that is what the staged test.yml runs (Running one case).

The two minutes are wall clock, enforced by timeout, and a target that is interrupted at the bound has passed. libFuzzer’s own -max_total_time cannot bound the pass, because it starts counting only after the target has replayed its whole corpus, and seven targets are seeded with every fixture source in the repository (about 5,300 files). ide runs four to seven inputs a second, so its replay alone takes about 15 minutes. fuzz-all replays the seeds in random order (-prefer_small=0), so a target that cannot finish its corpus inside the bound reads a different sample on each run. For a run that mutates beyond the seeds on a slow target, use just fuzz-run with a long -max_total_time.

Without just, from fuzz/:

Terminal window
cargo -Zscript ../scripts/seed-fuzz-corpus.rs # seed from all fixture case sources
cargo fuzz run format -- -max_total_time=600 -print_final_stats=1
cargo fuzz run parse -- -max_total_time=600

The seeder copies every fixture case’s source in the repository into fuzz/corpus/{parse,format,semantics}/. Files are content-addressed, so running it again only adds. The corpus and the artifacts are gitignored, because a durable regression belongs in the REGRESSIONS battery or in a fixture case, not in the corpus.

When the fuzzer finds something, minimize it with cargo fuzz tmin format <artifact>, fix the bug, replay the artifact, and add the minimized input to REGRESSIONS, plus a golden fixture case when the shape deserves a pinned rendering. To judge corpus quality, you can get a coverage report (the pinned toolchain includes the llvm-tools component it needs) with cargo fuzz coverage format && cargo cov -- show, as the cargo-fuzz coverage guide describes. Deep runs are manual or scheduled work; the bounded in-tree batteries remain the gate in the default suite.

crates/loader/tests/test_reload_fuzz.rs generates packages from statements that define, wrap, read and rewrite a small pool of names, edits them at random, and checks that applying the reload plan to the loaded package gives what a fresh load of the edited package gives. A model executes both sides: a lazy binding’s value is its definition’s text, and a statement stores its text plus the values it read, so a statement that should have rerun shows up as a stale value. The same test checks that the incrementally derived index equals one built from scratch, and a second test feeds the index arbitrary bytes and asserts its positions stay inside the text. Removing either of the diff’s ordering rules turns it red within a few hundred seeds.

crates/ry/tests/test_cli.rs drives the real ry binary end to end. It covers:

  • diagnostic rendering, with 1-based character positions, snippet windows, and underlines;
  • JSON Lines output;
  • the exit-code contract;
  • finding the configuration, and failing to;
  • per-file # typing: directives and suppression comments;
  • NAMESPACE validation, and project stub overrides including their loader-problem reports;
  • data-masked evaluation;
  • the formatter’s --check mode, range formatting, and literate documents;
  • ry run and ry R without R: help output, usage errors, a missing script, and strict mode refusing a script before any R starts.

crates/ry/tests/test_command_line.rs property-tests and fuzzes argument parsing for ry run, ry R and the ry repl alias against a model: everything after the script reaches it verbatim, options resolve to the documented runtime and mode, and no argument vector panics the parser. test_repl_e2e.rs runs every entry point on both runtimes: script arguments, -i, piped consoles, and the earlier spellings (--runtime, RY_RUNTIME, ry repl).

crates/ry/tests/test_docs.rs runs the examples in the docs through the real ry binary, so a page that shows stale output fails the suite. A block opts in with a marker: the word ry after the fence’s language, then the command and its options. Expressive Code ignores the extra words, so the site shows a plain block, and the pages’ Markdown versions drop the marker. Here an output block follows a tab group:

<Tabs syncKey="annotation-style">
<TabItem label="Inline">
```r
greet <- function(name: character, greeting: character = "hello") -> character {
paste(greeting, name)
}
greet("Ada", greting = "hi")
```
</TabItem>
<TabItem label="Comments">
```r
#: fn(name: character, [greeting]: character) -> character
greet <- function(name, greeting = "hello") {
paste(greeting, name)
}
greet("Ada", greting = "hi")
```
</TabItem>
</Tabs>
```text ry check typing
type-mismatch
x this function has no parameter `greting`. Did you mean `greeting`?
```

A marked text block is the output of the nearest r block above it, or of every tab of the nearest tab group above it, and each tab must produce it on its own. Inside a <TabItem>, after the tab’s r block, it is the output of that tab alone, and a tab of the same group with no output block of its own must report no errors. The code runs in a throwaway project of one ry.toml and one file, main.R unless the output’s snippet names another file, as --[R/both.R:2:31] does. A tab group that no output block follows must give the same findings in every tab, with typing on. An unmarked output block is not tested.

Word Meaning
check / run Run ry check, or ry run main.R.
typing, strict Set that [check] key to true. Type checking is off unless a block says typing.
KEY=VALUE Any other [check] key, written as in ry.toml, such as object-systems=false.
runtime=ry / runtime=gnu-r Which runtime run uses; GNU R by default, as on the command line.
exact Compare the whole output instead of an excerpt.
continue Run the page’s earlier r blocks first, as one session, and compare only this block’s output. A check takes every earlier block; a run takes only those whose output the page checks, so an illustration that needs a package the machine lacks does not stop it.
tab=LABEL Only that tab produces this output, for the rare example whose spellings legitimately differ.
dput On an r block with #> output, print each result with dput() instead of print().

An r block can also hold its own output, as #> lines after the code that prints it, with a marker such as ```r ry run runtime=ry dput. Each run of code before a #> group shows its last value, and the comparison is always exact. A page’s #> blocks share one R process, each in its own local() environment, because a runtime start costs seconds and the standard library reference has over a hundred of them.

An excerpt matches when every non-blank line of it equals a line of the real output, both trimmed of surrounding whitespace, in the page’s order. Lines the page leaves out, such as the summary, a source snippet, or other findings, are free. A message the page wraps by hand matches one output line when its lines are joined with single spaces. This rule lets a page show the part of the output that matters while still pinning every word it does show. An exact block matches when the whole output agrees, ignoring trailing whitespace and the project’s temporary directory. Both compare standard output and standard error as one stream, interleaved as a terminal shows them, with color off.

just docs-test [PAGE] runs the pages whose path contains PAGE, and just docs-bless [PAGE] rewrites the output of exact and #> blocks from the real tool. An excerpt is curated by hand, so bless leaves it alone: the failure prints the full output to choose the lines from. A run block is skipped with a note when GNU R is missing, or for runtime=ry, when libR.so is not built beside ry. The landing page’s samples in docs/src/pages/index.astro are checked as excerpts with typing on, without markers.

crates/ry/tests/test_load.rs runs ry load and plain Rscript against a real GNU R and checks the loaded result: Collate order, imports, S3 and S4 registration, Depends, sysdata.rda, source references, reloads that rebind, rerun and remove, undoing deleted S4 definitions, the error locations, and the ry load --r-entry entry point. test_repl_e2e.rs checks that ry R loads the package it starts in and reloads an edit before the next command. Both skip without R.

The libr tests load libR.so the way an R host does, build packages from source with the system C and C++ compilers, load them with dyn.load (which runs R_init_<pkg>), and call their routines through .Call, all through the exported C API. The shared harness is crates/libr/tests/harness.rs. The packages are brio from the benchmark closure, vendored in tests/packages/brio, and tests/packages/rytest, written for these tests.

  • test_native.rs builds both against ry’s headers and checks the results, collection stress, the handle scope, PROTECT imbalance, errors unwinding through C and C++ frames, R_alloc release, finalizers, coercion, and the random number generator against GNU R’s output.
  • test_interrupts.rs checks that an interrupt raised while interrupts are suspended, from R or from C, stays pending until the next interrupt check. It loads base R, which changes how a C callable resolves for every later test in the session, so it runs in a process of its own.
  • test_gnu_r_binaries.rs builds both with GNU R’s R CMD SHLIB, against GNU R’s headers and linked to GNU R’s libR.so, and runs them under ry’s. It runs in a process of its own, so the two builds of one package never share a session.
  • test_headers.rs compiles every header as C11 and C++17 with warnings as errors, checks with the compiler that every declared name is exported, runs BLAS and LAPACK calls through the headers, and compares the graphics structs’ layout with GNU R’s. It also requires every name in tests/headers/r_api_names.txt (GNU R’s public header names that its R.dll exports, plus the internals packages link) to be declared by ry’s headers and exported by libR. With RY_GNU_R_INCLUDE and RY_GNU_R_EXPORTS pointing at a GNU R include/ and a list of its R.dll exports, api_names_cover_gnu_r checks the list against that release, and RY_UPDATE_API_NAMES=1 regenerates it.
  • test_altrep.rs and test_altrep_api.rs drive the altreptest package’s classes of every kind and base R’s own classes, against a report recorded from GNU R.
  • test_serialize.rs and test_serialize_native.rs read and write GNU R’s recorded streams byte for byte in every format and version, read installed lazy-load databases, and fuzz the reader.
  • test_connections.rs runs one connection scenario list in GNU R and in ry and compares the transcripts; test_connections_gnu_binary.rs repeats it with the ryconn package built by GNU R.
  • test_install.rs assembles ry’s R home and installs packages with R CMD INSTALL semantics. With RY_SOURCE_PACKAGES set to a directory of source tarballs, it builds each one against ry’s headers and reports the outcome.
  • test_corpus_binaries.rs loads every compiled package of the installed corpus (RY_CORPUS_LIBRARY, default .corpus/library) under ry, each in a child process with its dependencies, and writes target/tmp/corpus-binaries.tsv.
  • test_corpus_calls.rs replays native routine calls that tests/data/corpus_calls/record.R recorded under GNU R, reading arguments and results with ry’s unserializer, and requires every routine whose library loads to return what it returned under GNU R.

The two corpus suites pass without a corpus and report that they skipped.

These suites need a C and C++ toolchain, and GNU R for the second suite and the generator check. The parts that need GNU R find it through the R on PATH, taking its headers from R CMD config --cppflags and its Rscript from PATH, so Debian’s split layout and a source build both work. When no R is on PATH, or RY_SKIP_GNU_R=1 is set, those parts print why and skip.

crates/ry/tests/test_lsp.rs spawns the real ry server process and drives it over stdio with an async-lsp test client. It covers:

  • capability negotiation: position encodings, pull diagnostics, refresh, label offsets, and snippets;
  • document sync, both incremental and full-text, including untitled and non-file: buffers and re-reading from disk on close;
  • the two-wave push contract, its settled-superset invariant, and how a burst of edits settles under latest-edit-wins cancellation;
  • pull diagnostics: result ids, unchanged reports, retryable cancellation, and push suppression;
  • diagnostic refresh on save and on a configuration change;
  • configuration: live reload, a configuration in an ancestor directory, a broken configuration keeping the previous one in force, and the diagnostic on the configuration file itself;
  • every feature endpoint, including correct UTF-16 and UTF-8 ranges over BMP and non-BMP text, and safety for out-of-bounds positions;
  • semantic tokens for a #: body, and serving .Rtypes and NAMESPACE buffers.

The cancelled-pull test is made deterministic by a fault-injection seam. When the RY_TEST_DELAY_PULL_MS environment variable is set, the server announces each diagnostics pull by creating the RY_TEST_PULL_MARKER file, then holds the pull (for at most the delay) until the cancellation token flips. The test sends its didChange only after the marker appears, so the cancellation provably lands while the pull is in flight, and the retryable SERVER_CANCELLED response and the successful retry after it can be asserted without sleeping and hoping. Nothing else sets these variables.

crates/ry/tests/test_simulation.rs runs the in-process language-server simulator: random editor sessions under a seeded schedule, checked at random points against a server started fresh on the same disk and buffers. crates/ry/tests/test_scenarios.rs runs the scripted user journeys in crates/ry/tests/scenarios/. Robustness documents both, including the scenario format and the replay variables.

native_diagnostics_agree_with_the_typed_core in the same file holds the strict-mode native function findings to what the native compiler does. It checks each program of a corpus in strict mode, runs it with RY_TYPED_LOG=1, and fails when a function without a native or interpreted finding runs in the interpreter. The corpus is the typed benchmark programs, the friendly climb programs, and crates/ry/tests/native/*.R, where a program’s header states what it shows: # blocked: f, g names the functions that must get a finding, and # boundary: h the ones with an R type in their signature. Add a program there for a construct the compiler gains or the checker starts to report. The test also fails when no natively running function of the corpus calls a standard library function that semantics::native::COMPILED_STD lists as compiled.

A fixture checks what an edit produces, but not how much work it took to get there. crates/semantics/tests/test_incremental.rs checks the work: semantics::testing::ProbeDatabase logs every query execution, and checked_since names the items whose item_check ran. The tests pin these contracts:

  • A body edit that keeps the item’s scheme re-checks the edited item alone.
  • A body edit that changes the scheme also re-checks the items that read it.
  • A new definition re-checks only the items that read its name.
  • An inserted statement re-checks only the reads it intercepts.
  • A body edit re-derives one item’s load facts and its own file’s index layer, a NAMESPACE edit re-derives no item, and a statement that only moved keeps its load facts.
  • An edit parses one file. A cold check parses each file once, also on a diagnostics-only host whose store keeps two trees (ProbeDatabase::diagnostics_only), and that host reports exactly what an editor host reports.
  • A statement added beside a recursive function checks only itself, and a name a script adds resolves no package item’s names again.
  • A check resolves the names it spells through item_links without a frame_binder query, and finds its cyclic groups with one file_interface_groups walk per file.
  • Under invariants, the links agree with the per-name answers, and each file’s groups agree with plain reachability from the item, which shares only the edges with the component walk (checked in a child process, because invariants are process-wide and would perturb the counting tests).

The same file checks that a chain of re-exports resolves to its base, that a cycle of them pins to Unknown, and that a chain of 4,096 links resolves without overflowing the analysis thread’s stack. In the IDE harness, point_queries_recheck_only_what_an_edit_changed holds the features that run on every keystroke to the same rule.

crates/semantics/tests/test_perf.rs asserts performance budgets over the real-file corpus. It is ignored by default, because it needs the fetched corpus and a release build, and CI does not fetch the corpus:

Terminal window
cargo test-release -p semantics --test test_perf -- --ignored --nocapture

just perf runs the same command. Its budgets are measured numbers plus headroom, and cover the cold-pass wall time per line, resolve steps per line (the tripwire for a regression in resolve memoization), resident bytes per line, and the p50 and p95 latency from a keystroke to fresh diagnostics on the corpus’s largest package. It prints what it measured, so it doubles as the corpus-wide measurement, while ry debug analysis-stats diagnoses one workspace phase by phase.

Run optimized tests with cargo test-release, an alias for cargo test --profile iterate, which is what just perf uses. The iterate profile keeps opt-level = 3 but drops cross-crate LTO and uses 16 codegen units (Build profiles). Plain cargo test --release builds with the release profile’s fat LTO: building libr’s test_linpack that way peaks at 11.1 GiB of RSS and takes 30 minutes on four cores (it was OOM-killed once on a 16 GB machine), against 3.6 GiB and 11 minutes without it. A test that needs another package’s binary (libR.so, ry-rcmd) builds it with syntax::testing::cargo_build, a cargo build in the profile the test runs in, so it lands beside that ry. The helper drops the variables cargo sets for the test (CARGO_PKG_NAME, OUT_DIR and so on): build scripts such as ring’s rerun when those change, so a nested cargo build that inherits them rebuilds most of the dependency tree, and the next plain build rebuilds it back. A budget measured under cargo test-release can differ from a release binary’s, so a number you quote comes from a release build. Debug builds keep line tables only ([profile.dev] debug = "line-tables-only"), which keeps backtraces symbolized; set CARGO_PROFILE_DEV_DEBUG=true to inspect locals in a debugger.

crates/ry/tests/test_perf.rs holds the memory budget of a whole cold ry check --no-cache with type errors on. It generates the three 300,000-line benchmark code bases with tools/bench/tooling/generate.py, runs the release binary on each, and fails when one peaks above 256 MiB resident. It reads the child’s own peak with wait4, so it measures the process the user would run, allocator included. It is ignored by default for the same reasons:

Terminal window
cargo test-release -p ry-lang --test test_perf -- --ignored --nocapture

just perf runs both suites.

The hot suite: a runtime change measured in seconds

Section titled “The hot suite: a runtime change measured in seconds”

The three benchmark code bases take 60 to 135 s each, too long to run after every change to the runtime. The hot suite (tools/bench/hot/) replays the package calls that hold most of their time and one round of it, on both engines, takes about 35 s:

Terminal window
just bench hot --rounds 3 --extra-rounds 0 # every program, both engines
just bench hot --rounds 3 --extra-rounds 0 --filter rank
| program | GNU R | ry | GNU R / ry | flags |
| --- | ---: | ---: | ---: | --- |
| testthat::expect_equal | 20.369 ms | 14.012 ms | 1.68x (1.45 to 1.84) | wide_interval (ry) |
| base::rank | 14.323 ms | 22.401 ms | 0.64x (0.64 to 0.71) | wide_interval (ry) |
...
Geometric mean of GNU R / ry over 100 programs: 1.43x (95% interval 1.39 to 1.49)
Predicted GNU R / ry for finbench: 1.27x (95% interval 1.20 to 1.38; 83 programs carrying 100% of its weight)
Predicted GNU R / ry for dsbench: 1.39x (95% interval 1.29 to 1.46; 78 programs carrying 100% of its weight)
Predicted GNU R / ry for biobench: 1.56x (95% interval 1.37 to 1.63; 77 programs carrying 100% of its weight)

A program is one package entry: a function that the code bases’ own code calls in a package, such as stats::lm or zoo::rollapply, together with up to three of the calls they made to it. Each call was captured with its arguments while the code bases’ tests ran, and the harness (harness.R after replay.R) replays it as eval(call, env) in a loop. One process per engine runs every program, so the packages load once. A program’s batch holds about 20 ms of GNU R time, and its time is the median of five batches after one warm-up batch.

The number to watch is the predicted speed-up of each code base, not the geometric mean. Each program carries a weight per code base, its share of the code base’s GNU R time among the suite’s programs, and the prediction is the weighted harmonic mean of the programs’ speed-ups, which is what the code base’s own speed-up would be if it ran only these calls.

just bench-hot-capture DIR regenerates suite.toml and samples/ from the code bases that tools/bench/tooling/generate.py wrote to DIR. It runs the three steps of tools/bench/tooling/hot.py:

Step Runs Writes
capture each code base’s tests on ry with the recorder on and the JIT off per code base, the time in every package entry (entries.tsv, summary.tsv), up to eight calls of each with their arguments (samples.tsv, samples/N.rds), and the packages the code base imports (session.txt)
validate every sample under GNU R (validate.R) the samples that replay with the result they had and no side effect outside their temporary directory, with GNU R’s seconds per call and whether each call needs a fresh copy of the arguments (validated.tsv)
select nothing the 100 entries that hold the most time, the three most typical valid samples of each, and their weights

The recorder is part of the runtime (crates/interp/src/capture.rs) and is off unless RY_CAPTURE_PACKAGE names the user’s code:

Variable Meaning
RY_CAPTURE_PACKAGE comma-separated namespaces that count as the user’s code, or R_GlobalEnv
RY_CAPTURE_DIR the directory the recorder writes to
RY_CAPTURE_SAMPLES calls kept per entry (default 8), drawn uniformly from all of the entry’s calls
RY_CAPTURE_MAX_BYTES the largest serialized argument set or result kept (default 65536)

It splits the run’s time between the user’s code and the package entries it calls. Code is the user’s when it is a closure of the user’s namespace, and also when it evaluates in an environment whose top-level environment is the user’s, however it got there: a promise for an argument, a block passed to test_that() or withr::with_tempdir(), an eval() from a package, or an evaluation from C. A package entry therefore owns the time of the package code under it but not the time of a user block it runs, and the calls that block makes are entries of their own. The capture of the finance code base reads:

finbench: 9592 tests, 9592 passing in 152.7 s of test blocks
147.5 s on ry: 3% in the code base's own code, 83% in package entries it captured, 13% in entries it could not capture, 1% outside both; 355234 calls into 198 entries

A call cannot be captured when a replay could not rebuild it: it passes its caller’s ..., an argument names a user function or a local closure, the call reads its caller’s frame, or the recorder failed to inspect it (the reason uninspectable, for example when forcing a lazily bound package definition throws). The recorder counts its own failure as this reason and the program carries on. Such a call’s time counts for its entry all the same, in the uncapturable_* columns.

The samples of an entry are a reservoir of RY_CAPTURE_SAMPLES slots drawn uniformly from all of its capturable calls. The recorder serializes only the calls the draw selects. A selected call whose arguments serialize to more than RY_CAPTURE_MAX_BYTES empties its slot rather than leave the earlier sample there, because keeping it would bias the draw toward small calls, and the oversize column counts these calls. A slot a later draw refills holds a sample again, so samples counts the slots holding one at the end. A result over the limit is dropped and its sample is kept, so validation checks that sample only against GNU R’s own two replays.

The recorder writes every time as an integer number of nanoseconds, with the unit in the column name: nanoseconds and uncapturable_nanoseconds in entries.tsv, nanoseconds in samples.tsv, and total_nanoseconds, user_nanoseconds, covered_nanoseconds, uncapturable_nanoseconds and outside_nanoseconds in summary.tsv. Its own work (inspecting a call, serializing a sample, writing a file) is left out of every record’s time and of the total.

The validator then drops the samples that depend on the world outside the call (a file the capture read, a git repository, external pointers, options, a graphics device) or that print. select reports what the suite loses to both:

finbench: the programs hold 79.6% of the run's time on ry (74.4% in calls the recorder could capture) and 82.6% of the package time
lost, with no valid sample: NlcOptim::solnl (2.41%), openxlsx::saveWorkbook (2.20%), R_GlobalEnv::test_that (1.61%), ...

A program’s weight comes from a ratio estimate, because the capture measures ry’s time and the weight needs GNU R’s. The time a code base spent in an entry is scaled by GNU R’s seconds over the capture’s seconds for the same sampled calls. The samples are a uniform draw of the entry’s calls, so what the switched-off JIT adds to the entry’s time cancels in that ratio.

  • The session. Every replay first loads every package the code bases import (session in suite.toml) and sets testthat’s third edition, because loading a package changes what other calls do: zoo::rollapply on an xts series pads with NA once xts is loaded, and ompr finds a solver only after ROI.plugin.highs is. A package that is not installed is skipped and named in the output.

  • Conditions. A call runs under handlers that continue a testthat expectation and muffle messages and warnings, so an expectation replays outside a test_that() block.

  • Random numbers. A sample that draws random numbers gets its seed restored before each call.

  • Temporary paths. A path the capture made under its session’s temporary directory (/tmp/RtmpXXXXXX/...) is rewritten to lie under the replay’s own tempdir() by replay.R’s relocate, which the validator and the harness both call, and the parent directory of each such path in the call’s arguments is created. The calls run in a directory under tempdir(), so a relative path lands there too. A sample may create files only under its temporary directory, and a second call must leave the same set of paths as the first; the validator drops any other sample with its reason, and drops a sample that opens a graphics device.

  • Fresh arguments. The validator replays each sample once with the JIT off and compares its variables before and after. A sample whose call changed them is marked fresh: data.table’s := modifies its table in place, an R6 setter such as lgr’s $set_appenders its object, foreach(i = ...) %do% assigns i in the calling environment, and httr2’s resp_body_json fills the response’s cache. The harness rebuilds a fresh sample’s environment from a serialized copy for every call. It builds them before each batch, outside the timing and outside the program’s region, so a batch times only the calls.

  • Checksums. Each engine prints a checksum of every sample’s last result, and the run fails when ry’s differs from GNU R’s. The checksum is <kind>:<length>:<hash>:<magnitude>:<balance>. The hash covers what must match exactly: every type and length, the attributes (all but data.table’s .internal.selfref, at any depth), the text, the logical values, and how many NA, NaN, Inf and -Inf each numeric vector holds. Every numeric vector at any depth adds the logarithm of its weighted size, and a sum under a second set of weights over that size, to the last two fields, which must agree to an absolute 1e-10 (or a relative 1e-13). A root that uniroot finds one part in 1e13 away therefore agrees, and a change that moves a vector’s sums by one part in 1e10 does not, so one wrong element of a long vector shows only when it is that large a part of the whole. A sample whose result differs on every call under GNU R (a temporary file name, a timestamp) is recorded as varies and not compared. A result of another type fails even when its values agree. An engine whose seq.int(1, 18, by = 1) is a double vector, where GNU R’s is an integer vector, prints:

    FAILED hot/base-seq on ry: checksum dbl:18:32x252214378:2.131601681765849e+01:1.626932262278126e+00,...,
    GNU R printed int:18:33x270404436:2.131601681765849e+01:1.626932262278126e+00,...

Two passes run on ry alone after the timed rounds, and --rounds 0 runs them by themselves. Both rely on .ry_bench_region(name), a builtin that ends the open benchmark region and starts the next, which the harness calls around each program’s batches.

--breakdown attributes the runtime’s profiler samples to the region they fall in and prints, per program and weighted per code base, the share of ry’s time in each activity:

| program | samples | interpreter | builtin | call_setup | native | evaluator | s3_dispatch | compiled | jit_compile | ...
| testthat::expect_equal | 429 | 48% | 29% | 12% | 2% | 4% | 2% | 0% | 0% | ...
| base::mean | 575 | 20% | 19% | 10% | 0% | 9% | 12% | 4% | 25% | ...
| finbench, weighted by GNU R time | | 41% | 32% | 10% | 8% | 2% | 4% | 0% | 0% | ...
| dsbench, weighted by GNU R time | | 41% | 27% | 11% | 14% | 3% | 2% | 0% | 1% | ...
| biobench, weighted by GNU R time | | 37% | 32% | 11% | 9% | 4% | 4% | 0% | 1% | ...

After timed rounds each program weighs by its share of the code base’s ry time. A run that times nothing (--rounds 0, as here) has no speed-ups to derive that share from, so it weighs by the GNU R weight and the row says so.

interpreter is the bytecode interpreter, builtin base R’s builtins, call_setup a closure call’s argument matching, promises and frame, native code reached through .Call, .External or .C, compiled machine code the JIT produced and jit_compile the JIT compiling it. A program with a large interpreter share gains from the JIT, and one with a large builtin share gains only from a faster builtin, so the breakdown bounds what an optimization of one activity can give. A compilation that falls inside a program’s batches counts in full, as it does in the instruction counts, which is why a short program such as base::mean shows a quarter of its samples there.

--instructions runs one batch of each program under callgrind, with instrumentation on only inside the program’s region, and prints each program’s instruction count per call. Callgrind writes a dump each time a region ends, and the dumps of one region are summed.

| program | calls | ry, batch | per call |
| testthat::expect_equal | 20 | 45,452,106 | 2,272,605 |
| openxlsx::write.xlsx | 3 | 85,865,843 | 28,621,948 |
...
Weighted, finbench, ry: 3,292,951,540 instructions per second of GNU R work
Weighted, dsbench, ry: 2,572,317,861 instructions per second of GNU R work
Weighted, biobench, ry: 2,613,663,739 instructions per second of GNU R work

The weighted total of a code base is the number to compare between two builds. It is the instructions ry spends per second of GNU R time in that code base’s programs: each program’s instructions per call, over GNU R’s seconds per call, weighted like the predicted speed-up.

The pass deals the programs over four harness processes that run side by side, and the whole suite takes about 80 s. The split is by position in the suite and not by the machine’s core count, so a program shares its process with the same programs on every machine. Each process runs on one OpenMP thread, because callgrind counts the instructions idle worker threads spend spinning, which vary from run to run: a grouped data.table assignment counted 80 to 96 million instructions on four threads and 45.8 million on one.

Three options build on the counts:

  • --save-instructions FILE keeps them, and a later run with --baseline-instructions FILE prints each program’s change and its effect on each code base’s weighted total, largest effect first.
  • --jit-off counts ry with RY_JIT=off as well and prints the same comparison between the two, which shows what the JIT costs or saves each program.
  • --functions N adds the N functions of the runtime and of the packages’ compiled code that hold the most of each code base’s weighted instructions. It needs a build with symbols (cargo build --profile profiling, passed with --ry PATH --any-build).
  • --save-dumps DIR keeps each program’s callgrind dumps as DIR/<program>.<n>.callgrind, for a question the function ranking does not answer, such as who calls a function or how often.

just bench-mechanisms prices the mechanisms the suite’s programs are made of. Each of about seventy operations is one expression (a closure call with a default argument, a UseMethod dispatch, a tryCatch that catches nothing, plain$b) that a compiled loop evaluates. The tool runs the loop under callgrind for 20,000 and for 40,000 iterations and reports the difference per iteration, less the empty loop’s, so start-up and warm-up cancel and the count repeats exactly. The tier-1 JIT is off unless --jit is given, because package code runs in tier 0 almost all of the time.

Terminal window
just bench-mechanisms --ry target/release/ry --filter dispatch
| operation | expression | instructions |
| --- | --- | ---: |
| dispatch: `UseMethod`, class method | `describe(object)` | 6,437 |
| dispatch: `UseMethod`, default method | `describe(i)` | 7,067 |
| dispatch: `UseMethod` then `NextMethod` | `describe(child)` | 14,104 |
| dispatch: builtin to a method (`length`) | `length(object)` | 3,768 |
...

The whole table takes about five minutes on four cores, and one filtered group well under a minute. --save FILE keeps the counts and --baseline FILE prints the change against them, which is how a change to the call path or to dispatch is checked before the suite is run. --profile NAME prints the functions that hold one operation’s instructions, on a build with symbols.

How well the suite predicts the code bases

Section titled “How well the suite predicts the code bases”

just bench-hot-track DIR NAME=RY[,VARIABLE=VALUE...]... measures it. It times the suite and the code bases’ tests on several ry configurations (other builds, or one build with a feature off) and compares the change the suite predicts with the change the code bases show, each against the first configuration:

Terminal window
just bench-hot-track .bench/codebases head=target/release/ry nojit=target/release/ry,RY_JIT=off r043=/tmp/ry-0.4.3/ry
dsbench (time over GNU R's; change against head; ...)
| configuration | predicted | measured | predicted change | measured change | weight | tests |
| head | 0.717 | 0.596 | +0.0% | +0.0% | 100% | 9416 tests, 9416 passing |
| nojit | 0.693 | 0.553 | -3.3% | -7.2% | 100% | 9416 tests, 9416 passing |
| r043 | 1.180 | 0.945 | +64.7% | +58.4% | 100% | 9416 tests, 9416 passing |
...
dsbench: r = 0.986 between the predicted and the measured log changes of 11 configurations; a predicted change shows up 0.85 times in the measurement

A study of twelve configurations gave these correlations. The configurations were the build the suite was generated on, that build with the JIT off, with the builtin fast paths off and with native methods off, the same build a second time, the build before the recorder, and six releases back to 0.4.0-beta, whose predicted times range from 13% under the first configuration’s to 124% over it.

Code base r A predicted change shows up
finbench 0.987 0.78 times
dsbench 0.986 0.85 times
biobench 0.987 0.90 times
all three 0.976 0.85 times

The three code bases share the configurations, so the last row pools 33 pairs that come from eleven changes.

The suite therefore ranks builds the way the code bases do, with three limits.

  • A predicted change is larger than the measured one, because the programs hold 77 to 80% of a run’s time on ry and the rest of the run changes less.
  • The predicted level is pessimistic. The suite predicted 1.34x, 1.39x and 1.47x for the first configuration where the code bases measured 1.62x, 1.68x and 1.62x. What makes the difference is not established, so compare predictions with each other and not with a code base’s time.
  • Wall time on a shared machine cannot show a small change. The study ran the same build twice, 100 minutes apart, and the second run was 14 to 16% slower on the code bases and 21 to 27% slower in the suite, because the machine’s speed drifted. Within one run, one program’s median batch time varies by about 20% between identical runs on either engine. Time two builds in the same rounds (--baseline PATH) and read the predicted speed-ups, not a single program’s ratio. For a change under about 10%, compare instruction counts.

Two instruction passes over the same build agree to 0.01% on each code base’s weighted total (finbench 3,294,004,993 and 3,293,995,120). The median program differs by 0.013% and the worst, brio::write_lines, by 0.41%, so a change of 1% in one program is visible.

A change in the suite’s count shows up in a code base’s count with the same sign. The whole test run of dsbench under callgrind, which takes about 65 minutes, cost 318.11 billion instructions, 317.45 billion with the JIT off and 322.91 billion with the builtin fast paths off:

Change Suite’s total at dsbench’s weights dsbench’s tests under callgrind
JIT off -0.27% -0.21%
Builtin fast paths off +2.95% +1.51%

The code base’s count covers the whole process, with start-up and package loading, and each configuration ran once, so how closely two runs of one build agree is not measured.

By default the runner adds ten rounds when any program’s interval is wider than 10% either way, which on a shared machine is every time, so pass --extra-rounds 0 for a quick look.

The testthat workload (tools/bench/applications/testthat.R) gives two different wall times, and they answer different questions:

Number What it covers Command
Test phase one warm run of the tests, the median of RY_BENCH_REPEAT runs field 3 of the output of RY_JIT=on RY_BENCH_REPEAT=5 ry run --runtime ry testthat_full.R
Process wall start-up, loading testthat, one cold run and five warm runs time around the same command
Load start-up and loading testthat only time ry run --runtime ry load.R, where load.R holds only suppressPackageStartupMessages(library(testthat))

testthat_full.R is harness.R followed by testthat.R, and its checksum is 27:22821. Both builds use the same R_LIBS, a library that ry itself installed testthat into, and each number is the median of five fresh processes. On the Linux release build with the JIT on, the test phase measured 0.219 s and the process wall 1.44 s, which is how one workload can be quoted as both. The comment at the top of tools/bench/harness.R gives the full environment of each command. Benchmarks covers the published suites and how they are timed.

  • Add or tighten a fixture before writing a parser-local or crate-local unit test.
  • A new phase or module gets a fixture suite before it relies on ad hoc unit tests.
  • Use the lightest fixture change that captures the failing shape.
  • Change a fixture expectation deliberately, never as a side effect.

A test must pin a contract the docs state or a bug that once happened. The suite is judged by the bugs it can catch, not by its size, so a test that cannot fail for a plausible bug is deleted rather than kept for safety.

Keep:

  • fixture suites, oracle cases, scenarios, simulations and every fuzz, property or differential harness, with their pinned regression inputs;
  • a regression test for a real bug, even a narrow one, when no fixture or harness would catch the bug coming back;
  • a test of behavior a user or another crate can observe, which no fixture, oracle case or harness already covers.

Cut:

  • a test that restates the implementation: it asserts a private data layout, an internal constant, or the exact steps the code takes, and so breaks on a refactor that keeps the behavior;
  • trivia: a constructor returns what it was given, a getter reads a field, Debug or Display of an internal type has a certain spelling, a default is its default;
  • a duplicate: the same behavior is already pinned by a fixture, a scenario, a harness, or another unit test one line over, or by an oracle case that an automated run executes ry against. Neither a plain cargo test nor CI compares ry’s output with the recorded oracle cases: they replay the recordings under GNU R, and the collection-stress pass compares ry only with itself, so an oracle case alone does not cover a test of ry’s behavior;
  • a test of the test harness, unless the harness’s own failure would let real failures pass (a filter that matches nothing, a duplicate case name);
  • a snapshot of incidental output that no contract promises;
  • a slow test that finds nothing a faster one would not.

Fold before deleting. When a group of small unit tests covers one behavior, one fixture case, one oracle case or one table-driven test replaces them. The only coverage of a behavior is never dropped: it moves to the cheapest form that still catches the bug.