Testing
Most of ry is tested with fixtures: plain-text files that pair a piece of R with the output ry should produce for it. A fixture reads well in a diff, and a new case takes seconds to add. Reach for an ordinary Rust test only when the behavior is awkward to express as rendered output.
A fixture suite describes the desired semantics, not what the implementation happens to pass.
Record a known gap explicitly, in .agents/memory/backlog.md or as a case pinned to the
wrong-but-current output with a comment that says so, because an unrecorded gap looks intentional to
the next reader.
The fast loop
Section titled “The fast loop”Start every command with source tools/env.sh &&. Where nix is installed, which includes every
Claude cloud container, it loads the flake’s default dev shell: the Rust toolchain, the reference
GNU R, just and mold, the same versions CI tests with. It then selects mold for Linux links
(through RUSTC_WRAPPER=tools/mold-rustc, which cargo leaves out of its fingerprints) and reports
whether ry’s base R (target/base-r) is built:
$ source tools/env.shry env: nix dev shell; GNU R 4.6.2 at /nix/store/...-gnu-r-4.6.2/bin; base R built; mold linker| Step | Time on a 4-core cloud container |
|---|---|
First source tools/env.sh in a fresh container |
98 s, 8.8 GB downloaded into /nix/store, nothing built |
Every later source tools/env.sh |
0.06 s |
nix develop -c <command>, for comparison |
4.7 s per command |
First debug build of ry, libR and ry-rcmd in the shell |
5 min 20 s |
The script saves nix print-dev-env in target/nix-dev-env.sh and sources that file, which is why
later runs are fast; it rewrites the file when flake.nix, flake.lock, rust-toolchain.toml or
the GNU R pin in tools/base-r is newer. Binaries built in the shell link nix’s glibc, so do not mix shell and host builds in one
target directory: cargo rebuilds everything when you switch.
RY_ENV=system source tools/env.sh skips nix and uses the host’s tools, with the reference GNU R
from /opt/R/<version>/bin. Use it for the package corpus (just bench-packages), whose packages
link the Ubuntu libraries that sudo tools/dev-setup.sh installs and the shell does not carry.
Without nix the script behaves the same way.
Every command below is plain cargo, so it works without just. A passing run prints only cargo’s
summary, and a failure prints a rerun command for just the failing case.
Edited X, run Y
Section titled “Edited X, run Y”The time is from touching one source file of the crate to the result, on 4 cores in a debug build
with mold, GNU R on PATH and base R built.
| You edited | Run | Time |
|---|---|---|
the parser (syntax) |
cargo test -p syntax --test test_syntax_fixtures; the whole crate, cargo test -p syntax, takes 18 s, 15 of them in fuzz_edit_stream |
1 s |
the checker (semantics) |
cargo test -p semantics --test test_typing_fixtures |
10 s |
the formatter (format) |
cargo test -p format --test test_format_fixtures |
1 s |
an editor feature (ide) |
cargo test -p ide --test test_ide_fixtures |
4 s |
| the interpreter, to run R | cargo build -p ry-lang -p libr && target/debug/ry run --runtime ry x.R |
13 s |
| the interpreter, its tests | cargo test -p interp --test test_<area> |
4 s |
| the interpreter, against GNU R | cargo run -p oracle -- compare crates/oracle/cases/<case>.R |
10 s |
| the interpreter, every oracle case | cargo test -p oracle --test test_cases ry_matches_recorded_outcomes |
10 min |
the typed core (typed) |
cargo test -p typed |
4 s |
the CLI or the server (ry) |
cargo test -p ry-lang --test test_cli, or one journey: SCENARIO=<name> cargo test -p ry-lang --test test_scenarios scenarios_hold |
6 s |
cargo build -p ry-lang does not build the runtime: ry loads libR.so at run time, so an
interpreter edit needs -p libr too. Tests and oracle compare that run ry’s runtime build both
themselves (a no-op build takes 0.4 s), so they never run a stale libR.so.
Environment variables
Section titled “Environment variables”| Variable | Read by | Effect |
|---|---|---|
FIXTURE_FILTER |
every fixture suite | runs the cases whose group__case id contains the value (pipes__ runs a group) |
RY_BLESS=1 |
fixture suites, test_docs, oracle test_cases |
rewrites the expected output from the actual output |
FUZZ_SEED, FUZZ_ITERS, FUZZ_DEPTH |
every seeded fuzzer and simulator | replays one seed; sets the iteration count; multiplies the default count |
SCENARIO |
test_scenarios scenarios_hold |
runs the one scenario of that name, and fails if it skips |
RY_TEST_WORKERS |
fixture suites, oracle, scenarios, test_format_range |
the worker count; failures print the count the run used |
DOCS_PAGE |
test_docs |
runs the examples of pages whose path contains the value |
ORACLE_ENGINE |
oracle test_cases |
compares those engines (comma separated: ry, ry-jit, ry-gctorture, …) instead of ry-tier0,ry-jit, and fails if base R is not built or is another release than the recordings’ |
ORACLE_RY |
the oracle | the ry binary to run, which the oracle then does not build |
ORACLE_R, ORACLE_RSCRIPT |
the oracle | the GNU R to compare with (default: R and Rscript on PATH) |
RY_BASE_R |
ry |
the base R tree; a debug ry otherwise finds target/base-r |
R_HOME |
ry, scenarios |
a GNU R installation whose base R ry emulates |
RY_LIBR |
ry |
the libR.so to load instead of the one beside the binary |
R_LIBS, RY_CORPUS_DIR |
oracle, scenarios | the package library cases and journeys load from (default: .corpus/library) |
RY_SKIP_GNU_R=1 |
libr tests | skips the parts that need GNU R even when it is installed |
RUSTC_WRAPPER |
cargo | tools/mold-rustc links Linux targets with mold; tools/env.sh sets it on Linux |
A test that cannot run here (no GNU R, no base R, a missing package) passes and says why on
standard error, which cargo shows only for a failing test or with -- --nocapture. Stable Rust has
no way for a test to report itself skipped at run time, so a green run of the default oracle test
or of the scenarios may have checked less than it looks. A run you ask for by name does not pass
that way: SCENARIO=<name> fails when the scenario skips, and ORACLE_ENGINE=<engine> fails when
the engine is ry and base R is not built.
The parallel harnesses deal items out statically, so worker w of n takes items w, w + n,
and so on. A fixture suite reuses one analysis database per worker, so a failure that depends on
what rendered before it on the same worker replays with the whole suite and the same count, which
the failure message prints as RY_TEST_WORKERS=<n>.
What is slow and why
Section titled “What is slow and why”Whole test binaries, run only (already built), 4 cores, debug, GNU R on PATH:
| Binary | Time | Why |
|---|---|---|
jit test_tiers |
160 s | four seeded fuzzers (150 to 400 seeds each) run 115 to 125 s apiece in parallel; the binary is CPU-bound |
ry-lang test_scenarios |
120 s | 80 journeys and the typed benchmark programs, each spawning ry processes; parallel inside each test, CPU-bound |
oracle test_cases |
180 s with GNU R, 25 s without | every case under GNU R against its recording, and the collection-stress replays; without GNU R, 5 s of it is the 24-case ry sample and the rest the three stress replays |
frames test_standard_library |
82 s | standard_library_at_runtime evaluates every case in one process: the interpreter is a process-wide singleton, so it cannot run in parallel |
table test_table |
42 s | engine model fuzzers |
format test_format_range |
35 s | the range-format battery over every fixture and corpus source, in parallel |
cargo run -p oracle -- compare runs the whole oracle in 118 s against the recorded outcomes and
184 s against a live GNU R, one case per core, printing only the cases that need attention.
Fixture format
Section titled “Fixture format”Every suite in the shipping crates runs through one harness, syntax::testing::run_fixture_suite,
which syntax, semantics, format, and ide all share. A .test file holds groups of cases:
#==== group_name#---- case_name<input source>#++++<expected rendered output>group__caseis the case’s stable identity. It must be unique across the suite, and a duplicate is rejected rather than allowed to shadow the other case silently.- What the expected output shows (schemes, diagnostics, a tree dump, hover text) is up to the suite’s runner. Behavior specific to one suite belongs in its runner, never in the fixture syntax.
- A typing or IDE case may hold several files. Each
#~~~~ <path>line starts one (seesyntax::testing::split_files), and a path underR/is a package file while anything else is a script. A typing case renders each file’s output under an== <path>line. - The runner loads only
.testfiles, so anything else in a suite directory is ignored.
Running one case
Section titled “Running one case”To run a single case, set FIXTURE_FILTER and pick the suite’s test target. The filter selects
every case whose group__case id contains it, so pipes__ runs a whole group:
FIXTURE_FILTER=group__case cargo test -p semantics --test test_typing_fixturesFIXTURE_FILTER=group__case cargo test -p format --test test_format_fixturesFIXTURE_FILTER=group__case cargo test -p ide --test test_ide_fixturesOne test target can drive several suites, so a suite that holds no matching case skips quietly. But a filter that matches no case in any suite fails: that is almost certainly a typo, and a run of zero cases would otherwise report a pass.
A suite renders its cases on one thread per core and reports the failures in file order, each ending with the command that reruns it:
rerun: FIXTURE_FILTER=annotation_compact__atomic cargo test -p format --test test_format_fixtures format_fixtures -- --exactWhile iterating, cargo test -p semantics is the default command, since it covers analysis
behavior. just gate runs every test plus clippy and a formatting check.
Tests run locally, with just gate for everything and the two recipes below for the part that
needs no GNU R. CI runs tests only when a maintainer starts a workflow by hand or tags a release,
never on a push or a pull request. The workflow for the part without GNU R, test.yml, is staged
in .github/pending/ and runs nothing until a maintainer accepts it
(Releasing). It has two jobs:
| Job | Recipe | What it runs |
|---|---|---|
tests |
just test-fast |
Every test of the crates that build without the interpreter, libR and Polars (syntax, format, semantics, ide, sniff, loader, heap, numeric, ssa, rdb, crash, oracle, rcmd, repl and ry-lang): every fixture suite, the property fuzzers’ bounded passes, the simulator and the scenarios. A test that compares with GNU R or runs ry’s runtime skips, because the runner has neither |
fuzz |
just fuzz-all 60 runtime |
Every libFuzzer target that builds without Polars (15 of 17, all but csv and unserialize) for 60 seconds of wall clock each. Every target runs, and the job fails if any finds a failure. A failing input is uploaded with the run |
The staged release.yml calls test.yml, and its publish job needs it, so once both files are
accepted a release with a failing test publishes nothing.
The full gate is ci.yml. It runs just ci, which
builds base R and runs rustfmt, clippy and the tests with the CI runner’s variables, inside the
flake’s ci dev shell. That shell pins everything the tests touch (the Rust toolchain, GNU R 4.6.2
built as the oracle recorded it, GCC 13.3, glibc and the oracle’s locales), so this command
reproduces a CI run on any machine with nix:
nix develop .#ci -c just ciTo change what a workflow runs, edit its recipe or the shell in flake.nix; the workflow files only
call them. The full gate runs only when started by hand: the orchestrating agent starts it on main
at its hourly check when main has new commits. A pull request gets no run of its own.
Blessing expectations
Section titled “Blessing expectations”With RY_BLESS=1, a mismatch does not fail. Instead, the harness rewrites the #++++ blocks in the
.test files in place from the runner’s actual output:
RY_BLESS=1 cargo test -p semantics --test test_typing_fixturesOnly the body of each #++++ block is rewritten. Directive lines and the spacing around them are
kept, so a blessed file is byte-identical to what a person would have written, and a second run
without bless passes. FIXTURE_FILTER still applies, so you can bless a single case, and blessing a
suite that is already correct changes no bytes.
Review every blessed change before you commit it. Bless records whatever the runner produces, wrong outcomes included, so accept a changed expectation only when the behavior or the wording improved on purpose.
The fixture suites
Section titled “The fixture suites”crates/syntax/tests/syntax: golden lossless trees and syntax errors, rendered bydebug_dump.crates/syntax/tests/tsr: tree-sitter-r’s parser corpus, converted to the same format.crates/syntax/tests/errors: golden Elm-style error messages, organized by area (the lexer, delimiters, control flow, functions and calls, the#:type and directive grammar, and recovery locality). Its coverage contract has three parts: every distinct error message the lexer or parser can emit has at least one case here, valid but tricky shapes are pinned asno errors, and recovery cases assert that one broken construct reports once and leaves the rest of the file clean.crates/semantics/tests/naming: the name-resolution suite. It records which binding every name in an item resolves to, rendered flat in source order. This is the only suite that tests naming directly. Everywhere else naming is tested through its consequences, a type that comes out right or a diagnostic that does or does not fire, and consequences cannot see a read that resolves to the wrong binding while still producing the same type.crates/semantics/tests/typing: the typing suite. Each case runs the full semantic pipeline on one package file, or on several, with the shipped stubs installed. It renders the exported scheme of every top-level definition (name: <T: numeric> fn(x: T) -> T), the type of every other top-level statement as a bare line, and then the file’s diagnostics (start..end severity[code] message, in byte offsets). Each test thread reuses one database across cases (semantics::testing::with_fixture_project), because parsing the stub corpus costs far more than checking a case. Cases are grouped by feature, with one exception:programs.R.testholds whole small programs written the way real projects are written. A case earns its place there by exercising several features at once, and a clean case with an empty diagnostics block is a promise of zero false positives for that style of code.crates/semantics/tests/lowering: the HIR each item lowers to, dumped as an indented tree under the item’s name (or under# <statement>for an unnamed item). Argument tags are rendered inline, because matching arguments is what a call’s lowering has to get right. The suite exists because rewrites, such as the native pipe becoming a call, a replacement form, orlocal, would otherwise be tested only through the type that falls out at the far end, which cannot tell “lowered correctly” from “lowered to the wrong shape but typed the same”. A case whose comment saysWRONG-BUT-CURRENTpins a known defect next to the shape it should have, so fixing the defect turns the case red.crates/semantics/tests/typing-scripts: the same pipeline over script documents, which have one sequential, top-down scope. Besides script semantics, this suite holds the rules for single expressions (literals, operators, indexing, lists, and calls to stubbed functions), rendered one line per statement.crates/semantics/tests/annotations: one#:block per case, where the case’s lines are the block. It renders what the block lowers to (the kind of coercion, the declared type, the@newtarget, and each@typeor@aliasdefinition), followed by its grammar and lowering errors. This is the compact contract for the type notation, while thesyntaxsuite pins the trees and theerrorssuite the wording.crates/semantics/tests/stubs:.Rtypesdeclarations. Each case renders the declarations it loads, with overload candidates in order, and the problems that make the loader drop a line, in the wording the editor and the override report show.crates/semantics/tests/typing-inline-off: the ordinary rendering with[check] inline-annotations = false, which pins thery-syntaxwarning on records, tuples, tags andmatch, and shows that literals are still typed whilematchcoverage is not checked.crates/semantics/tests/typing-strict: the strict stream. After the ordinary rendering, each case appends the file’s typing mode and itsstrictdiagnostics.crates/semantics/tests/top-level: what a package file’s top level runs under strict mode. A case lists files with#~~~~separators, soR/files are library files andtests/or root files are entries. Each case loads the shipped stubs and renders, per file, every top-level item asdefinition,constantoreffect, with(pure)when the item declares its function pure, followed by the file’s top-levelstrictdiagnostics. Run it withcargo test -p semantics --test test_typing_fixtures top_level_fixtures.crates/semantics/tests/typing-imports: package metadata. A leading#namespaceline supplies the case’sNAMESPACEsource and a#descriptionline itsDESCRIPTIONsource, and both are installed as thePackageMetadatainput before rendering. The directive lines stay in the analyzed text as ordinary comments, so expected offsets are honest. Attaching a package needs no directive: alibrary(data.table)statement in the source activates the conditional stub namespace through the same scan the hosts run.crates/semantics/tests/lints,lints-styleandlints-object-systems-off:lints::lint_file, first under the default configuration, then undernaming-style = "snake_case",unused-parameter = "warn",shadows-builtin = "warn", andshadows-namespace = "warn", all of which are off by default, and last under[check] object-systems = false. Each case renders every finding asstart..end severity[code] message, orclean. Run all three withcargo test -p semantics --test test_lint_fixtures.crates/semantics/tests/load_index: the package-load index. A case listsR/files, and optionallyNAMESPACEandDESCRIPTION, with#~~~~separators, and renders the package’s name, theNAMESPACEdirectives a load honors, and, per file, each item’s position and whether it is bound lazily (as a function or, in a strict file, a constant) or runs eagerly, with what it writes, what it reads while running, and the registrations it makes. Run it withcargo test -p semantics --test test_load_index_fixtures.crates/loader/tests/reload: reload plans. A file headed#~~~~ before <path>or#~~~~ after <path>exists on that side only, so the same path on both sides is an edit and one side alone is an addition or a deletion, while a plain#~~~~ <path>is unchanged. The case loads thebeforeside, then renders the plan that takes it to theafterside: what is bound, run, removed, unregistered, re-registered, and which datasets are read again.R/files run in the orderDESCRIPTION’s collation field gives them, and adata/file is a dataset whose text stands in for its size and modification time. Run it withcargo test -p loader --test test_reload_fixtures.crates/format/tests/format: the formatter’s golden suite. Each case’s source must format to the expected block, and the runner formats that output again to assert idempotence on every case. When the expected result is a refusal, the case renders the structuredFormatError.crates/format/tests/format-range: range formatting. The source marks the selection with$0markers (two for a selection, one for a bare caret, none for the whole file), and the expectation lists the edits by one-based position and shows the document they leave behind, with each replacement between«and». Run it withcargo test -p format --test test_format_range_fixtures.crates/ide/tests/ide: the editor features. The source carries one$0cursor marker, which is stripped before analysis, and the expectation renders each feature’s answer at that position: the hover line with its absolute range, the definition’s target range, and the reference ranges. A target inside the stub corpus renders the token its range covers, as indeclared in stub source 0 at `print`, rather than a byte offset, because an offset shifts whenever an unrelated declaration is added, while the token proves the range is right.crates/ide/tests/structure: whole-file editor features. Each case renders the outline (nested by indentation), the folds as zero-based inclusive line ranges, the semantic tokens for R names, the quick fixes the diagnostics and default lints offer, and, when the source has a$0marker, every step of selection expansion from it. Run it withcargo test -p ide --test test_ide_fixtures structure.crates/ide/tests/migrate: the data.table to DataFrame rewrites behind the code actions andry migrate data-table. Each case is one script whose attached packages are the ones it attaches, and the expectation lists every rewrite as its title, the original lines (-) and the replacement (+), orno rewrite. A$0marker also lists the code actions offered there. Run it withcargo test -p ide --test test_ide_fixtures migrate.
Fuzzing and property tests
Section titled “Fuzzing and property tests”Fixtures check the answers someone wrote down. The harnesses in this section check properties that must hold for every input, over a great many inputs. Every pipeline stage gets fuzz coverage from its first commit.
The batteries of invariants are exported, so that the in-tree harnesses and the coverage-guided
targets share exactly the same contracts: syntax::testing::check_parse_invariants for the parser,
format::check_format_invariants for the formatter, and semantics::testing::check_semantics_input
for the semantic pipeline. The last one folds every lint, under a configuration that turns them all
on, into its rendering, so the lint layer inherits the same invariants. The format and semantics
harnesses, and those of syntax and ide, also carry a fuzz_regressions_hold_invariants battery
that pins every input the coverage-guided targets have ever broken.
The semantic pipeline
Section titled “The semantic pipeline”crates/semantics/tests/test_fuzz.rs runs generated programs, token soup, and two-file projects
through the full semantic pipeline, with the generator biased toward reference cycles, closures,
annotations, and typing-mode directives. On every input it checks four properties:
- Nothing panics, which among other things means every salsa fixpoint converges.
- The output is deterministic across fresh databases.
- Diagnostic ranges are well formed.
- Incremental equals fresh. Editing a file through the salsa setter gives the same output as a fresh database on the edited text, and editing it back restores the original.
FUZZ_ITERS scales the budgets. A bounded default runs inside cargo test -p semantics, the
ignored fuzz_deep test carries the long runs, and on a panic the harness prints the inputs that
produced it.
The formatter
Section titled “The formatter”crates/format/tests/test_fuzz.rs feeds the formatter valid-program seeds, byte-level mutations of
those seeds, token soup, random bytes, every fixture case source in the repository, and real corpus
files once they are fetched. It checks four properties: nothing panics (the formatter either
succeeds or refuses with a structured error), the output is deterministic, the code is preserved,
and formatting is idempotent, meaning that formatting a successful output again succeeds and
reproduces it byte for byte.
The refusal path is part of the contract. A file with a syntax error in its R is refused, while an
error in a #: annotation, which the parser marks in_annotation, only sends the affected block
down the verbatim path.
Preservation is the one property that can catch the formatter changing your code. Determinism, idempotence, and “the output formats again” all hold for a formatter that silently drops a statement or misspells a token. Preservation compares the sequence of non-trivia token kinds and spellings between input and output, with two allowances:
- Four kinds are ignored entirely (
{,},;, and#:), because the formatter may add or move them: it braces single-statement bodies, splits;chains onto separate lines, and re-lays out annotation blocks. - A string is compared by its content rather than its text, because the formatter chooses the quote character. Raw strings are copied byte for byte, so they keep their full spelling.
Comparing kinds alone would miss a miscompile: a stale or off-by-one source range makes the formatter emit the wrong bytes for an identifier, which renames a variable while preserving every kind. An injected bug of exactly that shape passed a kind-only comparison and fails the spelling comparison.
Every generated input also runs the range battery, check_range_format_invariants, over a bounded
spread of selections (the whole file, carets, whole lines, part-line spans, a range past the end).
Range edits must be deterministic, refuse exactly when the whole file refuses, cover whole lines in
order without overlapping, preserve the code, leave a document that still formats to the same
result, be a subset of the edits formatting the whole file in place makes, reproduce that result
when the selection is everything, and be stable when re-run on what they produced. On inputs of up
to 24 lines it also checks additivity (a span of lines makes exactly the union of its lines’ edits)
and composition (formatting line by line until nothing changes reaches the whole-file result, and
every step still formats to it). Composition is the strongest of these: it found whole-file
idempotence bugs that no single-pass check sees. crates/format/tests/test_format_range.rs runs the
battery over every fixture source, the program corpus, and re-indented copies of the fixtures (four
spaces, tabs, mixed steps), and checks every byte selection of a few short sources against an
independent line oracle.
The editor features
Section titled “The editor features”crates/ide/tests/test_fuzz.rs sweeps every feature over a sample of byte offsets in each input:
every token boundary plus a coarse stride, so positions off token boundaries and in the middle of
multi-byte characters are covered too. At each offset nothing may panic, and every range handed to
the editor must lie inside the text it indexes.
Containment alone is a weak check, though: it says a range is somewhere in the file, not that it is on the right thing. Wherever the cursor sits on a name, two stronger properties hold as well:
- Name identity: every rename edit, every reference, and the definition target cover text that
spells the same name (with backtick quoting normalized, since
`x`andxare one binding). - Round trip: if the cursor navigates to a declaration, asking for references from that declaration leads back to the cursor’s own token.
An off-by-one injected where item-relative ranges convert back to absolute offsets passes the containment check and fails name identity on the first seed.
The data.table rewrites are checked once per input: they never overlap, every range lies inside the text, and when the input parses cleanly, so does the text with every rewrite applied.
Completion is sampled once per completion context (the kind of token the cursor sits in or after) rather than at every offset, because it costs far more per call than any other feature and its answer depends on the syntactic context rather than the exact byte.
The program corpus
Section titled “The program corpus”crates/syntax/tests/programs/*.R.corpus holds 1,967 distinct R programs, read through
syntax::testing::program_corpus_sources. The syntax, format, and semantics batteries each
run the whole set. The corpus holds inputs only, with no expected output.
The inputs are usable on their own because the invariants need no expected output: nothing
panics, syntax checks the lossless reprint and the tree geometry, format checks preservation,
and semantics checks that every diagnostic range lies inside its file. The semantics arm shares
one database across the whole corpus instead of using the per-input battery, because a fresh
database re-parses the stubs every time. It therefore asserts only what a shared database can, and
leaves determinism and incremental equivalence to the generated inputs. Every arm also asserts that
the corpus is non-empty, so deleting it fails loudly instead of passing on zero cases.
The annotation round trip
Section titled “The annotation round trip”crates/semantics/tests/test_roundtrip.rs asserts that every type the checker prints is a type you
could write back. A #: TYPE annotation says the annotated value is compatible with TYPE, and the
checker has just proved the value has the type it rendered, so pasting an inferred scheme above its
own definition must add no finding.
No other test compares the renderer with the annotation grammar: every other suite reads a rendered type as a string, so a type that prints well but parses back as something else is invisible to them.
The test runs over every fixture case source. It skips cases that do not already check clean,
because “re-declaring adds no finding” needs a baseline with no findings, and it skips definitions
that already carry a #: block, because those render the user’s own text rather than a type the
checker chose. It re-points one database with the salsa setter instead of building a new one per
probe, because a fresh database re-parses the whole stub corpus.
KNOWN_UNWRITABLE lists schemes that are allowed not to round-trip. It is empty, and worth keeping
that way. Its entries name case ids rather than skipping by shape, so a new unwritable rendering of
any kind fails the test instead of blending into a known category.
Tree-sitter as a second opinion
Section titled “Tree-sitter as a second opinion”crates/syntax/tests/test_corpus.rs compares our verdict on “does this file have errors?” with
tree-sitter-r’s, in both directions and over two sets of inputs:
corpus_acceptanceruns over the fetchedcorpus/and gates the direction where we reject what tree-sitter accepts, which means a gap in our grammar.in_tree_acceptanceruns over inputs that are always present (the program corpus plus every fixture case source) and gates the other direction, where we accept what tree-sitter rejects.
That second direction is the only check that bounds the number of parse errors from below. Every other parser invariant bounds it from above, so without it an over-suppressing parser, the regression most likely to slip in given how much deduplication the parser does, would pass every fuzz battery while broken files went silently clean.
Each direction is gated only where it means something. Rejecting what tree-sitter accepts is noise
on in-tree input, which is full of #: annotations (comments, to tree-sitter) and of deliberately
broken sources. Accepting what tree-sitter rejects is gated there against
crates/syntax/tests/in-tree-acceptance-allowlist.txt. Each allowlist entry is adjudicated by
running R itself, and must say which of two things it is: tree-sitter being wrong, or a known gap in
this parser.
Inline type annotations are ry syntax that R rejects by design, so tree-sitter judges the plain R
program a source erases to (syntax::erase_inline_annotations), not the source itself. That makes
the test a check on erasure too: an annotated fixture whose erased text tree-sitter rejects fails it.
check_parse_invariants pins erasure from the other side, for every fuzz input that parses cleanly:
the erased text parses cleanly, holds no inline annotation, and erases to itself.
The runtime
Section titled “The runtime”The runtime is tested against models and against GNU R rather than with fixtures. The oracle’s
cases (crates/oracle/cases/, Corpus and oracle) hold the
R-level behavior, and crates/interp/tests/test_fuzz.rs fuzzes lowering and checks that the
tier-0 bytecode agrees with the evaluator. crates/jit/tests/test_tiers.rs does the same one tier
up: it runs random loop, branch and vector programs as closure bodies under tier 0 and under the JIT
forced on (jit::Mode::Force, which compiles a body on its second call and a loop on its first
backward jump), and every call must give the same value or error. Most generated programs must run
to a value, so a generator that drifts into erroring early fails the test instead of passing
without comparing anything. The test interpreter has no base R closures (numeric, as.list,
withCallingHandlers), so programs use builtins, and each hand-written seed states whether it is
meant to fail. Behavior that needs base R, such as a warning handler watching a compiled loop,
goes in the oracle case bytecode__hot_loops.R. crates/interp/tests/test_linalg.rs pins base R’s linear
algebra to GNU R’s results bit for bit. The matrix products are ry’s own kernels, so their bits hold
on every machine; the LAPACK cases hold only on the reference libraries, so on another BLAS (such as
OpenBLAS) the test compares them with a GNU R on the same libraries, and skips them without one. crates/table/tests/test_table/main.rs checks the table engine
against values read from data.table, lazy.rs beside it checks the Polars-style verbs, and
crates/libr/tests/test_frame_altrep.rs checks the columns as ALTREP vectors. crates/interp/tests/test_records.rs and test_arrays.rs and crates/frames/tests/test_standard_library.rs cover ry’s
own values at run time, the last one also holding the runtime’s function table against the
declarations in types/std.Rtypes. crates/libr/tests/test_rmath.rs, test_numerics.rs and
test_linpack.rs compare the C API’s numerics with answers recorded from GNU R, so they run
without R. The optimizer cases test_numerics.rs holds bit for bit are skipped, with a message,
where they cannot match: against a fresh GNU R that does not call the reference BLAS, and, for the
trust-region case, where the C library’s hypot rounds differently from the recording’s (its
hypot case fingerprints it). test_fuzz_rmath.rs checks properties of the distribution functions on random and
extreme arguments. crates/heap/tests/test_gc.rs builds random
object graphs (vectors, lists, interned strings, environments), drops roots, stores young objects
into old ones and collects at random points, with and without collection stress, and after every
collection checks each object the model still reaches. It also re-interns every string ever made,
so an intern-table entry that outlived its string shows up. A disabled write barrier, a frame whose
cells are not traced, and a minor collection that keeps dead strings in the table each turn it red.
Tests that load base R take it from the GNU R installation R_HOME names, and the runtime then
behaves as that version; without R_HOME they use ry’s own base R (just base-r), and the
scenarios skip when it is not built. The oracle’s expectations are GNU R 4.6.2 output, so just ci
sets R_HOME to 4.6.2.
Run them once per supported version, for example R_HOME=/opt/R/4.6.1/lib/R cargo test -p interp, and the oracle the same way with ORACLE_R (Corpus and oracle).
crates/libr/tests/test_fuzz_native.rs fuzzes the native boundary itself. It nests every
non-local exit native code can set up (Rf_eval, R_tryEval, R_ToplevelExec, R_tryCatchError,
R_UnwindProtect, R_ExecWithCleanup and a cpp11-style C++ unwind) around values and errors, and
checks the outcome, which cleanups ran, and that every C context, protection and R_alloc block was
released. A second battery drives random API sequences under collection stress against a model.
oracle fuzz runs whole R programs under GNU R and ry’s engines and reports every program on
which an engine differs. A seed and a kind name a program, so oracle fuzz --kind K --seed N --print prints it and --engines replays it. The five kinds find different bugs:
| Kind | A program is | It finds |
|---|---|---|
grammar (oracle::fuzz::program) |
several blocks from a grammar of language features: loops shaped for the JIT, closures, S3 dispatch, conditions and on.exit, environments, attributes, NA and overflow, and element stores into a function’s argument copy on both sides of 64 elements, in a function called three times so its body runs interpreted and compiled; about one in eight ends on a deliberate error |
interpreter and JIT bugs in control flow and scoping |
metamorphic (oracle::metamorphic::program) |
six to twelve trials; each builds one value, runs one body of updates and reads on it, and runs the same body in other forms that must give an identical result | a result that depends on how often or where code runs, or on how a value is stored |
builtins (oracle::metamorphic::builtins_program) |
builtins called on a zoo of values, each call caught | a builtin that accepts, rejects or computes differently from GNU R |
api (oracle::metamorphic::api_program) |
calls of functions drawn from everything base, stats and utils export, on two zoo values and constants of other types, each call caught | a function nobody listed: one that crashes, or accepts, rejects or computes differently from GNU R |
corpus (oracle::corpus) |
a snippet of real R code with one to four mutations | what no grammar author thought of |
A metamorphic trial draws its value from a zoo: every atomic type and lists, with NA, NaN,
Inf and -0; zero-length vectors and lengths on both sides of the sizes where storage changes
(the inline capacities, the 512-byte small-vector limit, the 64 KiB arena chunk, a compact 1:n); names that are empty, NA, duplicated or
non-ASCII, attached through names<-, setNames, structure, argument tags and do.call;
strings marked UTF-8, latin1 and bytes; matrices, arrays, data frames, factors and dates. The body
is a few statements over x: element, slice, matrix and name assignments, attribute and
replacement-function calls, <<-, loops, and reads by position and by name. The trial computes
the reference result once, then compares it by identical() with the body’s result in each
other form and prints same or DIFFERENT:
| Form | The body runs |
|---|---|
again xK |
K times in a row in the same closure, K from 1 to 300 |
warmed, cold |
in the closure after it ran on values of other types and sizes, and in a copy of it that never ran |
local, inline, evaluated, on.exit |
inside local(), in an anonymous closure, through eval(quote()) in a new environment, or in a closure with an on.exit handler |
top level |
as top-level statements on a global variable |
renamed NAME |
with x under another name, among them a non-syntactic one, a non-ASCII one, T and c |
caller |
called through do.call, lapply, Map, Reduce, replicate or tryCatch |
shared |
on a value that a second variable, a list, an environment, a closure or an attribute also holds, whose copy must come out unchanged |
collected |
with a gc() after every statement |
serialized v2, serialized v3 |
on the value after unserialize(serialize()) |
A DIFFERENT line is a bug in the engine that printed it, whatever GNU R says, so the kind is
also a test without GNU R. Each seed turns on a random subset of 33 features (the value types,
the name and encoding sources, the statement families, the forms), so a rare pair of features
gets programs of its own instead of drowning in the common ones.
Printed output hides much of a value, so every engine runs with the deep observer on
(crates/oracle/src/observe.R, ProcessEngine::observing, off with --no-observe). After each
top-level expression the driver records a canonical dump of its value and whether it was
visible: the type, length and object bits, every element (a double by its bits, a string by its
encoding mark and bytes), the attributes in the order attributes() gives them, and the
structure of calls, closures and environments. A vector of more than 96 elements shows its ends
and, for the middle, checksums that depend on the position of every element. Each condition is
recorded with its classes, message and call. A non-ASCII string shows the same mark whether it
is flagged UTF-8 or unflagged, because the two are the same text in a UTF-8 session;
ORACLE_OBSERVE=strict tells them apart.
The builtins kind calls a hand-written list of functions with arguments that suit them. The
api kind takes its functions from a table instead, crates/oracle/src/api_functions.txt, which
api_functions.R writes from GNU R’s namespaces (regenerate it after a GNU R upgrade:
Rscript crates/oracle/src/api_functions.R > crates/oracle/src/api_functions.txt). The script
leaves out what cannot be compared: functions that read or write files, connections, the terminal,
the clock, the session or the graphics device, most dot-prefixed internals, and one function on
which GNU R corrupts its own memory. A formal that names a file or a package is never passed. A
call gets up to three positional arguments (the first is usually the zoo value v, the second
w, the third a small count) and sometimes one named argument, and a replacement function is
often called through replacement syntax (x <- v; names(x) <- w). A third of the calls print
their result, so print methods run too. Most calls are errors in both engines, and then the
messages are compared.
The report of a diverging seed names each top-level expression that differs and how:
| Difference | Meaning |
|---|---|
crash |
the engine died of a signal or a panic |
value |
the expression’s value or visibility differs |
accepts, rejects |
one engine signals an error where the other returns |
message |
both signal an error, with another message, call or class |
conditions |
the values agree; a warning’s or message’s text, call or classes differ |
printed |
the values and conditions agree and the printed text does not |
--jobs N runs N programs at once, --minutes M keeps taking seeds until the time is up,
--findings DIR writes each diverging program and its report to a directory, and the summary
counts the differences by kind. For the builtins and api kinds it also counts them by the
function called and writes DIR/functions.txt, one count, kind and function per line, most
frequent first, which ranks the gaps:
14 accepts type.convert10 rejects aregexec7 value nargsoracle fuzz --kind all --count 500 --coverage (just fuzz-coverage) runs no engine. It generates
the programs and reports how many of the table’s callable functions each kind names in a call, so
a generator change shows what it added to the reach:
kind all base stats utils(the table) 1479 1000 427 52grammar 144 143 0 1metamorphic 131 129 1 1builtins 297 285 7 5api 1473 995 426 52corpus 649 458 178 13together 1474 995 427 52The count is a close upper bound, because a name inside a string literal counts as a call.
Every fuzzed engine runs with at most 2 GiB of writable memory (oracle::fuzz::MEMORY_LIMIT,
RLIMIT_DATA), so a call that asks for a huge vector gets R’s “cannot allocate” error in GNU R
and in ry instead of taking the machine’s memory. just fuzz-differential runs the grammar,
just fuzz-kind KIND any kind, and just fuzz-long HOURS BASE all five for a time budget, under
tier 0, the forced JIT and seeded collection stress.
crates/oracle/tests/test_differential.rs holds the bounded passes. Grammar programs run under
GNU R, tier 0 and the JIT, and under collection stress against ry unstressed. Metamorphic,
builtin and api programs run under tier 0 and the forced JIT, which must agree with each other,
print no DIFFERENT and never crash (fuzz_metamorphic_forms_agree_across_tiers,
fuzz_builtins_agree_across_tiers, fuzz_api_calls_agree_across_tiers); their comparison with
GNU R is the long run’s, because ry differs from GNU R on many builtin and api programs and a
default test must pass.
A divergent program is 50 to 300 lines, so a finding is reduced before it becomes an oracle case.
oracle reduce FILE --engine NAME (just fuzz-reduce FILE ENGINE, or oracle fuzz --reduce for
every diverging seed) shrinks it by delta debugging over the syntax tree: it removes statements,
block items, arguments and parameters in halving chunks, replaces a construct by one of its parts
(an if by a branch, a call by an argument, a loop by its body), replaces compound expressions by
NULL, 0, 1L, "" or TRUE, and shrinks string and number literals. A candidate counts only
when GNU R parses it, GNU R runs it to completion if it ran the original to completion, and the
engine still diverges with the same signature: the same crash signal, a timeout, or the first of
errors, warnings, messages, stdout, value and the deep observer’s dump that differs. A difference
in errors or warnings also keeps both engines’ messages, and a difference in the dump keeps the
way its first expression differs (the table above), so a reduction seldom slides to another bug;
a difference in printed output or the last value keeps only its channel, so read the reduced
program against the original report. Both engines run with the observer on. For example, a segfault of ry-tier0 in a replacement on a matrix,
buried in a 130-line fuzz program, reduces in about 300 candidates (a few minutes in a debug build)
to:
f <- function(m) { m[1, 1] <- 0; m }f(matrix())f(matrix(0, 8, 8))The command prints the reduced program, its signature and a replay command (with
ORACLE_OBSERVE=1 when only the observer’s dump differs), and --output writes it. --budget
caps the candidates (2000 by default), and the reduction is deterministic unless a candidate
runs into the timeout. oracle fuzz --reduce starts no candidate after the time budget and
leaves a timeout unreduced, because each candidate that still hangs costs the whole timeout.
oracle::reduce::reduce takes any predicate, so its unit tests run without an engine.
A grammar only covers what its author thought of, so a second source of programs is real R code,
mutated (oracle::corpus, oracle::mutate). The corpus is extracted at run time into
target/fuzz-corpus and never committed: the examples of every base package’s help pages
(tools::Rd2ex), the tests/*.R of the R source for the installed release (downloaded from CRAN
once by oracle fuzz --kind corpus), the oracle cases, and the tests of corpus packages that ship
them. Each file splits into snippets, runs of top-level expressions cut at blank lines, without
the paragraphs that touch graphics, files, the clock, the environment, addresses or unseeded
random numbers. A seed picks a snippet and applies one to four mutations on the syntax tree:
boundary values for literals, materialized a:b, named, reversed or emptied c(), a statement
wrapped in a closure, local, identity, a serialization round trip or eval(quote()),
repeated calls, swapped, dropped or duplicated arguments, and neighboring operators. Every engine
runs under the deep observer. The unmutated snippet runs first: an engine that already differs on
it is a plain compatibility gap, listed in target/fuzz-corpus/base-differs.txt, and only a
mutant that diverges while its snippet agrees is a finding. A diverging mutant runs under GNU R a
second time and is dropped when GNU R disagrees with itself. just fuzz-corpus SEED COUNT ENGINES
runs a range of seeds. The bounded pass, fuzz_corpus_mutants_never_crash, fails only when a
mutant crashes ry and prints the other divergences, because the snippet a seed picks depends on
the corpus on disk: it changes whenever an oracle case is added, and with the R source and the
packages a machine has. A corpus seed therefore replays only against the same corpus, and a
report carries the mutant’s text for that reason.
crates/oracle/tests/test_gallery.rs renders a seeded sample of the plot gallery under both
engines and compares the images (Corpus and oracle).
just gc-torture runs every oracle case on every core and takes about ten minutes on four cores
in a debug build, and just gc-torture young runs them under ry-gctorture-young.
just gc-torture repeat runs them twice in one process under seeded stress with the heap checked
(ry-repeat-gctorture-seeded), takes about forty minutes on four cores in a debug build, and
prints each case that takes over a minute. A case is killed after fifteen minutes for each pass
it makes. The generated
programs run under both torture engines. RY_GC_STRESS=1 stresses from the script’s start and
RY_GC_STRESS=startup from the first allocation, which takes about seven seconds on an empty script
in a debug build (Robustness lists the modes).
crates/frames/tests/test_debugger.rs drives the debug adapter (interp::debugger) in-process.
Scripted sessions pin breakpoints, stepping, the stack, variables, evaluate, browser() and
uncaught errors as transcripts, and a seeded simulator plays a client that sends random, sometimes
invalid, requests at every stop. Every request must get exactly one response, the session must end,
and the interpreter must be left clean. FUZZ_SEED=<seed> replays a failing session.
Known oracle failures
Section titled “Known oracle failures”crates/oracle/known_failures.toml lists the oracle cases that fail on main under one engine, so a
branch can tell its own regressions from failures it inherited. Each entry names the case (its file
name in cases/ without .R), the engine as --engine names it, the area of ry the bug lives in
(such as jit or gc), and a one-line reason; provisional = true marks a failure seen on a branch whose runtime matches main
but not yet confirmed on main itself.
[[failure]]case = "base__match_arg_native"engine = "ry-jit"area = "jit"reason = "the compiled body warns twice where GNU R warns once"oracle compare and cargo test -p oracle (with ry’s runtime built) sort every failure against the
file and fail the run on two things: a failure the file does not list, and a listed case that
passes. The second rule means the change that fixes a case also deletes its entry. Add an entry
only for a failure main already has, never for one your own change introduces. A run prints each
listed failure as known without its diff and ends with one line per engine:
ry-jit: 409 passed, 16 failed (16 known, 0 new), 0 skippedA crash is never a known failure. When an engine is killed by a signal, the run prints the case
as CRASHED and counts it as new even if the file lists it, because a crash is a memory-safety
bug and blocks until it is fixed. Where it shows can depend on the host, since it depends on
where the collector happens to run, which is why the gctorture engines (ry-gctorture,
ry-gctorture-young, ry-gctorture-seeded, ry-repeat-gctorture-seeded and ry-jit-gctorture)
exist. The file has no mark that lets a case pass on
some runs and fail on others. A beta release requires the file to be empty.
A count is only trustworthy if it covers every case, so oracle compare also refuses to start when
a case needs a package that is not installed or a locale that is not generated. A case names them
in a leading # oracle: requires comment: packages by name (# oracle: requires knitr, rmarkdown)
and a locale it switches to as # oracle: requires locale en_US.ISO-8859-1. The run lists each
such case and exits 1. Install the packages into the corpus library with just corpus, which the
oracle searches by itself (Corpus and oracle), and generate the
locales with sudo tools/dev-setup.sh (or tools/oracle-locales.sh alone). For a deliberately
partial local run, --allow-skips skips those cases and marks every summary line PARTIAL:
skip cases/datatable__fread.R: needs bit64skip cases/packages__xts_keeps_source_rownames.R: needs xtsok cases/arith__complex_math.Rok cases/base__match_arg_native.RPARTIAL ry: 2 passed, 0 failed (0 known, 0 new), 2 skipped (2 missing a package or locale)PARTIAL: --allow-skips left out 2 cases, so these counts do not cover the suiteThe oracle refuses a file whose entry names an unknown engine, a missing case, an unknown key, or
the same case and engine twice. known_failures_are_valid in test_cases.rs checks the file
without running R.
The table engine
Section titled “The table engine”crates/table/tests/test_table.rs drives the engine from Rust. Every expected value that encodes
data.table behavior was read from data.table under GNU R. Its differential fuzz generates random
tables and queries and checks the native kernels against the Polars plans
(query::execute_with_plans_only), which serve as the reference. FUZZ_ITERS raises the
iteration count; run it in release, as FUZZ_ITERS=3000 cargo test-release -p table fuzz. The
fuzz tables are small, so the kernels’ parallel paths get large-table tests of their own.
just bench-table <dir> [--rows 1e8 --groupby-only] runs the db-benchmark questions under
data.table, as R code under ry run and on the engine from Rust, fails if any checksum differs from
data.table’s, and rewrites the matching results table in development/design/r-runtime.md.
--groupby-only, --joins-only and --updates-only limit a run to the group-by questions, the
join questions, or the two := loops that write existing columns.
just bench-base [--rows 1e5,1e6 --only REGEX --memory] times base R’s own kernels (sorting,
string formatting and regex, grouping, merge, match, matrix margins, cumulative sums, xts and
zoo rolling operations) under ry run and GNU R, and prints the ratio ry / GNU R per kernel. A
kernel whose checksum differs from GNU R’s is marked MISMATCH. With --memory every kernel
runs in its own process, and the table adds both engines’ peak RSS. zoo and xts load from
RY_BENCH_RLIB (default ~/R/library).
just bench-ci [--only k1,k3 --runs 2 --scale 1] runs fifteen kernels cut from a user’s profiles
of a large package’s test suite (tools/bench/internal/ci/kernels.R) under GNU R and ry run.
Each kernel is one code shape a profile ranked high: creating an S4 object with a validity method
(k1), as.Date on a character scalar (k2), scalar calls of ifelse, isTRUE and unname
(k3), an argument checker over list(...) (k4), %in% and which on a small vector (k5),
an xts merge (k6), a data.table grown by rbind (k7), pmin/pmax on matrices (k8), and
parse with source references (k9), readLines (k10) and a recursive list.files (k11)
over forty generated source files, new() on a class of two plain slots (k12) and on a class
whose prototype holds a million doubles (k13), forcing every object of three installed
packages’ lazy-load databases (k14), and a grouped data.table j that calls a function
(k15). The table shows the fastest of --runs runs on each engine and the speed-up, and a
kernel whose checksum differs from GNU R’s is marked DIFFERS and fails the run. The packages (xts, zoo, data.table, purrr) load
from R_LIBS, by default the corpus library that just bench-packages install fills. The file
says which kernels follow code a user sent and which are reconstructions.
Compare peak RSS between two builds only from the same page-cache state. A freshly built .ryc
or libR.so sits in the page cache in large folios, so one fault maps a whole block, and the same
build reads back in small pages after eviction: two identical builds differed by 5 MB this way.
python3 tools/bench/evict.py RY... evicts every file each build reads at run time, recursively:
the executable’s directory, its base R tree with every package’s .ryc, ry’s cache directory, the
homes R_HOME lists (a path list, so one call covers a separate home per build) and the R_LIBS
libraries. tools/bench/bench.py and bench-base --memory run it before measuring.
Run it before every peak-RSS run you script yourself, because evicting only the top of base-r
leaves the .ryc images cached.
Judge a change’s peak RSS in three parts, because they behave differently. Anonymous memory and
the start image’s resident pages must not grow. The resident pages of libR.so’s code are not
judged per change. A plain release build has no function ordering, so where the linker happens to
place hot functions decides which neighbouring code readahead and fault-around map in with them,
and how far that moves depends on the change rather than its size. On the validation workload, a
no-op exported function added to an otherwise unchanged build moved resident libR.so text by
under 0.1 MB, while a change that grew .text by 5 KB moved it by +2.4 MB, with identical
anonymous and image RSS. That change’s extra resident pages held code the workload never runs (the
analysis core, polars, graphics): they were mapped because they sit next to hot code. Judge the
size of libR.so’s .text section instead (size -A), which is deterministic, and expect it to
grow only by about what the change adds. Resident text is checked on the shipped layout
(tools/layout/optimize.sh) when a release is cut. tools/bench/bench.py prints each program’s
peak in those parts (whole process, anonymous, start image and libR.so resident, sampled from
/proc/<pid>/smaps every 20 ms) and the .text size of each build’s libR.so. With
--baseline, it also prints the paired ratio baseline / ry for each program and its geometric mean
with a 95% interval, so a value above 1 means the build under test is faster.
A change that allocates less can still raise anonymous peak RSS, because fewer collections change which heap pages fill before the next one. Such a rise is exempt only when all three of these hold, and the change’s description shows each:
RY_GC_LOGgives the same peak old-generation bytes (the last column) on both builds;valgrind --tool=massifgives the same malloc peak, and with--pages-as-heap=yesthe same mapped size;- a control that puts the missing allocations back, such as one throwaway
c(1, 2)per loop iteration, closes the gap.
formatC()’s direct kernel call met all three: report’s VmHWM rose by about 1 MB, all of it in one GC arena, and the control closed it to about 0.1 MB.
Layout moves single-program wall time too, and with it callgrind’s simulated branch mispredictions
and cache misses. Only the instruction count (Ir) holds still. So judge a regression on one
program by Ir from valgrind --tool=callgrind --cache-sim=yes --branch-sim=yes on both builds:
a rise in Ir is real. A wall regression, or a rise in simulated mispredictions or cache misses,
without a rise in Ir blocks a change only when it reproduces on two layouts: the change’s build,
and the change’s build plus a no-op exported function (the control), which shifts the code placed
after it. On the public suite’s lapply program, one source built several ways gave:
libR layout |
Ir |
Indirect mispredictions | Conditional mispredictions | I1 misses |
|---|---|---|---|---|
| plain release | 6,650.6M | 8.6M | 16.6M | 69.4M |
| plain release + control | 6,650.6M | 10.6M | 12.2M | 68.3M |
| functions and blocks aligned | 6,645.4M | 10.6M | 14.5M | 23.4M |
| aligned + control | 6,645.4M | 8.6M | 16.1M | 27.8M |
| PGO and BOLT | 5,256.3M | 0.54M | 29.2M | 2.75M |
| PGO and BOLT + control, same profile | 5,256.3M | 0.55M | 28.7M | 2.74M |
| PGO and BOLT + control, own profile | 5,256.3M | 5.45M | 24.2M | 2.76M |
Wall time can follow these counts: a build with 8.6M indirect mispredictions ran lapply about
25% slower than one with 0.6M at the same Ir. Two layouts were tried as a way to take the noise
out of benchmark builds, and neither does. Aligning functions to 64 bytes and blocks to 32
(-C llvm-args=-align-all-functions=6 -C llvm-args=-align-all-nofallthru-blocks=5) grows
libR.so’s .text by 24 MB (27%), and the control swings the counts as far as before. The
shipped layout (tools/layout/optimize.sh, applied to libR for this table) holds still under
the control only when both builds reuse one training profile. A build that trains its own
profile, as every release does, gets a different function order and the swing returns. The builds
that judge a change therefore stay plain release builds, and the control is the test for a layout
effect. The published suites measure the shipped layout itself: they are recorded on the release
archive (bench.py --release, see Benchmarks).
Two builds can share target/base-r in a comparison, because each loads only images its own
name carries (ry-start.<build>.image, <package>.<build>.ryc, see
the architecture page). A build’s first start
writes its images into the base R tree, so tools/bench/bench.py starts each build once before
timing anything. A package library gets code images only when a package is installed, so bench.py
stops when a package in the corpus library or R_LIBS has only another build’s code image, and
names the packages to reinstall. RY_IMAGE_LOG=1 makes ry print ry: image unused: <image> was written by another build of the runtime (or why else it skipped an image) on standard error, and
ry: image written: ... when it writes a build’s images. bench.py sets it on every timed process
and fails one that prints either, because that process timed a slow start. It refuses a runtime
that predates the flag. Under --release, the archive extracts its runtime and base R into a
cache directory of its own, so it loads the images it shipped, and bench.py reads its runtime from
the R home it reports. two_builds_alternate_starts_without_displacing_each_others_images in
crates/ry/tests/test_perf.rs pins both halves.
just bench-stdlib [filter] [n] times every List, Str, Dict and Result function at
n = 1e6 by default against its fastest GNU R equivalent, and prints one row per function with
the ratio ry / GNU R. The cases live in tools/bench/internal/stdlib: ry.R holds ry’s side and gnu.R
the R alternatives in base R, purrr, stringr and stringi under the same case names, and the
fastest alternative sets the bar. A callback function is compared with R’s callback idioms
(vapply, Filter, Reduce, a for loop, purrr), and a function on one string with R’s
vectorized string functions. Where ry’s input is a vector, the case also runs it on a List,
ry’s unboxed vector, and the faster of the two counts. Each case runs in its own process, so a case slower than
--timeout shows as timeout without stopping the rest, and a check column compares
checksums of both results. Packages load from R_LIBS_USER, and an alternative whose package is
missing is skipped and named in the row. --sort ratio puts the slowest functions first, and
--write saves the table, as in tools/bench/internal/stdlib/results.md.
Coverage-guided fuzzing
Section titled “Coverage-guided fuzzing”fuzz/ is a cargo-fuzz crate, in its own workspace and excluded from the main one. Each target is a
thin wrapper around a battery its crate exports, which the crate’s in-tree harness runs too:
| Target | Battery |
|---|---|
parse |
syntax::testing::check_parse_invariants |
format |
format::check_format_invariants |
semantics |
semantics::testing::check_semantics_input |
ide |
ide::testing::check_ide_invariants |
position |
ry::position::check_position_invariants |
config |
ry::config::check_config_invariants |
lsp |
the language-server simulator, run_session, reading its choices from the input |
load_index, reload |
loader::testing::check_load_index_input, check_random_reload |
sniff |
sniff::check_sniff_invariants |
csv |
table::io::check_csv_bytes |
interp_lower, interp_unserialize, bytecode |
interp::testing::check_lowering_input, check_unserialize_input, check_bytecode_agreement |
interp_code_image, interp_code_image_file |
interp::testing::check_code_image_round_trip, check_code_image_file |
unserialize |
R::serialize::check_unserialize_input |
The semantics target derives its document kind and an incremental edit from the input bytes, so a
single byte stream drives the full battery, the spliced reparse of the edit included. The generator-driven targets
(lsp, reload, bytecode) read the generator’s choices from the input through
syntax::testing::ByteStream, so a byte mutation is a mutation of the program or session.
Two features of fuzz/Cargo.toml select the targets. runtime adds the interpreter’s targets, and
polars adds csv and unserialize, whose crates link Polars. Building Polars with coverage
instrumentation needs more memory than everything else in the workspace: in a 15 GB container
with four build jobs, the compiler is killed while it builds polars-core. On such a machine,
cargo fuzz build --no-default-features --features runtime builds the other 15 targets, and
--no-default-features alone builds the analysis targets.
A target that links the runtime’s heap runs without leak detection. LeakSanitizer does not scan
mapped memory, which is where the heap keeps its objects, so memory that a live object owns would
be reported as leaked. The heap crate turns the check off for a fuzz build
(__lsan_default_options in crates/heap/src/space.rs); the analysis targets keep it, except
reload and load_index. The loader parses ahead on a detached thread, and a leak check that runs
while that thread exits reports the thread’s own bookkeeping, so those two targets turn the check
off themselves.
interp_unserialize runs with a ceiling of 16 MB on one allocation
(Heap::set_allocation_ceiling, set by check_unserialize_input). A valid stream can name more
memory than it holds, because a compact sequence is three numbers and reads as a vector of any
length. Under the ceiling such a stream fails with the error for a vector that does not fit, so
the target’s memory is bounded by the size of its input.
The reader treats its input as hostile, because unserialize() and readRDS() take any file. It
refuses a length that the rest of the input cannot hold, a byte-code reference outside its table,
and an item of the wrong kind in a position that implies one: a symbol’s name, a string’s element,
an attribute list or an attribute’s name, an environment, the name of a binding, of an argument or
of a formal argument. Every such stream is pinned in crates/interp/tests/test_fuzz.rs, where one
test asserts that each is refused and that every stream GNU R wrote for libR’s conformance cases
still reads. That test runs under collection stress, since the damage a wrong kind did showed only
at a later collection.
It needs nightly, which the repository’s pinned toolchain already is. Run cargo install cargo-fuzz, then just fuzz-run format runs one target, and just fuzz-all 120 seeds the
corpora, builds every target and runs each for two minutes. A second argument names the features, so
just fuzz-all 60 runtime leaves out the two Polars targets, and that is what the staged test.yml
runs (Running one case).
The two minutes are wall clock, enforced by timeout, and a target that is interrupted at the
bound has passed. libFuzzer’s own -max_total_time cannot bound the pass, because it starts
counting only after the target has replayed its whole corpus, and seven targets are seeded with
every fixture source in the repository (about 5,300 files). ide runs four to seven inputs a
second, so its replay alone takes about 15 minutes. fuzz-all replays the seeds in random order
(-prefer_small=0), so a target that cannot finish its corpus inside the bound reads a different
sample on each run. For a run that mutates beyond the seeds on a slow target, use just fuzz-run
with a long -max_total_time.
Without just, from fuzz/:
cargo -Zscript ../scripts/seed-fuzz-corpus.rs # seed from all fixture case sourcescargo fuzz run format -- -max_total_time=600 -print_final_stats=1cargo fuzz run parse -- -max_total_time=600The seeder copies every fixture case’s source in the repository into
fuzz/corpus/{parse,format,semantics}/. Files are content-addressed, so running it again only adds.
The corpus and the artifacts are gitignored, because a durable regression belongs in the
REGRESSIONS battery or in a fixture case, not in the corpus.
When the fuzzer finds something, minimize it with cargo fuzz tmin format <artifact>, fix
the bug, replay the artifact, and add the minimized input to REGRESSIONS, plus a golden fixture
case when the shape deserves a pinned rendering. To judge corpus quality, you can get a coverage
report (the pinned toolchain includes the llvm-tools component it needs) with
cargo fuzz coverage format && cargo cov -- show, as the
cargo-fuzz coverage guide describes.
Deep runs are manual or scheduled work; the bounded in-tree batteries remain the gate in the default
suite.
Reloading
Section titled “Reloading”crates/loader/tests/test_reload_fuzz.rs generates packages from statements that define, wrap,
read and rewrite a small pool of names, edits them at random, and checks that applying the reload
plan to the loaded package gives what a fresh load of the edited package gives. A model executes
both sides: a lazy binding’s value is its definition’s text, and a statement stores its text plus
the values it read, so a statement that should have rerun shows up as a stale value. The same test
checks that the incrementally derived index equals one built from scratch, and a second test feeds
the index arbitrary bytes and asserts its positions stay inside the text. Removing either of the
diff’s ordering rules turns it red within a few hundred seeds.
End-to-end suites
Section titled “End-to-end suites”The CLI
Section titled “The CLI”crates/ry/tests/test_cli.rs drives the real ry binary end to end. It covers:
- diagnostic rendering, with 1-based character positions, snippet windows, and underlines;
- JSON Lines output;
- the exit-code contract;
- finding the configuration, and failing to;
- per-file
# typing:directives and suppression comments; NAMESPACEvalidation, and project stub overrides including their loader-problem reports;- data-masked evaluation;
- the formatter’s
--checkmode, range formatting, and literate documents; ry runandry Rwithout R: help output, usage errors, a missing script, and strict mode refusing a script before any R starts.
crates/ry/tests/test_command_line.rs property-tests and fuzzes argument parsing for ry run,
ry R and the ry repl alias against a model: everything after the script reaches it verbatim,
options resolve to the documented runtime and mode, and no argument vector panics the parser.
test_repl_e2e.rs runs every entry point on both runtimes: script arguments, -i, piped consoles,
and the earlier spellings (--runtime, RY_RUNTIME, ry repl).
Docs examples
Section titled “Docs examples”crates/ry/tests/test_docs.rs runs the examples in the docs through the real ry binary, so a page
that shows stale output fails the suite. A block opts in with a marker: the word ry after the
fence’s language, then the command and its options. Expressive Code ignores the extra words, so the
site shows a plain block, and the pages’ Markdown versions drop the marker. Here an output block
follows a tab group:
<Tabs syncKey="annotation-style"><TabItem label="Inline">
```rgreet <- function(name: character, greeting: character = "hello") -> character { paste(greeting, name)}greet("Ada", greting = "hi")```
</TabItem><TabItem label="Comments">
```r#: fn(name: character, [greeting]: character) -> charactergreet <- function(name, greeting = "hello") { paste(greeting, name)}greet("Ada", greting = "hi")```
</TabItem></Tabs>
```text ry check typingtype-mismatch
x this function has no parameter `greting`. Did you mean `greeting`?```A marked text block is the output of the nearest r block above it, or of every tab of the
nearest tab group above it, and each tab must produce it on its own. Inside a <TabItem>, after the
tab’s r block, it is the output of that tab alone, and a tab of the same group with no output
block of its own must report no errors. The code runs in a throwaway
project of one ry.toml and one file, main.R unless the output’s snippet names another file, as
--[R/both.R:2:31] does. A tab group that no output block follows must give the same findings in
every tab, with typing on. An unmarked output block is not tested.
| Word | Meaning |
|---|---|
check / run |
Run ry check, or ry run main.R. |
typing, strict |
Set that [check] key to true. Type checking is off unless a block says typing. |
KEY=VALUE |
Any other [check] key, written as in ry.toml, such as object-systems=false. |
runtime=ry / runtime=gnu-r |
Which runtime run uses; GNU R by default, as on the command line. |
exact |
Compare the whole output instead of an excerpt. |
continue |
Run the page’s earlier r blocks first, as one session, and compare only this block’s output. A check takes every earlier block; a run takes only those whose output the page checks, so an illustration that needs a package the machine lacks does not stop it. |
tab=LABEL |
Only that tab produces this output, for the rare example whose spellings legitimately differ. |
dput |
On an r block with #> output, print each result with dput() instead of print(). |
An r block can also hold its own output, as #> lines after the code that prints it, with a marker
such as ```r ry run runtime=ry dput. Each run of code before a #> group shows its last
value, and the comparison is always exact. A page’s #> blocks share one R process, each in its
own local() environment, because a runtime start costs seconds and the standard library reference
has over a hundred of them.
An excerpt matches when every non-blank line of it equals a line of the real output, both trimmed of surrounding whitespace, in the page’s order. Lines the page leaves out, such as the summary, a source snippet, or other findings, are free. A message the page wraps by hand matches one output line when its lines are joined with single spaces. This rule lets a page show the part of the output that matters while still pinning every word it does show. An exact block matches when the whole output agrees, ignoring trailing whitespace and the project’s temporary directory. Both compare standard output and standard error as one stream, interleaved as a terminal shows them, with color off.
just docs-test [PAGE] runs the pages whose path contains PAGE, and just docs-bless [PAGE]
rewrites the output of exact and #> blocks from the real tool. An excerpt is curated by hand, so
bless leaves it alone: the failure prints the full output to choose the lines from. A run block
is skipped with a note when GNU R is missing, or for runtime=ry, when libR.so is not built
beside ry. The landing page’s samples in docs/src/pages/index.astro are checked as excerpts with
typing on, without markers.
Package loading
Section titled “Package loading”crates/ry/tests/test_load.rs runs ry load and plain Rscript against a real GNU R and checks
the loaded result: Collate order, imports, S3 and S4 registration, Depends, sysdata.rda,
source references, reloads that rebind, rerun and remove, undoing deleted S4 definitions, the error
locations, and the ry load --r-entry entry point. test_repl_e2e.rs checks that ry R loads
the package it starts in and reloads an edit before the next command. Both skip without R.
Native packages
Section titled “Native packages”The libr tests load libR.so the way an R host does, build packages from source with the system C
and C++ compilers, load them with dyn.load (which runs R_init_<pkg>), and call their routines
through .Call, all through the exported C API. The shared harness is crates/libr/tests/harness.rs.
The packages are brio from the benchmark closure, vendored in tests/packages/brio, and
tests/packages/rytest, written for these tests.
test_native.rsbuilds both against ry’s headers and checks the results, collection stress, the handle scope,PROTECTimbalance, errors unwinding through C and C++ frames,R_allocrelease, finalizers, coercion, and the random number generator against GNU R’s output.test_interrupts.rschecks that an interrupt raised while interrupts are suspended, from R or from C, stays pending until the next interrupt check. It loads base R, which changes how a C callable resolves for every later test in the session, so it runs in a process of its own.test_gnu_r_binaries.rsbuilds both with GNU R’sR CMD SHLIB, against GNU R’s headers and linked to GNU R’slibR.so, and runs them under ry’s. It runs in a process of its own, so the two builds of one package never share a session.test_headers.rscompiles every header as C11 and C++17 with warnings as errors, checks with the compiler that every declared name is exported, runs BLAS and LAPACK calls through the headers, and compares the graphics structs’ layout with GNU R’s. It also requires every name intests/headers/r_api_names.txt(GNU R’s public header names that itsR.dllexports, plus the internals packages link) to be declared by ry’s headers and exported bylibR. WithRY_GNU_R_INCLUDEandRY_GNU_R_EXPORTSpointing at a GNU Rinclude/and a list of itsR.dllexports,api_names_cover_gnu_rchecks the list against that release, andRY_UPDATE_API_NAMES=1regenerates it.test_altrep.rsandtest_altrep_api.rsdrive thealtreptestpackage’s classes of every kind and base R’s own classes, against a report recorded from GNU R.test_serialize.rsandtest_serialize_native.rsread and write GNU R’s recorded streams byte for byte in every format and version, read installed lazy-load databases, and fuzz the reader.test_connections.rsruns one connection scenario list in GNU R and in ry and compares the transcripts;test_connections_gnu_binary.rsrepeats it with theryconnpackage built by GNU R.test_install.rsassembles ry’s R home and installs packages withR CMD INSTALLsemantics. WithRY_SOURCE_PACKAGESset to a directory of source tarballs, it builds each one against ry’s headers and reports the outcome.test_corpus_binaries.rsloads every compiled package of the installed corpus (RY_CORPUS_LIBRARY, default.corpus/library) under ry, each in a child process with its dependencies, and writestarget/tmp/corpus-binaries.tsv.test_corpus_calls.rsreplays native routine calls thattests/data/corpus_calls/record.Rrecorded under GNU R, reading arguments and results with ry’s unserializer, and requires every routine whose library loads to return what it returned under GNU R.
The two corpus suites pass without a corpus and report that they skipped.
These suites need a C and C++ toolchain, and GNU R for the second suite and the generator check.
The parts that need GNU R find it through the R on PATH, taking its headers from R CMD config --cppflags and its Rscript from PATH, so Debian’s split layout and a source build both work. When no R is on PATH, or RY_SKIP_GNU_R=1 is set, those parts print why and skip.
The language server
Section titled “The language server”crates/ry/tests/test_lsp.rs spawns the real ry server process and drives it over stdio with an
async-lsp test client. It covers:
- capability negotiation: position encodings, pull diagnostics, refresh, label offsets, and snippets;
- document sync, both incremental and full-text, including untitled and non-
file:buffers and re-reading from disk on close; - the two-wave push contract, its settled-superset invariant, and how a burst of edits settles under latest-edit-wins cancellation;
- pull diagnostics: result ids, unchanged reports, retryable cancellation, and push suppression;
- diagnostic refresh on save and on a configuration change;
- configuration: live reload, a configuration in an ancestor directory, a broken configuration keeping the previous one in force, and the diagnostic on the configuration file itself;
- every feature endpoint, including correct UTF-16 and UTF-8 ranges over BMP and non-BMP text, and safety for out-of-bounds positions;
- semantic tokens for a
#:body, and serving.RtypesandNAMESPACEbuffers.
The cancelled-pull test is made deterministic by a fault-injection seam. When the
RY_TEST_DELAY_PULL_MS environment variable is set, the server announces each diagnostics pull by
creating the RY_TEST_PULL_MARKER file, then holds the pull (for at most the delay) until the
cancellation token flips. The test sends its didChange only after the marker appears, so the
cancellation provably lands while the pull is in flight, and the retryable SERVER_CANCELLED
response and the successful retry after it can be asserted without sleeping and hoping. Nothing
else sets these variables.
crates/ry/tests/test_simulation.rs runs the in-process language-server simulator: random editor
sessions under a seeded schedule, checked at random points against a server started fresh on the same
disk and buffers. crates/ry/tests/test_scenarios.rs runs the scripted user journeys in
crates/ry/tests/scenarios/. Robustness documents both, including the
scenario format and the replay variables.
native_diagnostics_agree_with_the_typed_core in the same file holds the strict-mode
native function findings to what the native compiler
does. It checks each program of a corpus in strict mode, runs it with RY_TYPED_LOG=1, and fails
when a function without a native or interpreted finding runs in the interpreter. The corpus is
the typed benchmark programs, the friendly climb programs, and crates/ry/tests/native/*.R, where a
program’s header states what it shows: # blocked: f, g names the functions that must get a
finding, and # boundary: h the ones with an R type in their signature. Add a program there for a
construct the compiler gains or the checker starts to report. The test also fails when no natively
running function of the corpus calls a standard library function that
semantics::native::COMPILED_STD lists as compiled.
Incrementality
Section titled “Incrementality”A fixture checks what an edit produces, but not how much work it took to get there.
crates/semantics/tests/test_incremental.rs checks the work: semantics::testing::ProbeDatabase
logs every query execution, and checked_since names the items whose item_check ran. The tests
pin these contracts:
- A body edit that keeps the item’s scheme re-checks the edited item alone.
- A body edit that changes the scheme also re-checks the items that read it.
- A new definition re-checks only the items that read its name.
- An inserted statement re-checks only the reads it intercepts.
- A body edit re-derives one item’s load facts and its own file’s index layer, a
NAMESPACEedit re-derives no item, and a statement that only moved keeps its load facts. - An edit parses one file. A cold check parses each file once, also on a diagnostics-only host
whose store keeps two trees (
ProbeDatabase::diagnostics_only), and that host reports exactly what an editor host reports. - A statement added beside a recursive function checks only itself, and a name a script adds resolves no package item’s names again.
- A check resolves the names it spells through
item_linkswithout aframe_binderquery, and finds its cyclic groups with onefile_interface_groupswalk per file. - Under invariants, the links agree with the per-name answers, and each file’s groups agree with plain reachability from the item, which shares only the edges with the component walk (checked in a child process, because invariants are process-wide and would perturb the counting tests).
The same file checks that a chain of re-exports resolves to its base, that a cycle of them pins to
Unknown, and that a chain of 4,096 links resolves without overflowing the analysis thread’s stack.
In the IDE harness, point_queries_recheck_only_what_an_edit_changed holds the features that run on
every keystroke to the same rule.
Performance and memory
Section titled “Performance and memory”crates/semantics/tests/test_perf.rs asserts performance budgets over the real-file corpus. It is
ignored by default, because it needs the fetched corpus and a release build, and CI does not fetch
the corpus:
cargo test-release -p semantics --test test_perf -- --ignored --nocapturejust perf runs the same command. Its budgets are measured numbers plus headroom, and cover the
cold-pass wall time per line, resolve steps per line (the tripwire for a regression in resolve
memoization), resident bytes per line, and the p50 and p95 latency from a keystroke to fresh
diagnostics on the corpus’s largest package. It prints what it measured, so it doubles as the
corpus-wide measurement, while ry debug analysis-stats diagnoses one workspace phase by phase.
Run optimized tests with cargo test-release, an alias for cargo test --profile iterate, which is
what just perf uses. The iterate profile keeps opt-level = 3 but drops cross-crate LTO and
uses 16 codegen units (Build profiles). Plain cargo test --release builds
with the release profile’s fat LTO: building libr’s test_linpack that way peaks at 11.1 GiB of RSS
and takes 30 minutes on four cores (it was OOM-killed once on a 16 GB machine), against 3.6 GiB and
11 minutes without it. A test that needs another package’s binary (libR.so, ry-rcmd) builds it
with syntax::testing::cargo_build, a cargo build in the profile the test runs in, so it lands
beside that ry. The helper drops the variables cargo sets for the test (CARGO_PKG_NAME,
OUT_DIR and so on): build scripts such as ring’s rerun when those change, so a nested cargo build that inherits them rebuilds most of the dependency tree, and the next plain build rebuilds
it back. A budget measured under cargo test-release can differ from a release
binary’s, so a number you quote comes from a release build. Debug builds keep line tables only ([profile.dev] debug = "line-tables-only"), which
keeps backtraces symbolized; set CARGO_PROFILE_DEV_DEBUG=true to inspect locals in a debugger.
crates/ry/tests/test_perf.rs holds the memory budget of a whole cold ry check --no-cache with
type errors on. It generates the three 300,000-line benchmark code bases with
tools/bench/tooling/generate.py, runs the release binary on each, and fails when one peaks above
256 MiB resident. It reads the child’s own peak with wait4, so it measures the process the user
would run, allocator included. It is ignored by default for the same reasons:
cargo test-release -p ry-lang --test test_perf -- --ignored --nocapturejust perf runs both suites.
The hot suite: a runtime change measured in seconds
Section titled “The hot suite: a runtime change measured in seconds”The three benchmark code bases take 60 to 135 s each, too long to run after every change to the
runtime. The hot suite (tools/bench/hot/) replays the package calls that hold most of their time
and one round of it, on both engines, takes about 35 s:
just bench hot --rounds 3 --extra-rounds 0 # every program, both enginesjust bench hot --rounds 3 --extra-rounds 0 --filter rank| program | GNU R | ry | GNU R / ry | flags || --- | ---: | ---: | ---: | --- || testthat::expect_equal | 20.369 ms | 14.012 ms | 1.68x (1.45 to 1.84) | wide_interval (ry) || base::rank | 14.323 ms | 22.401 ms | 0.64x (0.64 to 0.71) | wide_interval (ry) |...Geometric mean of GNU R / ry over 100 programs: 1.43x (95% interval 1.39 to 1.49)Predicted GNU R / ry for finbench: 1.27x (95% interval 1.20 to 1.38; 83 programs carrying 100% of its weight)Predicted GNU R / ry for dsbench: 1.39x (95% interval 1.29 to 1.46; 78 programs carrying 100% of its weight)Predicted GNU R / ry for biobench: 1.56x (95% interval 1.37 to 1.63; 77 programs carrying 100% of its weight)A program is one package entry: a function that the code bases’ own code calls in a package,
such as stats::lm or zoo::rollapply, together with up to three of the calls they made to it.
Each call was captured with its arguments while the code bases’ tests ran, and the harness
(harness.R after replay.R) replays it as eval(call, env) in a loop. One process per engine
runs every program, so the packages load once. A program’s batch holds about 20 ms of GNU R time,
and its time is the median of five batches after one warm-up batch.
The number to watch is the predicted speed-up of each code base, not the geometric mean. Each program carries a weight per code base, its share of the code base’s GNU R time among the suite’s programs, and the prediction is the weighted harmonic mean of the programs’ speed-ups, which is what the code base’s own speed-up would be if it ran only these calls.
How the suite is generated
Section titled “How the suite is generated”just bench-hot-capture DIR regenerates suite.toml and samples/ from the code bases that
tools/bench/tooling/generate.py wrote to DIR. It runs the three steps of
tools/bench/tooling/hot.py:
| Step | Runs | Writes |
|---|---|---|
capture |
each code base’s tests on ry with the recorder on and the JIT off | per code base, the time in every package entry (entries.tsv, summary.tsv), up to eight calls of each with their arguments (samples.tsv, samples/N.rds), and the packages the code base imports (session.txt) |
validate |
every sample under GNU R (validate.R) |
the samples that replay with the result they had and no side effect outside their temporary directory, with GNU R’s seconds per call and whether each call needs a fresh copy of the arguments (validated.tsv) |
select |
nothing | the 100 entries that hold the most time, the three most typical valid samples of each, and their weights |
The recorder is part of the runtime (crates/interp/src/capture.rs) and is off unless
RY_CAPTURE_PACKAGE names the user’s code:
| Variable | Meaning |
|---|---|
RY_CAPTURE_PACKAGE |
comma-separated namespaces that count as the user’s code, or R_GlobalEnv |
RY_CAPTURE_DIR |
the directory the recorder writes to |
RY_CAPTURE_SAMPLES |
calls kept per entry (default 8), drawn uniformly from all of the entry’s calls |
RY_CAPTURE_MAX_BYTES |
the largest serialized argument set or result kept (default 65536) |
It splits the run’s time between the user’s code and the package entries it calls. Code is the
user’s when it is a closure of the user’s namespace, and also when it evaluates in an environment
whose top-level environment is the user’s, however it got there: a promise for an argument, a
block passed to test_that() or withr::with_tempdir(), an eval() from a package, or an
evaluation from C. A package entry therefore owns the time of the package code under it but not
the time of a user block it runs, and the calls that block makes are entries of their own. The
capture of the finance code base reads:
finbench: 9592 tests, 9592 passing in 152.7 s of test blocks 147.5 s on ry: 3% in the code base's own code, 83% in package entries it captured, 13% in entries it could not capture, 1% outside both; 355234 calls into 198 entriesA call cannot be captured when a replay could not rebuild it: it passes its caller’s ..., an
argument names a user function or a local closure, the call reads its caller’s frame, or the
recorder failed to inspect it (the reason uninspectable, for example when forcing a lazily bound
package definition throws). The recorder counts its own failure as this reason and the program
carries on.
Such a call’s time counts for its entry all the same, in the uncapturable_* columns.
The samples of an entry are a reservoir of RY_CAPTURE_SAMPLES slots drawn uniformly from all of
its capturable calls. The recorder serializes only the calls the draw selects. A selected call
whose arguments serialize to more than RY_CAPTURE_MAX_BYTES empties its slot rather than leave
the earlier sample there, because keeping it would bias the draw toward small calls, and the
oversize column counts these calls. A slot a later draw refills holds a sample again, so
samples counts the slots holding one at the end. A result over the limit is dropped and its
sample is kept, so validation checks that sample only against GNU R’s own two replays.
The recorder writes every time as an integer number of nanoseconds, with the unit in the column
name: nanoseconds and uncapturable_nanoseconds in entries.tsv, nanoseconds in
samples.tsv, and total_nanoseconds, user_nanoseconds, covered_nanoseconds,
uncapturable_nanoseconds and outside_nanoseconds in summary.tsv. Its own work (inspecting a
call, serializing a sample, writing a file) is left out of every record’s time and of the total.
The validator then drops the samples that depend on the world outside the call (a file the capture
read, a git repository, external pointers, options, a graphics device) or that print. select
reports what the suite loses to both:
finbench: the programs hold 79.6% of the run's time on ry (74.4% in calls the recorder could capture) and 82.6% of the package time lost, with no valid sample: NlcOptim::solnl (2.41%), openxlsx::saveWorkbook (2.20%), R_GlobalEnv::test_that (1.61%), ...A program’s weight comes from a ratio estimate, because the capture measures ry’s time and the weight needs GNU R’s. The time a code base spent in an entry is scaled by GNU R’s seconds over the capture’s seconds for the same sampled calls. The samples are a uniform draw of the entry’s calls, so what the switched-off JIT adds to the entry’s time cancels in that ratio.
What a replay controls
Section titled “What a replay controls”-
The session. Every replay first loads every package the code bases import (
sessioninsuite.toml) and sets testthat’s third edition, because loading a package changes what other calls do:zoo::rollapplyon an xts series pads withNAonce xts is loaded, and ompr finds a solver only afterROI.plugin.highsis. A package that is not installed is skipped and named in the output. -
Conditions. A call runs under handlers that continue a testthat expectation and muffle messages and warnings, so an expectation replays outside a
test_that()block. -
Random numbers. A sample that draws random numbers gets its seed restored before each call.
-
Temporary paths. A path the capture made under its session’s temporary directory (
/tmp/RtmpXXXXXX/...) is rewritten to lie under the replay’s owntempdir()byreplay.R’srelocate, which the validator and the harness both call, and the parent directory of each such path in the call’s arguments is created. The calls run in a directory undertempdir(), so a relative path lands there too. A sample may create files only under its temporary directory, and a second call must leave the same set of paths as the first; the validator drops any other sample with its reason, and drops a sample that opens a graphics device. -
Fresh arguments. The validator replays each sample once with the JIT off and compares its variables before and after. A sample whose call changed them is marked
fresh: data.table’s:=modifies its table in place, an R6 setter such as lgr’s$set_appendersits object,foreach(i = ...) %do%assignsiin the calling environment, and httr2’sresp_body_jsonfills the response’s cache. The harness rebuilds a fresh sample’s environment from a serialized copy for every call. It builds them before each batch, outside the timing and outside the program’s region, so a batch times only the calls. -
Checksums. Each engine prints a checksum of every sample’s last result, and the run fails when ry’s differs from GNU R’s. The checksum is
<kind>:<length>:<hash>:<magnitude>:<balance>. The hash covers what must match exactly: every type and length, the attributes (all but data.table’s.internal.selfref, at any depth), the text, the logical values, and how many NA, NaN, Inf and -Inf each numeric vector holds. Every numeric vector at any depth adds the logarithm of its weighted size, and a sum under a second set of weights over that size, to the last two fields, which must agree to an absolute 1e-10 (or a relative 1e-13). A root thatunirootfinds one part in 1e13 away therefore agrees, and a change that moves a vector’s sums by one part in 1e10 does not, so one wrong element of a long vector shows only when it is that large a part of the whole. A sample whose result differs on every call under GNU R (a temporary file name, a timestamp) is recorded asvariesand not compared. A result of another type fails even when its values agree. An engine whoseseq.int(1, 18, by = 1)is a double vector, where GNU R’s is an integer vector, prints:FAILED hot/base-seq on ry: checksum dbl:18:32x252214378:2.131601681765849e+01:1.626932262278126e+00,...,GNU R printed int:18:33x270404436:2.131601681765849e+01:1.626932262278126e+00,...
Where a program’s time goes
Section titled “Where a program’s time goes”Two passes run on ry alone after the timed rounds, and --rounds 0 runs them by themselves. Both
rely on .ry_bench_region(name), a builtin that ends the open benchmark region and starts the
next, which the harness calls around each program’s batches.
--breakdown attributes the runtime’s profiler samples to the region they fall in and prints, per
program and weighted per code base, the share of ry’s time in each activity:
| program | samples | interpreter | builtin | call_setup | native | evaluator | s3_dispatch | compiled | jit_compile | ...| testthat::expect_equal | 429 | 48% | 29% | 12% | 2% | 4% | 2% | 0% | 0% | ...| base::mean | 575 | 20% | 19% | 10% | 0% | 9% | 12% | 4% | 25% | ...| finbench, weighted by GNU R time | | 41% | 32% | 10% | 8% | 2% | 4% | 0% | 0% | ...| dsbench, weighted by GNU R time | | 41% | 27% | 11% | 14% | 3% | 2% | 0% | 1% | ...| biobench, weighted by GNU R time | | 37% | 32% | 11% | 9% | 4% | 4% | 0% | 1% | ...After timed rounds each program weighs by its share of the code base’s ry time. A run that times
nothing (--rounds 0, as here) has no speed-ups to derive that share from, so it weighs by the
GNU R weight and the row says so.
interpreter is the bytecode interpreter, builtin base R’s builtins, call_setup a closure
call’s argument matching, promises and frame, native code reached through .Call, .External
or .C, compiled machine code the JIT produced and jit_compile the JIT compiling it. A
program with a large interpreter share gains from the JIT, and one with a large builtin share
gains only from a faster builtin, so the breakdown bounds what an optimization of one activity
can give. A compilation that falls inside a program’s batches counts in full, as it does in the
instruction counts, which is why a short program such as base::mean shows a quarter of its
samples there.
--instructions runs one batch of each program under callgrind, with instrumentation on only
inside the program’s region, and prints each program’s instruction count per call. Callgrind
writes a dump each time a region ends, and the dumps of one region are summed.
| program | calls | ry, batch | per call || testthat::expect_equal | 20 | 45,452,106 | 2,272,605 || openxlsx::write.xlsx | 3 | 85,865,843 | 28,621,948 |...Weighted, finbench, ry: 3,292,951,540 instructions per second of GNU R workWeighted, dsbench, ry: 2,572,317,861 instructions per second of GNU R workWeighted, biobench, ry: 2,613,663,739 instructions per second of GNU R workThe weighted total of a code base is the number to compare between two builds. It is the instructions ry spends per second of GNU R time in that code base’s programs: each program’s instructions per call, over GNU R’s seconds per call, weighted like the predicted speed-up.
The pass deals the programs over four harness processes that run side by side, and the whole suite takes about 80 s. The split is by position in the suite and not by the machine’s core count, so a program shares its process with the same programs on every machine. Each process runs on one OpenMP thread, because callgrind counts the instructions idle worker threads spend spinning, which vary from run to run: a grouped data.table assignment counted 80 to 96 million instructions on four threads and 45.8 million on one.
Three options build on the counts:
--save-instructions FILEkeeps them, and a later run with--baseline-instructions FILEprints each program’s change and its effect on each code base’s weighted total, largest effect first.--jit-offcounts ry withRY_JIT=offas well and prints the same comparison between the two, which shows what the JIT costs or saves each program.--functions Nadds the N functions of the runtime and of the packages’ compiled code that hold the most of each code base’s weighted instructions. It needs a build with symbols (cargo build --profile profiling, passed with--ry PATH --any-build).--save-dumps DIRkeeps each program’s callgrind dumps asDIR/<program>.<n>.callgrind, for a question the function ranking does not answer, such as who calls a function or how often.
What one mechanism costs
Section titled “What one mechanism costs”just bench-mechanisms prices the mechanisms the suite’s programs are made of. Each of about
seventy operations is one expression (a closure call with a default argument, a UseMethod
dispatch, a tryCatch that catches nothing, plain$b) that a compiled loop evaluates. The tool
runs the loop under callgrind for 20,000 and for 40,000 iterations and reports the difference per
iteration, less the empty loop’s, so start-up and warm-up cancel and the count repeats exactly.
The tier-1 JIT is off unless --jit is given, because package code runs in tier 0 almost all of
the time.
just bench-mechanisms --ry target/release/ry --filter dispatch| operation | expression | instructions || --- | --- | ---: || dispatch: `UseMethod`, class method | `describe(object)` | 6,437 || dispatch: `UseMethod`, default method | `describe(i)` | 7,067 || dispatch: `UseMethod` then `NextMethod` | `describe(child)` | 14,104 || dispatch: builtin to a method (`length`) | `length(object)` | 3,768 |...The whole table takes about five minutes on four cores, and one filtered group well under a
minute. --save FILE keeps the counts and --baseline FILE prints the change against them,
which is how a change to the call path or to dispatch is checked before the suite is run.
--profile NAME prints the functions that hold one operation’s instructions, on a build with
symbols.
How well the suite predicts the code bases
Section titled “How well the suite predicts the code bases”just bench-hot-track DIR NAME=RY[,VARIABLE=VALUE...]... measures it. It times the suite and the
code bases’ tests on several ry configurations (other builds, or one build with a feature off) and
compares the change the suite predicts with the change the code bases show, each against the
first configuration:
just bench-hot-track .bench/codebases head=target/release/ry nojit=target/release/ry,RY_JIT=off r043=/tmp/ry-0.4.3/rydsbench (time over GNU R's; change against head; ...)| configuration | predicted | measured | predicted change | measured change | weight | tests || head | 0.717 | 0.596 | +0.0% | +0.0% | 100% | 9416 tests, 9416 passing || nojit | 0.693 | 0.553 | -3.3% | -7.2% | 100% | 9416 tests, 9416 passing || r043 | 1.180 | 0.945 | +64.7% | +58.4% | 100% | 9416 tests, 9416 passing |...dsbench: r = 0.986 between the predicted and the measured log changes of 11 configurations; a predicted change shows up 0.85 times in the measurementA study of twelve configurations gave these correlations. The configurations were the build the suite was generated on, that build with the JIT off, with the builtin fast paths off and with native methods off, the same build a second time, the build before the recorder, and six releases back to 0.4.0-beta, whose predicted times range from 13% under the first configuration’s to 124% over it.
| Code base | r | A predicted change shows up |
|---|---|---|
| finbench | 0.987 | 0.78 times |
| dsbench | 0.986 | 0.85 times |
| biobench | 0.987 | 0.90 times |
| all three | 0.976 | 0.85 times |
The three code bases share the configurations, so the last row pools 33 pairs that come from eleven changes.
The suite therefore ranks builds the way the code bases do, with three limits.
- A predicted change is larger than the measured one, because the programs hold 77 to 80% of a run’s time on ry and the rest of the run changes less.
- The predicted level is pessimistic. The suite predicted 1.34x, 1.39x and 1.47x for the first configuration where the code bases measured 1.62x, 1.68x and 1.62x. What makes the difference is not established, so compare predictions with each other and not with a code base’s time.
- Wall time on a shared machine cannot show a small change. The study ran the same build
twice, 100 minutes apart, and the second run was 14 to 16% slower on the code bases and 21 to
27% slower in the suite, because the machine’s speed drifted. Within one run, one program’s
median batch time varies by about 20% between identical runs on either engine. Time two builds
in the same rounds (
--baseline PATH) and read the predicted speed-ups, not a single program’s ratio. For a change under about 10%, compare instruction counts.
How much to trust an instruction count
Section titled “How much to trust an instruction count”Two instruction passes over the same build agree to 0.01% on each code base’s weighted total
(finbench 3,294,004,993 and 3,293,995,120). The median program differs by 0.013% and the worst,
brio::write_lines, by 0.41%, so a change of 1% in one program is visible.
A change in the suite’s count shows up in a code base’s count with the same sign. The whole test run of dsbench under callgrind, which takes about 65 minutes, cost 318.11 billion instructions, 317.45 billion with the JIT off and 322.91 billion with the builtin fast paths off:
| Change | Suite’s total at dsbench’s weights | dsbench’s tests under callgrind |
|---|---|---|
| JIT off | -0.27% | -0.21% |
| Builtin fast paths off | +2.95% | +1.51% |
The code base’s count covers the whole process, with start-up and package loading, and each configuration ran once, so how closely two runs of one build agree is not measured.
By default the runner adds ten rounds when any program’s interval is wider than 10% either way,
which on a shared machine is every time, so pass --extra-rounds 0 for a quick look.
Comparing two builds on testthat
Section titled “Comparing two builds on testthat”The testthat workload (tools/bench/applications/testthat.R) gives two different wall times, and they
answer different questions:
| Number | What it covers | Command |
|---|---|---|
| Test phase | one warm run of the tests, the median of RY_BENCH_REPEAT runs |
field 3 of the output of RY_JIT=on RY_BENCH_REPEAT=5 ry run --runtime ry testthat_full.R |
| Process wall | start-up, loading testthat, one cold run and five warm runs | time around the same command |
| Load | start-up and loading testthat only | time ry run --runtime ry load.R, where load.R holds only suppressPackageStartupMessages(library(testthat)) |
testthat_full.R is harness.R followed by testthat.R, and its checksum is 27:22821. Both
builds use the same R_LIBS, a library that ry itself installed testthat into, and each number is
the median of five fresh processes. On the Linux release build with the JIT on, the test phase
measured 0.219 s and the process wall 1.44 s, which is how one workload can be quoted as both. The
comment at the top of tools/bench/harness.R gives the full environment of each command.
Benchmarks covers the published suites and how they are timed.
Guidelines
Section titled “Guidelines”- Add or tighten a fixture before writing a parser-local or crate-local unit test.
- A new phase or module gets a fixture suite before it relies on ad hoc unit tests.
- Use the lightest fixture change that captures the failing shape.
- Change a fixture expectation deliberately, never as a side effect.
What earns a test its place
Section titled “What earns a test its place”A test must pin a contract the docs state or a bug that once happened. The suite is judged by the bugs it can catch, not by its size, so a test that cannot fail for a plausible bug is deleted rather than kept for safety.
Keep:
- fixture suites, oracle cases, scenarios, simulations and every fuzz, property or differential harness, with their pinned regression inputs;
- a regression test for a real bug, even a narrow one, when no fixture or harness would catch the bug coming back;
- a test of behavior a user or another crate can observe, which no fixture, oracle case or harness already covers.
Cut:
- a test that restates the implementation: it asserts a private data layout, an internal constant, or the exact steps the code takes, and so breaks on a refactor that keeps the behavior;
- trivia: a constructor returns what it was given, a getter reads a field,
DebugorDisplayof an internal type has a certain spelling, a default is its default; - a duplicate: the same behavior is already pinned by a fixture, a scenario, a harness, or another
unit test one line over, or by an oracle case that an automated run executes ry against. Neither
a plain
cargo testnor CI compares ry’s output with the recorded oracle cases: they replay the recordings under GNU R, and the collection-stress pass compares ry only with itself, so an oracle case alone does not cover a test of ry’s behavior; - a test of the test harness, unless the harness’s own failure would let real failures pass (a filter that matches nothing, a duplicate case name);
- a snapshot of incidental output that no contract promises;
- a slow test that finds nothing a faster one would not.
Fold before deleting. When a group of small unit tests covers one behavior, one fixture case, one oracle case or one table-driven test replaces them. The only coverage of a behavior is never dropped: it moves to the cheapest form that still catches the bug.