Robustness
ry’s long-run priority is robustness. A compiler, a language server and a runtime that most of the time give the right answer are not good enough, because the rare wrong answer is the one a user cannot diagnose. So every component is designed so that a machine can drive it with generated input, check it against an independent answer, and reproduce any failure from one number. This page is the contract for that design: the rules every component follows, and what each layer’s harness checks today. The fixture contract and the individual fuzz batteries are documented in Testing.
The approach borrows from deterministic simulation testing as FoundationDB, TigerBeetle and Antithesis practice it, and from coverage-guided and differential fuzzing. Three kinds of harness work together:
- Simulation runs a whole component (today, the language server) in one thread under a seeded scheduler, with every source of nondeterminism behind a seam the seed drives, and checks it against a component started from scratch.
- Fuzzing feeds a stage generated or mutated input and checks properties that must hold for every
input. Seeded batteries run inside
cargo test; the same batteries back the coverage-guidedcargo fuzztargets. Differential fuzzing compares two implementations of the same semantics, such as ry’s runtime against GNU R. - Scenarios script a realistic user journey (open a package, type, rename, run
ry check, install a package) in a small readable format, so a new journey costs a few lines.
The rules
Section titled “The rules”Every source of nondeterminism sits behind a seam. Time, randomness, IO, the file system,
threads, scheduling and allocation must be injectable, so a harness can drive them from a seed. A
component that reads the clock, spawns a thread or iterates a hash map in a way the output can
observe is a bug, even if it never misbehaves in production, because it makes failures
unreproducible. Ordered collections (BTreeMap, BTreeSet) are the default wherever iteration order
reaches the output.
Every failure reproduces from its seed. Harnesses draw all randomness from
syntax::testing::SplitMix64 (or ByteStream, which reads a fuzzer’s bytes through the same
Entropy trait), print the seed and a replay command when they fail, and never consult the
wall clock or the OS’s randomness. Running the same seed twice must produce the same transcript, and
the simulator pins that property with a test. A seeded battery loops through
syntax::testing::for_each_seed, which runs seeds 0..N and on a failure prints the exact command, such as
FUZZ_SEED=14 cargo test -p loader --test test_reload_fuzz a_reload_matches_a_fresh_load -- --exact.
Three variables move the window. FUZZ_SEED alone replays that one seed. FUZZ_ITERS sets the
count, starting at FUZZ_SEED or 0. FUZZ_DEPTH multiplies each loop’s own default count instead,
so a test with several loops keeps their proportions when a campaign deepens it. Each case builds
its generator from its seed alone, as SplitMix64(CONSTANT ^ seed), because a generator shared
across cases makes a replayed seed draw different inputs and a deeper round rerun the same ones.
for_each_seed takes the plain default count: wrapping it in fuzz_iterations applies
FUZZ_DEPTH twice.
Invariants are always on in simulation. Cross-checks that are too expensive for production run
whenever a harness asks for them, and panic at the step that broke them with the inputs that
reproduce it. syntax::testing::enable_invariants() switches them on for the process. Today it makes
every splice reparse also parse from scratch and compare the two trees. A new incremental shortcut
gets such a cross-check the day it lands.
The oracle is a from-scratch run. Incremental results are checked against the same computation done from nothing: a fresh salsa database, a fresh language server on the same disk and buffers, a full parse. Runtime results are checked against GNU R and against ry’s own tier-0 interpreter.
A bounded pass of every harness runs in the default test suite. FUZZ_ITERS scales every
budget, so cargo test stays fast while a long run is one environment variable away. Long runs have
just recipes.
A failure becomes a regression case. A bug a harness finds is fixed together with a pinned
input: a fuzz_regressions_hold_invariants entry, a splice regression, a fixture, or a scenario.
A bug that is not small and local is filed in .agents/memory/backlog.md with its reproducing seed.
A harness and its pinned inputs are never cut for being slow or quiet. The rule for which tests earn their place is in What earns a test its place: unit tests that duplicate a harness go, and the harness stays.
The language-server simulator
Section titled “The language-server simulator”crates/ry/src/server/simulation.rs runs the real server, the same ServerState and Worker code
that serves an editor, without its worker thread or socket. Two seams in server.rs make that
possible:
- The outbox. Everything the worker sends the client (diagnostics, messages, capability
registrations, refresh requests) goes through an
Outbox. In production it is the socket; in a simulation it records the messages in order. - The job step.
run_jobis one step of the worker loop: initialize, refresh cancellation, then run one queued job or one unit of idle work. The production loop calls it on its thread; the simulator calls it whenever the seed says the worker gets to run, so the seed decides how notifications, requests and idle work interleave.
Cancellation is injected the way a real edit causes it. cancel_at_checkpoint makes the worker’s
cancellation token flip at the Nth cancellable checkpoint of the current job, and the simulator
always follows the flip with a real notification, because the server relies on a newer edit being
queued whenever a query is cancelled.
run_session(entropy, root, actions) plays a random editor session against a package workspace:
- edits, including several ranged changes in one notification, typing bursts, full-text replacements, emoji and CJK text, and UTF-8 or UTF-16 position encodings;
- opening, closing and saving files, untitled buffers, and an R Markdown document whose positions count on the document’s own text, not on the R script its chunks make;
- disk changes behind the editor’s back to R files,
DESCRIPTION,NAMESPACE,ry.toml, data files and test files, with watcher events delayed, duplicated, and delivered in shuffled batches; - requests of every kind at valid and invalid positions, abandoned requests, and requests that race edits;
- protocol violations, such as changing, saving or closing a document that was never opened;
- push and pull diagnostics clients, and sessions whose first actions arrive before the startup warm-up has run.
At random checkpoints the session lets everything settle and checks the server against a fresh one started on the same disk with the same open buffers:
- every outstanding request got a response;
- the server’s text for each open document is the client’s;
- every analysis input (files, kinds, project order, configuration, package metadata, data-file layouts) equals the fresh server’s;
- diagnostics and random requests at random positions answer exactly as the fresh server does;
- every range in every response lies inside the document, in the negotiated encoding;
- the push contract holds: the last push for an open document is the settled wave, it contains the first wave when nothing else changed in between, and it equals a pull.
The bounded pass is cargo test -p ry-lang --test test_simulation. just simulate 1000 120 runs a
thousand seeds of 120 actions, and SIMULATION_SEED=<seed> replays one, printing the session’s last
actions on failure.
The workspace is a real directory that only the simulator writes. An in-memory file system with injected IO errors is not built yet, so read failures and partial writes are not simulated.
Scenario tests
Section titled “Scenario tests”Each crates/ry/tests/scenarios/*.scenario file is one journey. Editor steps drive the language
server in-process through the simulator and settle after every step; command-line steps run the real
ry binary in the scenario’s workspace. The in-process server trusts the check records that binary
writes, as a real server trusts its own, so a journey can run ry check and then open the project.
When a journey ends, its editor session is checked against a
fresh server, exactly as a simulation checkpoint is. A scenario reads like this:
# Renaming from a call site rewrites the definition in a file that is not open.file R/area.R| area <- function(width, height) width * heightfile R/report.R| report <- function() area(2, 4)
open R/report.Rrename R/report.R "area" surfaceexpect buffer R/area.R "surface <- function"
run ry checkexpect exit 0One command sits on each line, # starts a comment, and | lines form the body of the command
above them. Quoted words may hold spaces and the escapes \n, \t, \" and \\.
| Command | Meaning |
|---|---|
requires gnu-r / requires ry-runtime / requires r-package:NAME[>=VERSION] |
Skip the scenario unless GNU R is installed with its shared library (--enable-R-shlib), libR.so is built beside ry and ry’s base R is built (just base-r), or the package loads, at VERSION or later (packageVersion), in every runtime the scenario requires: ry’s runtime for ry-runtime, and for gnu-r the GNU R that --runtime gnu-r picks. The skip names the runtime and, for an old package, the installed version. A misspelled requirement fails the scenario. |
hide gnu-r |
Run every later ry with no GNU R: R and Rscript vanish from PATH while everything beside them (such as sh in /usr/bin) stays, and R_HOME is unset. |
ignore SIGINT |
Start every later run with SIGINT ignored, as a background job of a non-interactive shell starts. |
client pull / client utf8 / client cold |
Start the editor as a pull-diagnostics or UTF-8 client, or one whose first steps arrive before the startup warm-up has run. Must come before the first editor step. |
file PATH + body |
Write a file; a running editor sees a watcher event. $ROOT in the body is the scenario’s workspace. |
stdin PATH |
Hand the next run that file as its standard input, which is how a scenario types into a console. |
env NAME VALUE |
Set a variable for every later run; $ROOT in the value is the scenario’s workspace. |
unset NAME |
Remove a variable, including one the test process inherited, for every later run until an env sets it again. |
remove PATH |
Delete a file, with a watcher event. |
open, close, save PATH |
Buffer lifecycle. |
type PATH after "A" "T" / type PATH at end "T" |
Type text one character per change, without settling in between. |
replace PATH "old" "new" / erase PATH "text" |
One ranged change at the first occurrence. |
rename PATH "anchor" NEW |
Rename the symbol at the anchor and apply the edit, opening files it touches. |
format PATH |
Format the buffer and apply the edits. |
expect diagnostics PATH none / expect diagnostics PATH "needle" |
No diagnostics, or one whose code or message contains the needle. |
expect no diagnostic PATH "needle" |
No diagnostic contains the needle. |
expect early diagnostics PATH "needle" / expect no early diagnostic PATH "needle" |
The first diagnostics the server pushed for the file do, or do not, contain the needle. |
expect hover PATH "anchor" "needle" |
The hover at the anchor contains the needle. |
expect definition PATH "anchor" TARGET "text" |
Goto-definition lands at the text in the target file. |
expect completion PATH after "anchor" LABEL |
Completion after the anchor offers the label. |
expect buffer PATH [lacks] "needle" / expect file PATH [lacks] "needle" |
Buffer or disk content. |
expect signature PATH after "anchor" "needle" |
Signature help after the anchor contains the needle, with the active parameter in brackets. |
expect symbol PATH "a/b/c" |
The outline has that path of nested symbols. |
expect workspace-symbol QUERY PATH NAME |
The workspace symbol query finds the name in the file. |
expect fold PATH "from" "to" |
A fold spans the lines of the first occurrences of both needles. |
expect selection PATH "anchor" "text" |
Expanding the selection from the anchor reaches exactly the text. |
expect lens PATH "anchor" "title" |
A code lens on the anchor’s line has that title, after resolving it. |
expect token PATH "anchor" CLASS |
The semantic token at the anchor has that type, with modifiers as +name (function+defaultLibrary). |
fix PATH "anchor" TITLE |
Apply the quick fix with that title at the anchor. |
refactor PATH "selection" TITLE |
Select the first occurrence of the text and apply the code action with that title. |
expect [no] action PATH "selection" KIND TITLE |
Asking for the code actions of that kind (refactor.extract, or all) over the selection does, or does not, offer the title. |
debug start PATH [entry] [prelude "R"] [test "name"] [break N | break FILE:N]... |
Launch ry debug-adapter on the script, with breakpoints in it or in another file. |
debug continue / next / step-in / step-out / disconnect / evaluate "expr" |
Drive the debug session. |
expect stopped REASON at N / expect stack "f@2 g@7" / expect variable NAME "value" |
The next stop, its frames, and a variable of the top frame. |
expect evaluated "value" / expect debug-output "needle" / expect exited N |
The last evaluation, the program’s output so far, and its exit status. |
run ry ARGS... |
Run the ry binary in the workspace. |
run ./PATH ARGS... |
Run an executable the scenario wrote, such as one ry build --aot --exe built, with the environment run ry gets. |
install DIR |
Install a source package through ry’s R CMD INSTALL into library/. |
spawn ry ARGS... |
Start the ry binary in the workspace and leave it running, as a watcher runs. |
await stdout|stderr "needle" [before "other"] |
Read the spawned process’s output until a line contains the needle, failing if none does within two minutes, or if a line with other comes first. |
stop |
Kill the spawned process and everything it started; its output becomes the last run’s, for expect stdout|stderr. |
expect exit N / expect stdout|stderr [lacks] "needle" |
Check the last run, install or stop. |
cargo test -p ry-lang --test test_scenarios runs them all, and just scenarios <name> runs one.
A scenario that requires an R package loads it from the corpus library (just corpus, in
$RY_CORPUS_DIR/library, else .corpus/library) when every package in it was built by the R
release ry’s runtime presents and, for a gnu-r scenario, by the release of the GNU R it runs. The
runner then sets R_LIBS and R_LIBS_USER to that library for every ry and R the scenario
starts. Other scenarios see the caller’s variables, so the ones about library defaults test the
defaults. A library built by another release is not used, because library() would warn. The
runner prints the library it uses, or why it uses none, before the first scenario. To use another
library, set R_LIBS yourself; R_LIBS= (empty) turns the default off.
Each layer
Section titled “Each layer”The table lists what drives each layer and what checks its answers. A layer marked as a gap has no harness of that kind yet; the gap is filed in the backlog.
| Layer | Driven by | Checked against |
|---|---|---|
| Parser | generated programs, token soup, byte mutations, edit streams (syntax test_fuzz); cargo fuzz parse |
lossless round trip, tree invariants, splice reparse equals full parse |
| Lowering, naming, inference, diagnostics | generated programs and projects (semantics test_fuzz); cargo fuzz semantics |
no panic, determinism, well-formed ranges, incremental equals fresh |
| Incrementality and cancellation | the simulator’s edits and injected cancellations; test_incremental |
a fresh server; per-item firewalls |
| Formatter | seeds, mutations, fixture sources (format test_fuzz, test_format_range); cargo fuzz format |
preservation, idempotence, determinism; for range formatting also convergence, restriction to the whole-file edits, additivity and composition |
| IDE features | generated documents and positions (ide test_fuzz); cargo fuzz ide |
no panic, ranges inside the document |
| Language server | the simulator; scenarios; cargo fuzz lsp |
a fresh server, the push contract, range geometry |
| NAMESPACE conditionals | generated if/else directive trees over R version and platform tests (semantics test_fuzz namespace_conditionals_resolve_to_the_taken_branches) |
the directives of the taken branches, evaluated by a model |
| Loader | random reload sequences that edit R files, NAMESPACE (with if blocks and S3 registrations), DESCRIPTION and data (loader test_reload_fuzz); cargo fuzz reload, load_index |
a fresh load; a model of the namespace’s bindings and S3 table; every name whose call graph reaches changed code, directly or through S3 dispatch, is among the callers ry test selects; the test-watch scenario |
| Deferred S4 | first-use probes (dispatch, is(), initialize with callNextMethod, primitives, show, objects read back with readRDS) in fresh processes (ry test_load); tools/loader-bench/s4_check.R on the seeded just bench-s4-load package |
the same probes with options(ry.load.lazy_s4 = FALSE), and GNU R |
Package builds (R CMD under ry load) |
generated makefiles with recursive, simple, appended, conditional and self-referencing variables, continuations, escapes and include lines (rcmd make_expansion_agrees_with_gnu_make); GNU R’s own Makeconf |
GNU make’s expansion of the same variables |
| Help server | generated requests written back to back on one connection, and their byte mutations (interp help_server tests) |
the parsed request round trips; percent-decoding and query splitting match tables recorded from GNU R |
| Debugger | scripted sessions and a seeded client that sends random requests, including ry’s own ry* requests, at every stop (frames test_debugger); valid and mutated message framing (ry test_debug_framing); simulated editors attaching to the console, reinitializing and racing each other (frames test_console_server); scenarios |
transcripts; every request answered once, the session ends, the interpreter left clean; frames read back exactly, a malformed or oversized frame is refused without a panic or an unbounded allocation |
| Interpreter | generated programs, including reads and writes of attributes, names and elements and c() calls that compiled code sends to the builtins’ direct entries (interp test_fuzz); closures that write elements, rows and columns of an argument whose length sits at a storage threshold, called three times (fuzz_replacements_agree_across_repeated_calls); differential fuzzing (oracle fuzz) of five kinds of program: generated grammar programs, including element stores into argument copies of 64 elements and more, interpreted and compiled; metamorphic trials, where one body on one value from a zoo of types, sizes, names and encodings must give an identical result however often and wherever it runs (fuzz_metamorphic_forms_agree_across_tiers); builtins on the same zoo (fuzz_builtins_agree_across_tiers); calls of functions drawn from every export of base, stats and utils with arguments of any type (fuzz_api_calls_agree_across_tiers); and mutated snippets of real R code (base help examples, R’s regression tests, the oracle cases), compared only where the unmutated snippet agrees (fuzz_corpus_mutants_never_crash is the bounded pass), each compared through a canonical dump of every top-level value and condition; code images: generated code round-tripped, mutated image files and mutated-but-rehashed payloads loaded (fuzz_code_image_round_trip, fuzz_code_image_file, fuzz_code_image_load); cargo fuzz interp_lower, bytecode, interp_code_image, interp_code_image_file |
GNU R, tier 0; the same statements run once at top level, and the caller’s value unchanged; for code images the original code object, the evaluator’s value, and an error or a warning, never a panic |
| Native package functions | seeded random frame stacks of closures, operators, ::, do.call, lapply, vapply, Reduce, handlers, restarts, eval, local, Recall, S3 dispatch and trimming (fuzz_native_trace_back_matches_its_r_body in oracle test_differential, at tier 0 and under the JIT, FUZZ_SEED replays; the conditions__rlang_trace_back_generated oracle case) |
the function’s own R body on the same frame, by identical(); GNU R, through the oracle case |
| Environment variables | seeded random Sys.setenv, Sys.unsetenv and Rust-side writes over prefix-sharing, empty, NA, =-holding and non-ASCII names and values (interp test_fuzz fuzz_environment_variables_agree_with_the_c_library); the system__sys_env_round_trips oracle case |
the C library’s environment, on every Sys.getenv, Sys.setenv and Sys.unsetenv result; GNU R, through the oracle case |
| Native helpers (withr, base) | seeded random calls of withr’s environment and option helpers, Sys.getenv, Sys.setenv, Sys.unsetenv, intersect and structure(names =) with NA, empty, unset and non-ASCII values (fuzz_native_helpers_match_their_r_bodies in oracle test_differential, at tier 0 and under the JIT, FUZZ_SEED replays) |
each function’s own R body, by identical() on the value, visibility, condition and environment and option state; GNU R, through the oracle cases |
| Graphics renderer | random drawings with nested, dangling and wrong-kind definitions and non-finite geometry (interp test_fuzz); the graphics__ and grid__ oracle cases; the plot gallery against GNU R (oracle test_gallery) |
no panic, identical bytes on a second render, one file per page, and files a reader parses (PDF cross-reference offsets and page count, balanced SVG markup, a decodable PNG, BMP, TIFF, JPEG and PostScript headers); GNU R’s warnings and capabilities |
| Garbage collector | seeded allocation and mutation sequences (heap test_gc), including string and symbol interning in every encoding; RY_GC_STRESS and RY_GC_STRESS=young over the oracle cases and generated programs; RY_GC_STRESS=seed:N with RY_GC_VERIFY over the cases that run native code, and over cases evaluated twice in one process; native code permuting a list or character vector through every element pointer accessor (libr element_pointers_keep_permuted_young_elements_alive) |
a model of the live graph; reference HashMaps of the interned strings and symbols; the unstressed run, the write-barrier invariant, a full collection over every reconcile’s frees |
| C API | native fuzzing (libr test_fuzz_native); mutated serialized streams (rdb mutated_streams_never_panic); cargo fuzz unserialize |
GNU R’s behavior; an R error, never a panic; the PROTECT balance check on every .Call |
| JIT | test_tiers, including generated typed closures over vectors, records, tuples, tags and match that the first call compiles (fuzz_typed_closures_agree_with_tier_zero); random lists through every library traversal the JIT loops (fuzz_stdlib_traversals_agree_with_model); random comparison, &, ` |
and!programs over lists and scalars withNA, NaN and length mismatches through the fused mask count (typed test_typed values::fused_counts_follow_three_valued_logic); differential fuzzing with RY_JIT=force` |
Compiling ahead of time (ry run --aot) |
seeded scripts of typed functions built from the shapes that make compiling ahead differ from compiling at a first call: a call before the callee’s definition, a name defined twice or rebound between calls, a definition inside a block, alias-typed parameters, conversions of what R returns, a conversion whose vector is narrower than the checker’s type or empty, a conversion whose type changes between calls after an R call that printed, a literal that warns when it is lowered, a changed keep.source, a script that stops partway (ry test_scenarios typed_scripts_agree_however_they_compile, FUZZ_SEED replays); every typed benchmark program with --aot (typed_benchmarks_run_natively); a script with more definitions than the session’s limit on compiled closures; the run-aot scenario |
the same script interpreted (RY_JIT=off), compiled at first call, compiled ahead, compiled ahead with every sibling guess failing (RY_AHEAD_MISS=1), compiled ahead with tier 1 forced, and with its annotations erased: one exit status, standard output and standard error; no compile after the script starts on the benchmark programs |
Typed code files (ry install, ry build --aot) |
random valid metadata written and read back, then truncated, bit-flipped and rewritten with a fixed hash (typed test_typed file::fuzz_typed_code_parse, cargo fuzz typed_code_parse); an installed file damaged in its header, metadata or code, and a second write against the same install (ry test_scenarios fuzz_typed_code_file); every generated typed script installed as a package and run with its file and with every attach failing (typed_scripts_agree_however_they_compile); RY_GC_STRESS and the copied-code path over attached code, shapes interned in another order, a run-time copy and a reinstall in a running session (install-typed-package scenario) |
the metadata as written; the same bytes from both writes; what the script prints with an intact file or none |
Executables (ry build --aot --exe) |
random bytes read as an executable, and payloads appended to random bytes, read back, and then read again with bytes of the trailer and metadata changed (ry test_fuzz fuzz_executable_payloads, cargo fuzz executable_payload); every generated typed script built into an executable and run (typed_scripts_agree_however_they_compile); arguments, an error partway, every attach failing and the default output name (build-exe scenario) |
the payload as written, or none or a damaged one, never a panic; what ry run prints |
| Table engine | table property tests: random and mutated delimited text with quotes, embedded line breaks and random read options, read whole, through the slicer and through a memory-mapped file; grouped aggregates; sorting (fuzz_csv_reading, fuzz_grouped_aggregates_against_a_reference, fuzz_sort_unique_and_runs_against_a_reference); cargo fuzz csv, whose byte decoding fuzz_csv_target_bytes runs in the default suite |
a whole read of the same text; a reference implementation; the eager implementation; the sniffer’s promised columns |
File removal (unlink) |
seeded trees of files, nested and read-only directories, links into and out of the tree, dangling links, FIFOs and Unix sockets, removed with random recursive, force and working directory (interp test_file_removal, FUZZ_SEED replays); the io__unlink oracle case |
a port of GNU R’s R_unlink on a copy of the same tree: the failure result, and every entry left, with its mode and link target; GNU R, through the oracle case |
Console (ry repl) |
typed keys, typeahead split at random, input chunks with hook code (repl test_console_input); seeded sessions against GNU R’s console (test_console_session) |
a model of the key decoder and of R’s grammar; GNU R’s prompts and output |
| Package install | the install-package scenario; test_install |
the package’s own results |
| CLI | scenarios run the real binary | expected output and exit status |
| Check cache | seeded edits to a generated package (ry cli::tests::cached_checks_match_fresh_checks_across_seeded_edits, CACHE_SEED, FUZZ_ITERS) |
a fresh check of the same files |
| Docs examples | every marked output block in the docs, and the landing page samples (test_docs) |
the output the page shows |
The PROTECT balance check runs on every .Call and .C in every build: when a routine returns with a
different protect-stack depth than it entered with, ry warns stack imbalance in '<routine>', as GNU R
does. A stricter checker that catches an unprotected object at the moment it is collected is
RY_GC_STRESS, which collects before every allocation.
Plain stress runs a minor collection each time, which ages every survivor into the old generation
at once, and a minor collection never frees an old object. An object that survived one collection
and is then held only by a Rust local is therefore safe under plain stress, although outside
stress it is still young and the next minor collection frees it. RY_GC_STRESS=young catches that
case. Every allocation runs a collection that traces again every object it has not tenured, so
survivors stay collectable until every 64th collection tenures them, and major collections keep
the plain mode’s spacing. A small script runs about five times slower than under plain stress.
RY_GC_STRESS |
Stress starts | Each allocation runs |
|---|---|---|
1 |
when the script starts | a minor collection |
startup |
at the first allocation | a minor collection |
young |
when the script starts | a collection that keeps survivors untenured |
startup-young |
at the first allocation | a collection that keeps survivors untenured |
seed:N |
when the script starts | a collection at random gaps drawn from seed N |
startup-seed:N |
at the first allocation | a collection at random gaps drawn from seed N |
Both modes above collect so often that an object is promoted before the operation that made it
is done with it, so neither catches a store that skips the write barrier: by the time native code
moves a young object, the object is old. RY_GC_STRESS=seed:N collects at random gaps instead,
from one allocation to a few thousand with a heavy tail, so objects routinely stay young across a
whole native call. Seven collections in eight are minor, the rest intermediate, and a major one
runs once per as many allocations as there are live objects. With RY_GC_VERIFY=1 each
collection short of a major one first checks the barrier’s invariant: every old object that
points at a young one is remembered, has the element on a dirty card, or is escaped (see the
barrier contract in R runtime). A missed barrier then panics
at the next collection instead of freeing a live object only when the timing is unlucky.
The same verification checks every root before the collector traces it, and every reference a
reachable object holds as it is traced. A root or reference that is not nil and not the start of
an allocated object panics with heap corrupted. A value kept without a root, and a write that
runs past an object’s storage into its neighbor, then fail at the next collection instead of when
the damaged object is used. The report says what holds the reference, what the reference points
at, and where it is stored:
heap corrupted: protect stack entry 0 of 1 is 0x2000000200c0, which is an object that collection 1 freed. Found by collection 2.
heap corrupted: a reachable List of length 1 at 0x2000000200f0 holds 0x2000000200c0 as element 0,which is an object that collection 1 freed. The reference is the word at 0x200000020110. Found bycollection 2.
heap corrupted: a reachable List of length 2 at 0x2000000200c0 holds 0x3ff0000000000000 aselement 1, which is no address of the heap (the bits of the double 1). The reference is the wordat 0x2000000200e8. Found by collection 1.| The report says | What happened |
|---|---|
an object that collection N freed |
Something kept the reference without a root across collection N. The holder names where: a protect stack entry, a handle, a handed-out value, a value the interpreter’s root source names, or a field of another object. |
no address of the heap (the integer N) or (the bits of the double X) |
A write ran past the end of a neighboring object and stored data over the reference. |
an address N bytes into the S-byte slot of a live object |
The reference was computed, or a write overlapped it by part of a word. |
a free S-byte slot |
The slot holds no object: none was allocated there yet, or one was freed before poisoning was on. |
Small objects live at fixed addresses, so a run with the same seed reports the same address and the same collection number for them. A large object’s address changes from run to run.
A build with debug assertions also checks the mutator between collections: a store into an object, a stored value and a value being rooted as a handle must each be an allocated object. A builtin that allocates its result without rooting it, then fills it while allocating each element, fails at its first store after the collection that freed the result, and one that roots a value too late fails where it roots it:
heap corrupted: the object a store writes into is 0x200000b15000, which is an object that collection 2 freed.
heap corrupted: a value being rooted is 0x2000000200c0, which is an object that collection 1 freed.The panic’s backtrace names the builtin. A release build leaves this case to the poison, because the check would cost every store and every root a branch.
The default suite runs a seeded sample of the oracle cases (cases_survive_gc_torture,
cases_survive_young_gc_torture) and of the generated programs (fuzz_programs_survive_gc_torture)
under the first two modes, compared with ry unstressed. A third test
(native_cases_keep_the_write_barrier_under_seeded_gc_torture) runs three cases from the groups
that run the most native code (datatable, packages, altrep, serialize) and one more from
anywhere under seed:N with verification, where N is FUZZ_SEED. A fourth
(cases_survive_a_second_pass_under_seeded_gc_torture) runs twelve cases under the same stress
with each case evaluated twice in one process (the ry-repeat-gctorture-seeded engine in
Corpus and oracle). A function’s second call runs through what
its first call left: a compiled body, a call site’s plan, a reused frame. A value kept there
without a root, or kept in a representation the second call does not expect, shows up only then.
Each sample always includes the cases in REGRESSION_CASES and the seeds in REGRESSION_SEEDS,
which once read freed memory or lost a live object.
Collecting often is not enough on its own, because each sweep hands a freed slot out again at the
next allocation, and a stale reference then reads a valid object. So while RY_GC_STRESS or
RY_GC_VERIFY is on, in any build, the collector poisons every object it frees (fills it with
0xDB) and quarantines its memory: the allocator skips the freed small slots until 16 MiB of newer
ones have joined them, and holds freed large objects until 64 MiB have. In a debug build, reading
the kind of a freed object then panics with read the kind of a freed object at <address>. A
release build skips that check, because it would cost every call of the hottest accessor in the
runtime, so there a freed object reads as FREESXP and its other fields as poison. Without stress
or verification, debug builds poison freed slots but reuse them at once, and release builds do
neither.
just asan runs the heap and interp tests in a build with AddressSanitizer. The heap maps its
own memory, which the sanitizer knows nothing about, so in such a build the collector tells it:
a freed object’s memory past its first word, which holds the kind, is marked unaddressable, and a
slot is marked addressable again when the allocator hands it out. A read or write of a freed object’s fields or elements then
stops at the access, in Rust or in C, instead of reading poison:
ERROR: AddressSanitizer: use-after-poison on address 0x2000000200d0 ...READ of size 8 ... #1 heap::object::Sexp::length crates/heap/src/object.rs:995 #2 heap::object::Sexp::slice crates/heap/src/object.rs:1039The sanitizer sees no boundary between two live objects, because slots sit side by side with no gap, so a write past one object into the next is left to the verifier’s reference check above. Code the JIT generates and a package’s compiled code are not instrumented, which is why the recipe covers the two crates’ tests and not the oracle cases.
Before the argument-reference rooting fix, datetime__year_overflow crashed unstressed under the
oracle harness. In a debug build of that tree, RY_GC_STRESS=young with quarantine panicked
with the message above in 3 of 3 runs, when a collection traced a root that still held a freed
object, while plain RY_GC_STRESS=1 passed 3 of 3, since it tenures the object before it can die.
Running the long harnesses
Section titled “Running the long harnesses”| Recipe | What it runs |
|---|---|
just simulate COUNT ACTIONS BASE |
The language-server simulator over COUNT seeds from BASE. |
just scenarios [NAME] |
Every scenario, or one. |
just fuzz-differential SEED COUNT ENGINES |
Generated grammar programs under GNU R and ry’s engines. |
just fuzz-kind KIND SEED COUNT ENGINES |
One kind of program (grammar, metamorphic, builtins, api, corpus) or all, with each diverging program and its report written to target/fuzz-findings. |
just fuzz-coverage COUNT |
No engine: how many of the functions base, stats and utils export the programs of COUNT seeds of each kind name in a call. |
just fuzz-long HOURS BASE ENGINES |
Every kind for HOURS of wall clock from seed BASE, under tier 0, the forced JIT and seeded collection stress unless ENGINES says otherwise. The staged Fuzz workflow runs it. |
just fuzz-reduce FILE ENGINE |
One diverging program shrunk to a minimal one that diverges the same way (oracle reduce), the form a finding takes as an oracle case. |
just fuzz-corpus SEED COUNT ENGINES |
Mutated snippets of real R code (base help examples, R’s regression tests, the oracle cases) under GNU R and ry’s engines, observing. |
just gc-torture [young|seeded|repeat] |
Every oracle case under collection stress, under RY_GC_STRESS=young, under RY_GC_STRESS=seed:N with the heap checked, or under the last with each case evaluated twice in one process. |
just asan [ARGS] |
The heap and interp tests under AddressSanitizer, which stops at a read or write of a freed object. |
just ci-stress SEED |
One of the release gate’s stress jobs: just gc-torture repeat with FUZZ_SEED=SEED, then 300 generated programs under GNU R, tier 0, the JIT and the repeated seeded stress. CI runs seeds 0, 1 and 2. |
just fuzz-deep [MINUTES] [BASE] [DEPTH] |
Every seeded harness at a deep budget until the time limit, described below. |
just fuzz-run TARGET |
One cargo fuzz target under libFuzzer. |
just fuzz-all SECONDS [FEATURES] |
Every cargo fuzz target for SECONDS of wall clock each, after seeding the corpora and building the targets. FEATURES limits the targets: runtime leaves out the two that link Polars. |
just docs-test [PAGE] / just docs-bless [PAGE] |
The docs examples, or a rewrite of their exact output from the real tool. |
The deep campaign
Section titled “The deep campaign”just fuzz-deep runs tools/fuzz-deep.py, which reads the sources for every #[test] (and every
harness = false main) that loops through for_each_seed, fuzz_seeds or fuzz_iterations,
directly or through a function in its file, so a new harness joins without an entry anywhere. Each
test is then looked up in its binary’s --list, which gives a test inside a module its full path; a
test the binary does not list, or a run that runs no test, is a failure. It then works in three steps:
- The libr suites that run vendored packages’ native code run once under
RY_GC_STRESS=startupand once underRY_GC_STRESS=startup-young, both withRY_GC_VERIFY=1. - Each round runs every harness with
FUZZ_SEEDat the round’s start andFUZZ_DEPTH=DEPTH, and four times that depth for the language-server simulation (at 150 actions), the C API sequences and the collector. Rounds start ten million seeds apart. The oracle’s GC torture runs on a fresh sample of cases each round. - Rounds repeat until
MINUTEShave passed.
A failure does not stop the campaign: it is logged to target/fuzz-deep.log with its replay command,
and the run exits 1 at the end. A run the campaign’s deadline cuts short is logged as cut, not as a
failure, and a timed-out run is killed with every process it started. Successive rounds and a different BASE cover new seeds, so a
scheduled run passes a fresh base, for example just fuzz-deep 480 $(date +%s). The wall clock bounds
the campaign only; every harness still draws its choices from its seed.
- The file system seam for the language server is a real directory, not an in-memory file system, so IO errors are not injected.
- Allocation failure is not injectable; the heap aborts on an out-of-memory condition.
- The loader, the REPL and the runtime are not driven by a seeded scheduler; their nondeterminism (GNU R’s own, threads in packages) is controlled only by running them single-threaded.
- The simulator interrupts a step only with a notification, so a request preempting idle work is not simulated.
- The parser’s completeness check (
syntax::input_is_unfinished) treats a string left open across lines as finished; the console’srepl::input_is_completeholds such a string back itself. Both treatpkg::, a trailing\, a leading;,f(1\n2and a bare[as unfinished where GNU R reports an error. The differential against GNU R skips thepkg::and\endings.