Testing

This project prefers fixture tests for source-driven behavior because they are:

• easy for a human to read in diffs
• easy to extend into many cases quickly
• a good fit for verifying AI-generated changes against an explicit text contract

Fixture suites should describe the desired semantics as exhaustively as practical. Do not shape the matrix around what the current implementation already happens to pass. If the suite is still migrating toward the desired contract, record the missing coverage explicitly — in .agents/memory/backlog.md, or as a case pinned to the wrong-but-current output with a comment saying so — instead of treating current gaps as intentional.

Use ordinary Rust tests only when the behavior is awkward to express as a rendered fixture.

The rewrite stack’s suites

The greenfield stack (crates/syntax, crates/semantics) has its own harness, syntax::testing::run_fixture_suite: a .R.test file holds #==== group / #---- case sections whose source is followed by a #++++ expectation block; RY_BLESS=1 rewrites expectations and FIXTURE_FILTER=group__case runs one case. Suites:

• crates/syntax/tests/syntax — golden lossless trees plus syntax errors (debug_dump)
• crates/syntax/tests/tsr — tree-sitter-r’s parser corpus converted to the same format
• crates/syntax/tests/errors — golden Elm-style error-message rendering, organized by area (lexer, delimiters, control flow, functions/calls, #: type and directive grammar, recovery locality). The coverage contract: every distinct error message the lexer or parser can emit has at least one case here, valid-but-tricky shapes are pinned as no errors, and recovery cases assert that one broken construct reports once and leaves the rest of the file clean
• crates/semantics/tests/typing — the typing suite: each case runs the full semantic pipeline on one package file (shipped stubs installed) and renders every named top-level definition’s exported scheme (name: <T: numeric> fn(x: T) -> T) followed by the file’s diagnostics (start..end severity[code] message, byte offsets). Cases are grouped by feature, except programs.R.test, which holds whole small programs written the way a real project is: a case there earns its place by exercising several features at once, and a clean case with an empty diagnostics block is a zero-false-positive contract for that style of code
• crates/semantics/tests/typing-scripts — the same pipeline over script documents (one sequential top-down scope)
• crates/semantics/tests/typing-strict — the strict stream: the per-file typing mode and the strict-code diagnostics appended after the ordinary rendering
• crates/semantics/tests/typing-imports — package-metadata cases: leading #namespace lines form the case’s NAMESPACE source and #description lines its DESCRIPTION source, installed as the PackageMetadata input before rendering. The directive lines stay in the analyzed text as ordinary comments, so expectation offsets are honest. Attach facts need no directive: a library(data.table) statement in the case source activates the conditional stub namespace through the same scan the hosts run. This suite is new-stack only — the oracle had no package-metadata concept — and was deliberately absent from the retired differential fixture arm
• crates/format/tests/format — the formatter golden suite (ported from the legacy suite): each case’s source formats to the expected block, and the runner re-formats the output to assert idempotence on every case; a case whose expectation is a refusal renders the structured FormatError
• crates/ide/tests/ide — IDE feature fixtures: the case source carries one $0 cursor marker (stripped before analysis) and the expectation renders each feature’s result at that position (hover line with its absolute range, definition target range, reference ranges). A target inside the stub corpus renders the token its range covers (declared in stub source 0 at print“) rather than a byte offset: an offset into the shipped corpus shifts whenever an unrelated declaration is added, which forced a re-bless on every corpus edit while proving nothing, and the token proves the range points where it should

The retired identity differentials, and the benchmark harness

The rewrite was verified against the frozen previous implementation by a family of differential suites (typing, scripts, strict, fuzz, the legacy-corpus sweep, the real-file corpus arm, and a per-position IDE comparison with adjudicated divergence ledgers). That program is complete and retired: the new stack no longer proves equivalence to the oracle — its own fixture suites are the semantics contract, and improvements land without oracle adjudication. What remains of the differential crate is the cross-stack benchmark harness (legacy/differential/tests/test_stats.rs): the same corpus timed and memory-measured through both stacks, kept until the legacy code is deleted.

The semantics fuzz harness

Fuzzing is pipeline-wide and from each stage’s first commit, not a parser-only concern. crates/semantics/tests/test_fuzz.rs runs generated programs (biased toward reference cycles, closures, annotations, and typing-mode directives), token soup, and two-file projects through the full semantic pipeline, checking on every input: never-panic (salsa fixpoints must converge), determinism across fresh databases, diagnostic-range geometry, and incremental equivalence — output after editing a file through the salsa setter equals a fresh database on the edited text, and editing back restores the original. FUZZ_ITERS scales the budgets; the bounded default runs in cargo test -p semantics, and fuzz_deep (ignored) carries long runs. On a panic the harness prints the generating inputs.

The formatter fuzz harness

crates/format/tests/test_fuzz.rs holds the formatter’s arm of the same doctrine. On every input — valid-program seeds, byte-level seed mutations, token soup, random bytes, and real corpus files when fetched — it checks: never-panic (format either succeeds or refuses with a structured error), determinism, and idempotence: whenever formatting succeeds, formatting the output again must succeed and reproduce it byte-for-byte. The refusal path is part of the property: a file with an R-grammar syntax error is refused, while errors raised by the #: annotation grammar (marked in_annotation by the parser) only send the affected block down the verbatim path. The invariant battery itself is exported as format::check_format_invariants (with syntax::testing::check_parse_invariants for the parser and semantics::testing::check_semantics_input for the semantic pipeline — the latter folds every lint under an everything-on configuration into the rendering, so the lint layer inherits the never-panic, determinism, geometry, and incremental invariants) so the coverage-guided targets below share the exact same contracts, and each harness’s fuzz_regressions_hold_invariants pins every input those targets have ever broken.

Coverage-guided fuzzing

fuzz/ is a cargo-fuzz crate (its own workspace, excluded from the main one) with libFuzzer targets parse, format, and semantics, each a thin wrapper over the exported invariant batteries (the semantics target derives its document kind and incremental edit from the input bytes, so one byte stream drives the full battery including the splice cache). It needs nightly: cargo install cargo-fuzz, then from fuzz/:

cargo +nightly -Zscript ../scripts/seed-fuzz-corpus.rs   # seed corpus from all fixture case sources
cargo +nightly fuzz run format -- -max_total_time=600 -print_final_stats=1
cargo +nightly fuzz run parse  -- -max_total_time=600

The seeder extracts every fixture case’s source across the repository into fuzz/corpus/{parse,format,semantics}/ (content-addressed, so re-running only adds). Corpus and artifacts are gitignored — durable regressions belong in the REGRESSIONS battery or a fixture case, not in the corpus. On a find: cargo +nightly fuzz tmin format <artifact> minimizes it; fix the bug, replay the artifact, add the minimized input to REGRESSIONS (plus a golden fixture case when the shape deserves a pinned rendering). A coverage report for judging corpus quality needs the llvm-tools component: cargo +nightly fuzz coverage format && cargo cov -- show per the cargo-fuzz coverage guide. Deep runs are manual/scheduled work; the bounded in-tree batteries remain the default-suite gate.

The lint fixture suites

crates/semantics/tests/lints/ runs lints::lint_file under the default configuration and tests/lints-style/ under naming-style = "snake_case" plus unused-parameter = "warn" (both off by default); each case renders every finding as start..end severity[code] message (or clean). Run with cargo test -p semantics --test test_lint_fixtures.

The CLI contract suite

crates/ry/tests/test_cli.rs drives the real ry binary end to end: diagnostic rendering (1-based positions, gutters, carets), JSON Lines output, the documented exit-code contract (0 clean, 1 findings, 2 usage/configuration/IO errors), configuration discovery and failure, per-file # typing: directives, suppression comments, NAMESPACE validation, project stub overrides (including loader-problem reporting), data-masked evaluation, and the formatter’s --check/--diff/in-place modes.

The LSP behavioral suite

crates/ry/tests/test_lsp.rs spawns the real ry server process and drives it over LSP stdio with an async-lsp test client. Coverage: capability negotiation (position encodings, pull diagnostics, refresh, label offsets, snippets), document sync (incremental and full-text, untitled/non-file: buffers, close-time disk rereads), the two-wave push contract and its settled-superset invariant, burst settling under latest-edit-wins cancellation, pull diagnostics (result ids, unchanged reports, retryable cancellation, push suppression), diagnostic refresh on save and config change, the config matrix (live reload, ancestor configs, failure keeps the previous config, the config-file diagnostic), every feature endpoint including UTF-16/UTF-8 range correctness with BMP and non-BMP content and out-of-bounds safety, semantic tokens for #: bodies, and .Rtypes/NAMESPACE buffer serving.

The cancelled-pull test is deterministic through a fault-injection seam: with the RY_TEST_DELAY_PULL_MS environment variable set, the server announces each diagnostics pull by creating the RY_TEST_PULL_MARKER file and then holds the pull (until the cancellation token flips, bounded by the delay) before computing. The test sends its didChange only after the marker appears, so the edit’s flip provably lands while the pull is in flight — the retryable SERVER_CANCELLED response, and the successful retry after it, can be asserted without sleeping-and-hoping. The variables exist only for this test; production runs never set them.

The perf and memory witnesses

legacy/differential/tests/test_stats.rs (ignored; needs the fetched corpus and a release build) carries the measurement instruments — one process per stack reporting wall, phase splits, and resident/peak memory into target/stats-{new,legacy}.txt — and stats_witness, the CI-checkable assertion form: cold-pass wall per line, resident bytes per line, and resolve steps per line (the resolve-memoization regression tripwire) against budgets set from the measured gate numbers with headroom.

Fixture format

Every suite on the shipping stack runs through one harness, syntax::testing::run_fixture_suite (shared by the syntax, semantics, format, and ide crates). A .test file holds groups of cases:

#==== group_name
#---- case_name
<input source>
#++++
<expected rendered output>

Rules:

• group__case is the stable test identity; names must be unique across the suite — duplicates are rejected rather than silently shadowing one another
• what the expectation shows is the suite runner’s contract (schemes, diagnostics, tree dumps, hover output, …); suite-specific behavior belongs in the runner, not in fixture syntax
• the runner only loads files with the .test extension, so a suite directory may hold notes (a README.md describing its rendered-output contract, say) without the runner treating them as cases

Focused runs

Run one focused fixture case with FIXTURE_FILTER and the suite’s test target, for example:

FIXTURE_FILTER=group__case cargo test -p semantics --test test_typing_fixtures -- --nocapture
FIXTURE_FILTER=group__case cargo test -p format --test test_format_fixtures -- --nocapture
FIXTURE_FILTER=group__case cargo test -p ide --test test_ide_fixtures -- --nocapture

The default crate test command while iterating is cargo test -p semantics (analysis behavior); just gate runs the whole battery plus clippy and a formatting check before a change lands.

Blessing expectations

Set RY_BLESS=1 to rewrite the #++++ expectation blocks in the source .test files in place from the actual runner output instead of failing on a mismatch:

RY_BLESS=1 cargo test -p semantics --test test_typing_fixtures

Behavior:

• only the content body of each #++++ block is rewritten; directive lines and surrounding spacing are preserved, so a blessed file is byte-identical to what a human would have written and re-running without bless passes
• FIXTURE_FILTER still applies, so you can bless a single case
• blessing an already-correct suite changes no bytes

Review every blessed change before committing: bless captures whatever the runner currently produces, so it will happily record an intentionally wrong outcome if the implementation is wrong. Fixtures are the desired-semantics contract, not a regression archive — accept a changed expectation only when the behavior or wording intentionally improved.

The frozen legacy stack’s harnesses

The previous implementation stays in-tree (legacy/analysis-legacy, legacy/engine-legacy, legacy/ry-legacy, with its own legacy/fixtures harness) as the cross-implementation oracle the differential compares against, and as the benchmark baseline. Its suites still run — cargo test -p analysis-legacy / -p engine-legacy / -p ry-legacy, and the workspace-wide battery covers them — but the stack is frozen: do not extend its fixtures or harnesses, and never share code between the two stacks. Its fixtures crate parses the same Simple shape plus a MultiFile shape (explicit file paths and grouped workspace edits) that its engine-era suites use. Beyond fixtures, the legacy engine carries its own differential regression net (engine output asserted equal to a from-scratch rebuild over adversarial edit streams, per-position IDE parity against a fresh-analysis oracle, exec-counter and memory witnesses); it documents the bar the rewrite’s own harnesses were built to meet.

Testing guidance

• Prefer adding or tightening fixtures before writing parser-local or crate-local unit tests unless the behavior is genuinely awkward to express as a fixture.
• When adding a new phase or module, add or extend a fixture suite for that phase before relying on ad hoc unit tests.
• Use the lightest fixture change that captures the failing shape.
• Keep fixture expectation changes deliberate when output changes.
• Prefer keeping typecheck coverage in fixtures unless the behavior is too low-level or too mechanical to express clearly in rendered fixture output.