Skip to content
ry

Development

ry is a Rust workspace whose crates live under crates/. Structure maps every source file, and Architecture explains how the analysis fits together. The editor extensions live in editors/code (VS Code) and editors/zed (Zed).

With nix installed, nothing needs provisioning for the build, the tests and the oracle: source tools/env.sh loads the flake’s default dev shell (the Rust toolchain, the reference GNU R, just, mold), and a Claude cloud container needs no setup script. The first run downloads 8.8 GB and takes about 100 seconds; the fast loop has the details.

The package corpus and the benchmarks against GNU R need the host’s libraries. On Ubuntu 24.04, one script provisions everything the build, the tests, the oracle and the benchmarks need, and just doctor checks the result:

Terminal window
sudo tools/dev-setup.sh # apt libraries, just, GNU R 4.6.2 in /opt/R, the oracle's packages
just doctor # each check prints ok, warn or FAIL with the command that fixes it

The script is idempotent, so a cloud environment can run it as its setup script in every fresh container. It installs just 1.59.0 from PyPI, because the justfile needs 1.40 or newer and Ubuntu’s package is older, and it wraps rustc in sccache, so a fresh worktree compiles its dependencies from a shared cache (a cold debug build of ry takes about 1 minute instead of 4.5). The cache lives in ~/.cache/sccache and grows to sccache’s default cap of 10 GB; set SCCACHE_CACHE_SIZE to change it. The oracle’s reference GNU R and the packages its cases name (rlang pinned to 1.3.0) come from tools/reference-r.sh, which restores a prebuilt copy from the public binary cache ry.cachix.org in about five seconds and builds from source (about three and a half minutes on four cores) when the cache holds no build of the current scripts. The script also restores the package-test corpus library from the same cache (about ten seconds). When the cache has no build of the pinned set, just bench-packages install installs it from source in about eight minutes.

Restoring needs nix, which the cloud image has, and takes zstd from nixpkgs when it is not installed; without nix, the script builds R from source. tools/cache.lock names each prebuilt tree by a hash of the files it was built from, so editing tools/reference-r.sh or a pinned version makes the next container build from source, and just doctor warns until the lock has a build of the current inputs. After such a change, push the new build and commit the lock:

Terminal window
sudo tools/dev-setup.sh # builds R from source, since the cache has no build of the new inputs
just cache-push r # packs /opt/R/4.6.2, pushes it to ry.cachix.org, rewrites tools/cache.lock

just cache-push refuses a tree that is not what its inputs build (an R that fails tools/reference-r.sh check, a library that lacks a locked version), and a push that would take the trees in the lock past 2.5 GB, half of the cache’s 5 GB. Anyone can read the cache, so push only these trees, which hold GNU R and CRAN packages, and never ry’s own builds or anything private. A cloud session’s proxy holds the write token for app.cachix.org, which is why the push names that host; elsewhere, set CACHIX_AUTH_TOKEN.

nix develop (the default and ci shells) downloads everything from cache.nixos.org or ry.cachix.org and builds nothing. The custom GNU R (lib.gnuR) and the R packages built against it are not in nixpkgs, so ry.cachix.org holds them. The Cachix workflow rebuilds and pushes the two shells when flake.nix, flake.lock or tools/base-r changes on main; the Rust toolchain and crane’s ry-deps-* build are left out.

just doctor also checks that ry, libR.so and the start-up image were built from the working tree, by recomputing the runtime hash they embed, and it reports free disk and the size of each target/ profile. A cloud container’s disk quota is about 40 GB whatever df says. just prune frees space without forcing a full rebuild: it removes build configurations nothing has read for two days and incremental caches older than an hour, and just prune --tests also removes test executables.

The flake’s dev shells carry the same toolchain and a GNU R built like the reference one, and CI runs in them: nix develop .#ci holds what just ci needs, and nix develop adds the editing tools. tools/env.sh loads the default shell for agents.

The workspace builds on the dated nightly that rust-toolchain.toml pins, because std’s allocator API and other unstable features are worth having before they stabilize (the policy is in .agents/memory/decisions/docs-process.md, “Data-oriented design and explicit allocators”). With rustup, cargo downloads that toolchain, with clippy and rustfmt, on the first build. The Zed extension is the exception: editors/zed/rust-toolchain.toml pins a stable toolchain, because the nightly’s wasm32-wasip2 std imports a WASI function that Zed’s host does not provide yet. The nix dev shell reads the same file. Nightly rustfmt also applies the import settings in rustfmt.toml, so cargo fmt keeps a single use { ... } tree per file.

The justfile at the repository root names every day-to-day action, and just with no arguments lists them all. The common ones:

Terminal window
just ry check # build ry and its runtime if needed, then run `ry check`
just build # every binary and ry's runtime (`just build --release`)
just try # build ry and walk through every command in a demo project
just try-editor # build ry and print how to point an editor at it
just gate # tests, clippy -D warnings, and fmt --check
just battery # all workspace tests, excluding zed_ry
just fixture <group__case> # one fixture case
just bless -p semantics --test ... # rewrite fixture expectations from current output
just docs-test [page] # run the docs' marked examples through the real binary
just stats <path> # phase timings and memory for one workspace
just perf # corpus perf and memory budgets (needs corpus/)
just repl-e2e # drive `ry R` and `ry run` through a pty
just release-dry-run # package and smoke-test this machine's release archive
just docs-deploy <branch> # build the docs site and deploy it (main is production)
just doctor # check the toolchain, GNU R, the corpus and binary freshness
just prune # free disk in target/ without forcing a full rebuild
just cache-push r # push the reference GNU R to ry.cachix.org (or `corpus`)

The simulators, fuzzers and long oracle runs have their own recipes, which Robustness lists.

just fixture runs a case from the test_typing_fixtures target in semantics unless you pass a different target and package as its second and third arguments. Review the diff after every bless: bless records whatever the code produces, right or wrong.

Without just:

Terminal window
cargo build # the ry binary (the workspace default member)
cargo test # the ry crate's tests
cargo test --workspace --exclude zed_ry # every crate
cargo test -p semantics # the analysis core, the usual loop while iterating

Most behavior is pinned by fixture tests, plain-text .test files whose rendered output is compared to an expected block. Read Testing before you add or change one. Two environment variables come up every day:

  • RY_BLESS=1 rewrites the expected #++++ blocks in place.
  • FIXTURE_FILTER=group__case runs a single case.

Some suites and every performance instrument read a corpus of real-world R code. Fetch it with cargo -Zscript scripts/fetch-corpus.rs, which writes into the gitignored corpus/ directory. The resolved inventory is committed as scripts/corpus-manifest.txt.

Profile Command Use it for
dev cargo build everyday work and tests
release cargo build --release shipped binaries and every number you publish or quote
iterate cargo build --profile iterate, cargo test-release optimized tests and quick performance loops
profiling cargo build --profile profiling perf, callgrind and samply

iterate is release without cross-crate link-time optimization (lto = false, which keeps cargo’s thin LTO inside each crate). After an edit to the interpreter it rebuilds ry and libR.so in about 2 minutes on four cores, against about 6 for release, and its link stays under 16 GB of memory. No inlining then crosses a crate boundary, so a single program can run a quarter faster or slower than under release: ry --version names the profile (ry 0.4.3-beta (iterate build)), and tools/bench/bench.py refuses any binary that is not a release build. Its --any-build flag lifts that refusal for a look at where a build stands, for example just bench typed --ry target/iterate/ry --any-build, and such a run cannot be stored. profiling is release with line tables kept and the same fat LTO, so the profiler sees release’s inlining: the .text of ry and libR.so is within 0.2% of release’s, and its link peaks near 11 GB.

The repl crate loads R’s shared library at runtime instead of linking against it, so the whole workspace builds and its tests run on a machine with no R. Only running the console needs R. The end-to-end tests skip cleanly where R is missing; run just repl-e2e before you touch the console. REPL design records how the embedding works.

ry run and ry repl --runtime ry load ry’s own runtime, libR.so, from the directory of the ry executable. It is a separate crate (libr), so cargo build -p ry-lang does not rebuild it; just ry and just build do. To keep a leftover library from running old code, ry compares a hash of the runtime’s sources with the one stored in libR.so right after loading it, and refuses a mismatch with exit status 2:

x ry's runtime (target/debug/libR.so) is out of date: it has build 8fcdf6d9f4862983, but this ry is build
| 2588183553632a45; rebuild it with `just build` or `cargo build -p libr` (same profile as ry)

Build libr with the same profile as ry and the error goes away. The hash covers the sources of libr and the crates it links, so editing the formatter or the IDE layer never triggers it.

  • ry server --debug, or RY_DEBUG=1 in the server’s environment, adds a “Debug” section to every hover with the Lowering, Naming, and Parsing views of the expression under the cursor. It is deliberately not a ry.toml key, because the configuration file is a user-facing contract.

  • ry debug ast <file> prints a file’s syntax tree.

  • ry debug erase <file>... rewrites files in place as their erasure, the plain R that ry R runs.

  • ry debug analysis-stats [path] (or just stats) runs the full pipeline over a workspace through the language server’s queries, with [check] typing forced on, and prints:

    • the wall time and resident-memory growth of each phase, and the slowest files to type-check;
    • keystroke latency and rechecks per keystroke, from an incremental probe;
    • how often hover, goto-definition and signature help answer at sampled identifiers, by role (call head, variable, argument name, $ field, pkg::name);
    • how often completion of a truncated identifier puts the original name in the top five, with latency percentiles;
    • whether find-references on sampled package bindings agrees with an independent lexical resolver (crates/ry/src/scope_oracle.rs), and whether renaming a binding and back restores the text.

    Build with --release when absolute numbers matter; a debug build gives honest ratios.

The sidebar (docs/astro.config.ts) has four sections, and detail belongs in Reference or Development, never in the introduction or the guide:

Section Pages
Introduction getting-started.mdx, the landing page: a short pitch, what ry does, the benchmark summary and next steps, with no code example (by user direction); installation.md (binary, R, editors); benchmarks.mdx (bar charts, what each suite contains, how it is measured)
Learn guide.mdx (“Adopting ry”: one H2 per adoption step, each a sequence of snippets with real output); ry.mdx, the language tour (every feature once, code first, also served as /ry.md), kept current when a feature changes
Reference reference/, the contracts: type-system.mdx is the semantics contract, and the other pages document the tool
Development development/, for contributors; development/design/ holds unsettled drafts, the one place allowed to describe behavior that does not exist, kept out of the sidebar and listed on that folder’s index page

A moved or deleted page gets an entry in the redirects map there. docs/src/components/Sidebar.astro keeps the sidebar’s scroll position across page loads, because the theme scrolls .sidebar-content rather than the element Starlight’s own persister restores.

Typed examples are tab groups inside <Tabs syncKey="annotation-style">, so a reader’s choice follows them across pages: Untyped (the plain R a reader starts from), then Comments and Inline, with the form the page teaches second: Comments in the guide’s step 2 and Inline elsewhere. An example drops Untyped when its plain R would not be R (records, tags, match, Arrays, DataFrame, the standard library, nominal types) or when its subject is an annotation itself. Pages with tabs are .mdx. Use one tab group per idea, not one per snippet.

Every block of ry output on a page is a test that just docs-test runs through the real binary; Docs examples has the marker format and the rules for tabs.

benchmarks.mdx has one section per measured suite, in this order: public R JIT benchmarks, package test suites, numeric kernels (ours), application workloads (ours), hot package entries, and typed ry against Rust, with typed as a column where a typed version exists. A suite without committed results gets no section and no summary row. The suite against Rust reports typed ry’s time over Rust’s, where lower is faster, so it never shares a table, a chart or a sentence with the speed-ups over GNU R. Each suite is a directory under tools/bench/ holding suite.toml (its id, names, summaries, source paths) and results.json, which just bench <suite> --release ARCHIVE --write writes on the tagged release archive. The runner, tools/bench/bench.py, owns every too-short or refused flag and every geometric mean, and docs/src/lib/benchmarks.ts and tools/bench/readme.py only render those files, so never type a benchmark number into prose. just readme --check (in just gate) catches a stale README block or diagnostic image.

Agents read the site as Markdown. Every page is also served at its URL plus .md (for example /guide#step-2-type-your-code.md), with the frontmatter, imports and components removed and each tab group turned into one subheading per tab. /llms.txt lists the pages with their descriptions in sidebar order, and /llms-full.txt concatenates them all. Each page’s head links its Markdown version with <link rel="alternate" type="text/markdown">, and the breadcrumb row has a “View as Markdown” link. The code is docs/src/lib/markdown.ts, the three endpoints in docs/src/pages/, and docs/src/routeData.ts. A new MDX component must be handled in bodyToMarkdown, or it leaks into the Markdown as raw tags. A description containing # must be quoted, because YAML reads the rest of the line as a comment.

Run just docs for a live preview, and cd docs && bun run build (or pnpm build) for a full build, which needs Node 22.23.4 or newer. The live site changes only when someone deploys it with just docs-deploy or the Docs workflow (Releasing).

Formatting rules is generated, so do not edit it by hand. Edit crates/format/tests/formatter.template.md and regenerate the page with just format-docs.

  1. Run cd editors/code && bun install.
  2. Open the repository in VS Code and press Ctrl+Shift+B to compile the client in watch mode.
  3. In the Run and Debug view (Ctrl+Shift+D), select Launch Client and press F5.
  4. Open any .R file in the Extension Development Host window that appears.

A comment in R can sit between any two tokens: inside an if header, between a call and its argument list, after an operator. A declarative “format each field” style silently drops such a comment, so the formatter walks the concrete children token by token and decides placement at the token level. The parser attaches a trailing comment inside an expression node, so a decision about where a closing token goes has to look at tokens, never at whole elements. See crates/format/src/format.rs and the fixtures under crates/format/tests/format/.

AGENTS.md, “How to write”, sets the rules for everything an agent writes. Here is one idea written three ways. The first packs five facts into one sentence behind a colon:

x |> f(y) is syntax R’s own parser rewrites to f(x, y) before evaluation, and it types as exactly that call: the piped value becomes the first positional argument, every call rule above applies (arity, argument compatibility, overload selection), chains compose left to right, and a type error on the piped value blames the left-hand expression.

The second overcorrects. Every sentence is short, and none of them says how it relates to the one before:

R’s parser rewrites x |> f(y) into f(x, y) before it evaluates the code. ry types the pipe as that call and nothing else. The piped value becomes the first positional argument. All call rules above apply to it: arity, argument compatibility, and overload selection. Chains compose from left to right. If the piped value has a type error, ry reports it on the left-hand expression.

The third is the one to write:

R’s parser rewrites x |> f(y) into f(x, y) before anything runs, so the pipe types exactly like that call. The piped value becomes the first positional argument, and every call rule applies to it: arity, argument compatibility, and overload selection. A chain composes from left to right, and a type error in the piped value is reported on the left-hand expression.

Agents read whole source files, so every comment line costs context on every read. A Rust comment explains why the code is written as it is, and only when the reason is not obvious. just comments (part of just gate) runs tools/comments.py check, which flags a // or /// block longer than 12 lines, a //! module doc longer than 25, a banner (// ----, // ==== Section ====), a section heading (a short // Allocation. line between blank lines), and process words such as a milestone, a pull request number or a hash cited as a commit. A license header at the top of a file is exempt, as are patches/ and target/.

The check is a ratchet: tools/comments-baseline.txt lists the files still allowed some violations. A file over its count fails. A file under its count also fails, with a hint to run python3 tools/comments.py baseline so the count shrinks. python3 tools/comments.py report lists the worst blocks with their file, line and size. A block whose facts do not fit the limit stays over it and is carried by the baseline: shortening a comment must never drop a reason, an invariant or a measured cost.

The agent knowledge base is .agents/memory/, indexed by MEMORY.md. Settled decisions live in .agents/memory/decisions/, one file per area (syntax-semantics.md, interp.md, jit-ssa.md and nine more), and decisions/index.md lists every entry with a link and its status. A new decision is an H2 section with a **Status:** line, appended to the area file it belongs to, after which python3 tools/memory.py index regenerates the index. A crate’s own traps live in crates/<crate>/AGENTS.md.

python3 tools/memory.py check, part of just gate, fails when a single decisions.md reappears, when the index is stale, when one entry heading appears in two area files, when a file outgrows its byte budget in tools/memory.py (AGENTS.md, MEMORY.md, backlog.md, worklog.md and .agents/orchestrator.md, each about 5% above its size when the budget was set, and pruned rather than given more room), when a decisions area file passes the shared 260 KB cap (settled decisions are not pruned, so the area is split into two files), or when a line outgrows its file’s line limit (400 bytes in worklog.md, 2 KB in the orchestrator file). python3 tools/memory.py split moves the entries of a stray decisions.md into the area files.