# Architecture

The crate boundaries, the analysis database, and the language server's scheduling described here
are the contract for the implementation. What the type checker *means* is specified separately, by
the [typing reference](/reference/type-system.md).

## Crate graph

Layering is enforced by the crate graph. A crate boundary is the compiler-checked way of making an
illegal dependency unrepresentable.

- **`syntax`** is a hand-written lexer and a recursive-descent parser with Pratt parsing for
  expressions, emitting a flat syntax tree (see [The syntax tree](#the-syntax-tree)). Three
  properties matter downstream:
  - The trees are *lossless*: every byte, trivia included, reprints exactly.
  - Parsing is *error-resilient*: every parse yields a tree, and an error stays local to the place
    where the code broke.
  - Items are *position-independent*: an item's syntax is a window onto the file tree with offsets
    counted from the item's start, so an untouched statement's tree stays equal when something
    elsewhere in the file changes.

  A `#:` annotation is lexed as structured trivia and parsed into first-class nodes with real spans,
  so type syntax is grammar rather than comment text scraped after the fact. Inline annotations run
  the same type grammar on R's token stream, in `TYPE_ASCRIPTION`, `RETURN_TYPE`, and a function's
  `TYPE_BINDER_LIST` nodes, and `erase_inline_annotations` gives the plain R program an annotated one
  means. The crate depends on nothing semantic.
- **`semantics`** holds the salsa database and every analysis query: the per-item item tree and HIR,
  naming, interned types, Hindley-Milner inference, the stub corpus, the lints, and the point where
  diagnostics leave the analysis. It depends on `syntax` and salsa.
- **`ide`** implements the editor features (hover, navigation, rename, call hierarchy, completion,
  inlay hints, signature help, code actions and fixes, symbols and the outline, folding, selection
  ranges, and semantic tokens) as pure reads of `semantics` queries at byte offsets.
  Anything to do with the editor protocol lives in the server, never here. Reading an installed
  package's R code is I/O, so the server passes it in as a callback (`completion_with`,
  `signature_help_with`) rather than the analysis reading the R library itself. Completion ranks
  names by how often the project uses them, summed over tracked counts per chunk of 32 files, so a
  keystroke recounts one chunk instead of the whole project.
- **`format`** is the formatter. It depends on `syntax` alone, and because the compiler enforces
  that, it can never come to depend on an analysis result.
- **`loader`** is the package-development loader. It reads a package into the analysis, diffs the
  load index against what a session has loaded, and produces a backend-neutral plan. Its GNU R
  backend hands the plan to R code as an RDS file. The backend for ry's own runtime lives in `libr`
  (`loader_backend.rs`), so `loader` links no runtime crate: it builds the plan on the heap and
  binds each lazy definition as a promise over the analysis's own syntax tree.
- **`oracle`** is the differential test harness for the R runtime: it runs one R file under GNU R
  and under another engine and compares the observable results. It also generates the programs for
  differential fuzzing and runs the corpus packages' own tests and installs. It depends only on
  `syntax`.
- **`repl`** is the R console behind `ry run` and `ry R`. It loads R at run time, so no crate,
  `repl` included, needs R to build. For `ry run` it loads ry's `libR.so` instead of GNU R's,
  and `ry` itself does not link `interp` or `libr`.
- **`heap`** is the R runtime's object model and garbage collector: the `SEXP` object layout,
  allocation in bitmap-managed pages, the non-moving generational collector and its roots,
  `CHARSXP` interning, symbols, environment frames and the `R_alloc` arena. It depends on nothing else in the workspace.
- **`libr`** is R's C API, built as the shared library `libR.so` that compiled packages link
  against, with the C half of the native boundary and an R 4.6-compatible header set. It depends on
  `heap`, `interp`, the interpreter's leaf crates (`stats`, `graphics`, `frames`, `services`),
  `jit`, `table`, `loader`, `rcmd`, `semantics` and `syntax`, and it is the host that installs the
  runtime crates. Its design is the draft at
  [`development/design/r-runtime.md`](/development/design/r-runtime.md). Evaluation belongs to the
  interpreter, which plugs into it through the `Evaluator` hook in `eval.rs`.
- **`interp`** is the R interpreter: the evaluator, the tier-0 bytecode, conditions, S3 and S4
  dispatch, and base R's primitives and internals in Rust, plus the check of
  [inline annotations](/guide.md#inline-annotations) at function boundaries. It depends on
  `heap`, `numeric`, `rcmd` and `syntax`, and `libr` installs it through the `Evaluator` hook, so it knows
  nothing about C. The values that are ry's own live here too: records and tags, `List` and the
  standard library. A `for`, `while` or `repeat` loop in the global
  environment compiles to tier 0 before its first iteration, as GNU R's JIT compiles it, because
  compiled code binds the base functions it inlines at that moment
  ([`design/top-level-loops.md`](/development/design/top-level-loops.md)). A call object that
  native code evaluates with `Rf_eval` compiles to tier 0 at its third evaluation, keyed by the
  object, so data.table's per-group `j` runs compiled; an in-place change to the call recompiles
  it, and a call tier 0 cannot run as a promise (a loop, `return`) runs on the tree walker every time.
- **`numeric`** holds the numeric kernels that take and return plain numbers: R's nmath library,
  extended-precision sums, the vectorized elementary functions, the FFT, LINPACK and C's number
  formats. It sits below `interp`, so an edit to the interpreter recompiles none of it.
- **`stats`**, **`graphics`**, **`frames`** and **`services`** are the interpreter's leaf crates:
  the stats package's native routines, R's graphics engine with its devices and renderer, the
  `DataFrame` binding over `table`, and the HTTP client. Each depends on `interp`, and nothing
  but the host depends on them, so the interpreter builds without Polars, a rasterizer or an HTTP
  stack, and an edit to one of them rebuilds only that crate and the host. A leaf's `install()`
  registers its builtins and native routine tables with the interpreter and fills the
  interpreter's function-pointer table (`extensions.rs`) for the calls that go the other way,
  such as tracing the graphics devices' objects during a collection. A host that installs no
  leaf runs with the table's defaults: no device is open, no object is a table, and a transfer
  fails as an unsupported protocol.
- **`jit`** is the runtime's optimizing tier: it compiles hot tier-0 bytecode to machine code with
  Cranelift. Tier 0 calls it through the `Tier` hooks in `interp`'s `bytecode.rs` when a body has
  been entered often or a loop has jumped back often, and the compiled code runs on tier 0's own
  register window, so a failed speculation hands the activation back to the interpreter mid-body.
  It depends on `heap` and `interp`, and `libr` installs it after the interpreter. `RY_JIT` picks
  the mode: `on` (the default), `off`, or `force`, which the tier tests and the `ry-jit` oracle
  engine use.
- **`typed`** is the typed core. A strict typed closure called with values whose types it accepts
  is lowered from its body to a typed IR at its first call, given reference counts, and compiled
  with Cranelift to run on native values: scalars in registers, 16-byte strings, lists that
  view an R vector's data or own their items, dictionaries, and records, tuples and tags in
  counted boxes whose *shape* (layout plus field types) the code knows. A tag type is a union of
  shapes, and a `match` over one branches on the box's shape. `jit`'s `native::call` is the only
  caller, and a call that does not compile, or gives up while it runs, goes back to the
  interpreter unchanged. Typed code calls an R function from native code when the call is a
  statement or the argument of a `*.from_r` conversion; the arguments arrive as forced promises
  of the code written in the call, and the conversion's result type comes from the type the
  interpreter saw it give, so the first call of such a function runs interpreted. Code that
  gives up after calling R is run again by the interpreter, which takes the value of each R
  call the code already made from a record instead of calling R a second time
  (`interp::replay`). It depends on `heap` and
  `interp`. `RY_TYPED_LOG=1` prints each compilation with the time it took
  (`typed: compiled for [F64] -> F64 in 412 us`, the time of that function alone), each refused
  compilation with its time (`typed: refused [Str] after 85 us`), so the lines of a run add up to
  its whole compile time, and each typed call that runs in the interpreter with the reason the
  native path declined it. `RY_TYPED_DUMP=1` prints each compiled function
  twice: the typed core's own instructions after its passes, then the Cranelift IR built from them.
- **`table`** is ry's built-in typed table: an engine over Polars with a query IR that lowers
  `tbl[i, j, by]` and the `DataFrame` verbs to a lazy plan, plus the readers, writers and
  data.table helpers around it. `frames` owns the `DataFrame` value and calls the engine, and `libr`
  exposes a column to native code as an ALTREP vector, so the table holds only Arrow data.
- **`sniff`** reads the layout of a delimited text file: separator, header and each column's R type.
  The table engine's reader and the checker both use it, so the checker types a literal
  `fread("prices.csv")` as exactly the columns the engine reads. It depends on `syntax` alone.
- **`rcmd`** is ry's R home and the `R CMD` commands that build and install packages against
  ry's headers and `libR`. It depends on nothing else in the workspace; `libr`'s tests use it.
- **`ry`** is the product: the LSP server and the CLI, with the `check`, `fmt`, `server`, `run`,
  `R`, `load`, and `debug` commands. It owns configuration, conversion between position encodings,
  assembling and publishing diagnostics, and suppression comments.

## The semantics database

Analysis runs on salsa and is incremental at the granularity of an *item*: one top-level statement
of a file, such as a function definition, a value binding, or a bare call.

### Inputs

The analysis of a project has five inputs:

- `SourceFile` carries one file's text and document kind.
- `ProjectFiles` is a singleton listing the project's files: package documents first, in the
  `DESCRIPTION` `Collate` order when one is declared and by path otherwise, then the rest by path.
  The last-writer-wins symbol index and the CLI both rely on that order.
- `PackageMetadata` is a singleton carrying the imports from `NAMESPACE` and the dependency universe
  from `DESCRIPTION`.
- `StubSources` is a set-once singleton carrying the `.Rtypes` corpus.
- `AnalysisSettings` is a singleton carrying the configuration that changes what analysis means
  rather than which findings a host publishes: `[check] inline-annotations` and `[check] strict`.
  The first is read only where an inline annotation exists, so toggling it re-checks only the
  items that have one. The second is the project default behind `semantics::file_is_strict`,
  which decides a file's mode for the check (its `# typing:` directive wins). The package loader
  keeps its own `load_index::file_is_strict`, which reads the loaded package's default instead. A check records the
  origins strict mode reports only for an item in a strict file, so a project that is not strict
  never builds that table.

The package loader adds a sixth, `PackageSources`, described [below](#the-package-load-index).

### The syntax tree

A file's tree is one `syntax::Tree`: its text and four parallel arrays over its elements (nodes and
tokens alike) in preorder, holding each element's kind, start offset, the index one past its
subtree, and its parent's index. Kinds are numbered tokens first, so a comparison tells a node from
a token, and an element's end is where the next element after its subtree starts. A subtree is
therefore a contiguous index range and a node's text a contiguous slice of the file text, so a
whole-tree walk is a linear scan and reading a node's text never allocates.

An item's tree is a window onto its file's tree, not a copy. A `Tree` holds the file's arrays
behind an `Arc` plus the window's first index, length and base offset, and every accessor
subtracts those, so an item tree reports indices from zero and offsets relative to the item.
Equality compares kinds, subtree ends and those relative offsets, which is why moving an item
within its file does not invalidate what was computed from it. Copying item ranges instead would
duplicate every file's arrays, about 114 MiB on the finance bench.

Cursors are indices. An owned `SyntaxNode` or `SyntaxToken` is the tree's `Arc` plus a `u32`, so
moving one is arithmetic and a clone is one reference-count increment. A borrowed `NodeRef` is the
tree reference plus the index and costs nothing to copy, which is what the whole-file passes and
HIR lowering use. Lookups that a nested tree answers child by child use the layout instead: starts
never decrease in preorder, so `node_at`, `token_at_offset` and `child_or_token_at_range` binary
search the starts, and a previous sibling is found by climbing parents from the element before.

The parser emits elements in postorder, because it only knows that a node wraps the elements after
a checkpoint when it opens the node there. `TreeBuilder::finish` lays them out in preorder in one
backward pass: an element's preorder index is its parent's plus one plus the size of its earlier
siblings' subtrees, which postorder gives as a difference of first-element indices. The parse
invariants (`syntax::testing::check_parse_invariants`) check every shortcut against the
child-by-child definition on every parsed program, and with `enable_invariants()` every splice
reparse is compared with a parse from scratch.

### Parsing, and why it is not where incrementality happens

`parse(file)` is not a query. It returns the file's tree from the host's `TreeStore`
(`Db::trees`), the one home of syntax trees, which holds at most one tree per file. The store is
split into 16 shards by file, each behind its own lock, and a parse runs outside every lock. A
second reader of a file that is being parsed waits for that parse and takes its tree. When the file's
text differs from the stored tree's, the store works out the edit as the longest common prefix and
suffix, and `syntax::reparse` reparses only the statements the edit touched and copies the
untouched top-level runs of the old tree around them. A query that reads a tree records its read of
the file's text and keeps only what it derives, so no memo holds a file's tree. The one syntax a
memo keeps is the detached copy of an annotation's subtree, described below.

The spliced tree is byte-identical and error-identical to a parse from scratch. The edit-stream
fuzzer pins that equivalence, and the splice refuses and falls back to a full parse whenever the
equivalence is not provable, so the store is invisible to every query downstream. Invalidation below
the level of a whole file stays out of salsa on purpose.

The real incrementality happens one level down, in the item tree. An item's identity interns its
kind, its name, its parent, and a disambiguator, which makes it stable when other items are
inserted around it. An edit inside one item therefore leaves the derived values of its siblings
equal, and salsa's early cutoff prunes all the work downstream of them.

### The host decides how long a tree stays

Every host runs the same queries, and the only thing it chooses about trees is the store's
capacity. The server's store keeps every tree (`TreeStore::default`), because an edit reparses one
file by splicing its previous tree and a position request can read any tree at any time.

A command that checks once and exits (`RootDatabase::for_command`, which `ry check` uses) reads
each tree in its first pass and afterwards mostly what was derived from it. Its store is bounded
(`TreeStore::bounded`) at four trees per core. The worker that runs a file's first pass holds the
file (`TreeStore::hold`), and a held tree is never dropped, so the pass parses the file once
however many trees the other workers read meanwhile. The rest of the capacity keeps the most
recently read trees, which serves a pass that reads an item of a file it does not hold without a
parse per read. A tree read after it was dropped is parsed again, which costs time and changes no
result. `test_incremental.rs` pins that a cold check parses each file once on a store of two
trees, and that it reports what a store that keeps every tree reports.

`item_syntax` is a plain function that returns the item's window of the stored tree, so an item's
syntax has no memo of its own to keep the file's tree alive. What it records is a read of the
item's stamp (`item_stamp`), a number that names the syntax of one top-level node. The store keeps
one stamp per top-level node of each file. When it parses a file again, a node at the start or the
end of the file whose subtree equals its counterpart in the previous tree keeps its stamp, and
every other node gets a stamp that no node has carried before. Two nodes with one stamp therefore
have equal subtrees, by comparison and not by a hash.

After an edit, `item_stamp` runs again for every item of the edited file, which costs a lookup per
item, and comes out equal for every item outside the edited region. Salsa backdates those, so
lowering, annotations and naming run again only for the items whose syntax changed, wherever the
others moved in the file. Two edits at both ends of a file between two reads leave one changed
region that spans it: the items between them lower again, and `item_hir` comes out equal and cuts
off everything downstream. A bounded store may have dropped the tree a stamp names. `item_syntax`
then parses the file again, and its reader depends on the file's text like any other reader of
the tree. Under `enable_invariants()` every read of a stamp compares the subtree it names with the
item's node in the file as it stands.

A `#:` block is a sibling of its item in the file's tree, not part of the item's node. One query
per file (`file_item_annotations`) attaches the blocks to their items and stores detached copies
of their subtrees (`syntax::SyntaxNode::detached_subtree`), so an annotation never keeps the whole
file's tree alive either. It runs again after every edit to the file and comes out equal while no
block changed, because a detached subtree compares by content and not by position.

Every query that reads a tree is a memoized query whose result is small: the item HIR and
annotations, `parse_stage_diagnostics`, `file_error_ranges`, the annotation references and
agreement candidates in `diagnostics.rs`, `item_syntax_flags` (whether
an item binds only its own name, and whether it is declared pure), and `lints::file_syntax_lints`. So a check never walks another file's tree, and revalidating an
untouched file after an edit elsewhere never walks its own. A finding that also needs naming or
the project's declarations is split in two: the syntax-only half is memoized, and a cheap step
joins it with the rest (a bare `setClass` call is reported only when the file does not bind
`setClass` itself, and two annotations disagree only when their types differ once aliases
expand).

The CLI's cold check runs its passes over the files, largest first, spread over the cores. The
first computes every tree-derived fact (`diagnostics::derive_from_syntax`), including each file's
share of the project-wide maps (`warm_project_folds`), so that no later check derives another
file's facts on its own thread. The second names every item, so that two threads checking files
that reach the same item do not block on one claim. The project-wide name maps are then built once
per layer, each on a handle of its own, and the next two passes resolve every item's links and
every file's cyclic groups (see "Keeping project-wide work cheap"). The last pass assembles the
diagnostics.

Most tree readers look for a handful of node kinds in a large file, so they walk borrowed cursors
(`syntax::NodeRef`), which touch no reference count: `syntax::matching_nodes` takes a predicate
over a node and its parent and makes owned nodes only for the matches. The per-file site collectors
share one pass over the nodes, `file_syntax_sites`, which tests each role only on the kinds that
can carry it. The syntax lints scan the kind column for the element kinds a lint or an annotation region
can start at (`NodeRef::elements_of_kind`),
track the enclosing annotation region by position (`NodeRef::encloses`), and make an owned node
(`syntax::node_at`) only where a cheap test says a lint may fire. A borrowed `TokenRef` reads its
kind, range and text only when asked, because a child walk mostly checks kinds.

The cold check is CPU-bound and scales about 3 times on 4 cores, so per-file constant costs set its
wall time. Several of them are worth knowing:

- Every salsa read of a tracked function with several arguments interns its key and verifies a
  memo. So an item check resolves every name its naming lists through one query (`item_links`)
  and reads the answers from that query's arrays. `package_name` and `test_name` take only the
  name and read `ProjectFiles` themselves.
- `Ty` and `Name` are not salsa structs. A `Ty` is a four-byte id into one process-wide store
  (`crates/semantics/src/types.rs`), so equality and hashing compare the id. The store is two
  parallel append-only columns indexed by the id's low 27 bits: the kind, and the facts derived
  when the type is built (its size, which the inference-size cap reads without a walk, and the
  type names it applies). A kind is `Copy`, and its children are slices in an arena that is never
  reset. The id's upper bits say whether the type is an inference variable, and whether a
  variable, a named type, a rigid parameter or `Unknown` occurs anywhere in it, so a walk that
  rewrites only one of those returns on a bit test without reading the store. The leaf kinds and
  the first 4,081 inference variables have constant ids, so building one is arithmetic. Finding
  a type that exists probes a table that takes no lock (`crates/semantics/src/intern.rs`), and
  only storing a new type takes one. A `Name` is a `Symbol`, an id into a store of the same
  build that also keeps each spelling's hash, first eight bytes and length. Neither store ever
  frees an entry: a command that exits loses nothing, and the server's stores grow only with
  distinct types and spellings, which repeat heavily because types are structural. The type
  store holds at most 2^24 types. A type that contains an inference variable lives in the same
  store as every other type, because such types leave the item that made them in diagnostics
  and editor facts. A query keyed by one name takes a `QueryName`, the one salsa-interned
  wrapper left.
- The checker matches call arguments in a `bumpalo` arena that lives for one item check, and it
  borrows the item's checks and the stub schemes instead of cloning them.
- A `NAMESPACE` over 64 KiB parses in chunks on all cores (`metadata::parse_namespace`). Chunks split
  only before a line that opens a call and follows a line that is exactly one complete top-level
  call, so no directive straddles a boundary.
- The CLI creates every input at high durability except the attached namespaces, which it sets
  after the first pass. Memos that never read that set skip verification in the new revision.
- Every item keeps its HIR, naming and check memos for the whole run, so each 8-byte field on
  those values costs about half a megabyte across a 300K-line bench. `ItemNaming::extras` and
  `ItemCheck::editor` keep the rarely populated and the editor-only fields behind one optional
  box, and an item's links keep one row per name and one per binding in parallel arrays, with the
  item's interface edges in the same memo.
- mimalloc's `realloc` keeps the old block when the new size is at least half of it, so a `Vec`
  shrunk with `into_boxed_slice` or `shrink_to_fit` can still hold twice its length. A buffer that
  stays memoized is allocated at its exact size up front instead.
- mimalloc commits its arenas lazily (`crates/ry/src/main.rs`): eager commit made the kernel zero
  huge pages that a short-lived process never touched. What allocation still costs is mostly
  setting up fresh pages, which scales with the memory the analysis keeps, and mimalloc's purge
  and huge-page options measured no difference.

### The check cache

The cold check already uses about 3.4 of 4 cores in every pass, so more parallelism cannot bring
it much lower. What a repeat run can skip is work whose result is already known, so `ry check`
records each file's findings in `.ry_cache/check.json` (`crates/ry/src/check_cache.rs`). The salsa
memos themselves are not persisted. Salsa 0.28 exposes no dependency graph to save, and its
persistence feature would need every memo to serialize, including the process-wide `Ty` table.

The record has a key fingerprint and, for each file, four facts:

- **Key**: the ry version and build, the configuration, the project root, the data files,
  `NAMESPACE`, `DESCRIPTION`, and the project's stubs. A different key checks every file.
- **Content fingerprint**: the file's text.
- **Surface fingerprint**: two parts. The first is the text with the body of each top-level
  function definition left out, unless the body contains `<<-`, `->>`, or a call that declares or
  attaches something (`assign`, `library`, `setClass` and the rest of `DECLARING_CALLS`). The
  second is the file's share of every project-wide fold, listed by
  `diagnostics::project_fold_shares`: its definitions, conditional writers, type declarations,
  `globalVariables` names, S3 generics and attached packages. A body can feed a fold, as a
  `UseMethod` call or a `#: @type` inside it does, so the shares come from the queries rather
  than from the text. A new fold over the project's files must add its share there.
- **Dependencies**: the files whose items the check read through interface edges, transitively
  (`semantics::file_dependency_files`), plus the files that its findings point into.
- **Imports used**: the `NAMESPACE` imports the file's code uses, which the `NAMESPACE` findings
  depend on.

A run with no changed file replays every finding without building a tree. A run whose changed
files all keep their surface checks the changed files and the files whose dependencies include
one of them. Every surface then matches the record, so the attached set comes from the record as
well, and the `NAMESPACE` findings are recomputed from the imports used, which costs a few
milliseconds. Any other change checks every file.

A package-only run, such as `ry check R` or the strict check of `ry load`, analyses only the
package files. Tests and scripts read the package namespace but add nothing to it, so leaving
them unparsed changes no package finding. They stay in the record: a file the run left unanalysed
keeps its recorded content, surface, findings and imports used while its text is unchanged. When
its text did change, the record keeps no surface for it and marks every unanalysed file
unchecked, so the next run that analyses them checks them all. The `NAMESPACE` findings read only
the imports the analysed files use, so a package-only run and a full run can disagree on them
(an import only the tests use is unused to the first and used to the second). A replay therefore
recomputes them when the package files' imports differ from all files' imports. On the benches, a file's dependency closure has a median size of 0 and a maximum of 1, so
an edit inside a body rechecks one or two files.

A partial run still parses every file, because the project-wide maps fold every file's items. So
it warms those folds across all cores, and that parse, about 0.3 s of the 0.45 s total, is its
floor. Removing that floor would mean persisting the item trees. When invariants are on
(`syntax::testing::enable_invariants`), every run that reuses findings is compared with a fresh
run and panics on any difference. The seeded edit simulation in `cli.rs`
(`cached_checks_match_fresh_checks_across_seeded_edits`) drives that comparison.

### From item to scheme

An item's subtree lowers to HIR, with spans relative to the item. Its annotations lower
separately: a `#:` block and an inline annotation go through the same type lowering into the same
`Annotation`, and a statement annotated both ways gets one merged annotation
(`annotations::merge_annotations`), so the checker never knows which spelling it was given. The
rules that need no types live in `syntax::annotation`: which `#:` block attaches to a statement,
how a declared parameter pairs with a formal, and which form wins a slot. The runtime calls the
same module when it registers a function's signature, so the checker and the runtime cannot
disagree about what a function declares. Names
in HIR and naming are interned `Symbol`s (a 4-byte `Copy` id). Literal text is not interned,
because the interner never frees and an edit inside a string literal would otherwise keep every
spelling it passed through; it lives in the module's own text buffer, addressed by a `Copy`
`Text` span. The HIR module is
struct-of-arrays: an `ExprId` indexes separate kind and range arrays, and every list (arguments,
parameters, statements, record fields, match arms, pattern lists) lives in one side array per
element type that the expression addresses with a `Span`, so every HIR kind is `Copy` and a scan
over kinds never touches ranges. A name written at a site with its own range (an argument name, a
field after `$` or `@`, a loop variable, a tag) is a `NameId` into a `NameSite` side array that
holds the symbol and its range together, and every id is a `NonZeroU32`, so an optional child costs
no extra word. An expression kind is 16 bytes and an argument 8. Lowering reads the tree through
borrowed `syntax::NodeRef` cursors into a per-thread scratch (`hir::Parts`) that keeps its capacity
across items. Sealing copies every array and the text into one allocation of exactly the needed
size, a word header of section starts followed by the sections, so the memoized `Module` costs one
allocation and no spare capacity. When every range in the item ends below 64 KiB, as almost all
do, a range packs into one word. Naming
then runs the variable model, in which a variable is a mutable slot with reaching-write flow.
Finally, the inference walk, `item_check`, produces the expression types, the type errors, the
strict-mode origins, and the scheme the item exports.

**An inference variable must never leave the item that created it.** A variable is an index into one
item's inference table, so nothing that outlives that table may hold one. An exported scheme is
closed at the item boundary by `close_scheme`, and a value cached across a rollback of the table has
its variables erased first. A leaked id is not merely a crash risk: in a reader's table it either
points past the end, which panics, or it silently names some unrelated variable of the reader's own,
which is worse.

To keep that from happening by accident, every transformation that rebuilds a type (resolution,
substitution, generalization, erasure) goes through one structural map, `map_child_types`. That way
no walk can handle a member, such as a parameter's default, that another walk forgets. Reads go
through the matching `for_each_child_type` and `Parameter::types`.

### Cyclic definitions

When definitions refer to each other in a cycle, the package interface resolves the whole group
through one canonical fixpoint, so the result never depends on which member happened to be queried
first:

1. `item_group` finds the cyclic group an item belongs to. `item_interface_edges` gives each
   item, definition or top-level statement alike, the items its reads resolve to, following
   exactly the resolution a check performs: the frame binder in the item's own file, then the
   test and package layers, and every conditional writer of a name that has no unconditional
   winner. One iterative Tarjan walk per file (`file_interface_groups`) starts from the file's
   items and visits only what they reach, which is enough, because a cycle through an item lies
   inside that item's reachable set. Members come back in canonical project order, and an item
   finds its group by its position in the file's item tree.
2. `scc_exports` starts every member at `Unknown` and runs Jacobi rounds until the table of
   exports (each member's scheme and the schemes of the top-level names it binds) stops changing.
   If the round cap is reached, every member is pinned.
3. `item_check` adopts the canonical exports as the one exported truth.

A member check inside the fixpoint never re-enters `item_check`, so no salsa cycle forms. Salsa's own
cycle recovery remains only as a backstop, for a reference the static graph cannot see.

The graph must cover every edge a check can follow, not just the common ones. Salsa never backdates
the memo of a query that took part in a cycle, so when an edit breaks a cycle the former members
re-run with no cycle heads and salsa's backdating check panics on any member whose value did not
change. A missing edge turns into exactly that panic on the edit that removes and restores it.

That backstop has a trap of its own: **a cycle recovery must not pin a value that depends on the
round that produced it.** Salsa stops iterating when a recovery returns the same value as last time,
so the value pinned at the round cap has to come back unchanged. A pin computed from the freshly
recomputed value does not: the parts it leaves alone keep moving with the cycle, every round
differs, and salsa runs into its own `MAX_ITERATIONS` and panics. Pin a constant, or re-pin what was
already returned, and when the pinned type is a composite, cut every surface it exports, not just
the obvious one.

### Keeping project-wide work cheap

A check never names the whole project. The interface graph is walked per file, from that file's
items, and the walk reads each item's edges from `item_links`, whose value survives an edit that
only shifts ranges.

The one map that does cover every package item is the set of conditional writers, the items that
write a top-level name other than their own, which a read with no unconditional winner joins.
`file_conditional_writers` builds a file's share by screening each item's syntax first: only an
assignment operator, `for`, a destructuring `%<-%`, or a by-reference `setnames` outside a nested
function can write a top-level name, so an item with no such token besides its own binding needs
no naming pass. Only the rest go through `item_top_level_names`. The screen may pass too many items
but never too few, and the fuzz battery checks it against naming on every generated program.

The project-wide maps reach per-file data through one tracked memo per file (`file_definitions`,
`file_conditional_writers`, `file_type_definitions` and the others `warm_project_folds` lists),
never through per-item queries. After a keystroke salsa must
confirm that a project-wide map is still valid, and it does so by checking the map's dependencies.
Read per file, that is about 500 memos on a 300,000-line project, and every one outside the edited
file passes on durability alone (see "The language server"). Read per item, it was about 56,000
memos, which cost every editor request about 35 ms right after a keystroke.

A check never reads a project-wide map either. It reaches other items through three projections,
and every host uses the same three:

| Projection | Keyed by | Answers |
|---|---|---|
| `item_links` | item | where the project binds each name the item's naming lists (the binders in its file, a sourced file's definition, the test and package winners and conditional writers), and the item's interface edges |
| `package_name`, `test_name`, `frame_binder` | name, or item and name | the same bindings for a name the check constructs itself, such as an S3 method's (`Ops.Date`), which no naming lists |
| `file_interface_groups` | file | the cyclic groups of the file's items |

`item_links` is the item's own projection of the project-wide name maps. A new top-level name
anywhere re-runs it for every item, and it backdates unless one of the item's own names moved, so a
new definition re-checks only the items that read its name. Folding the edges into the same memo
costs one memo per item instead of two.

A constructed name is usually bound nowhere, and a per-item `frame_binder` memo for each such
lookup comes to about 100,000 memos (about 18 MB) on the finance bench. An item of a package file
needs none of them. Every top-level name a package file binds is a definition or a conditional
write of the package layer, so when `package_name` finds neither, the item's file has no binder for
the name, and the lookup stops there without a query. The helper files' layer (`test_name`) holds
a helper file's names the same way. An item of a test file or a script has no layer over its
file's names and asks `frame_binder`, which reads the binders of that one file.

No lookup folds the binders of every file. A file's binders come from naming each item that writes
a name besides its own, so such a fold would lower those items in every file of the project, test
files included, before the first file's check could finish.

`crates/semantics/tests/test_incremental.rs` pins these contracts: the queries an edit re-runs, the
ones a new definition re-runs, that a check resolves the names it spells through its links
without a `frame_binder` query, and that a package file's check reads the binders of its own file
alone. Under
`syntax::testing::enable_invariants()` every linked lookup is compared with the per-name answer,
every unbound verdict with `frame_binder`, and every item's group with a walk from that item alone.

A host that only reports diagnostics, such as `ry check`, answers `Db::editor_facts` with false,
so its item checks leave out expression and pattern types, literal expectations, loop variables,
selected overloads and callee targets, which only editor features and rewrites read. `ry migrate`
reads expression types for its rewrites, so it analyses on `RootDatabase::for_command(true)`.

A name being typed is a new top-level definition on every keystroke, so the folds that a new name
changes are kept to one pass: `package_definition_index` folds each package file's definitions
into the winners and, in the same loop, the names defined more than once. The duplicate check reads
those through `duplicate_binding_map`, a copy that is empty in a healthy project and so backdates.
Names in these maps are interned `Symbol`s whose hash was computed once at interning, so a fold
hashes four bytes per name and never rereads the spelling.

Types are hash-consed, so type equality is an id comparison and nothing is ever deep-cloned. Deep
resolution cuts any cycle to `Unknown`. It returns at once for a type whose id says it holds no
variable and applies no alias. A walk that has taken 64 steps starts a table of the subtrees it
finishes, which lives for that one call, so a subtree shared through variables or aliases
resolves once. The decision log records that design.

### The package-load index

`ry load` reads a package through its own input, `PackageSources`: the `R/` files in collation
order plus the `NAMESPACE` and `DESCRIPTION` text, as separate fields so a `NAMESPACE` edit never
re-derives an item. The index has three layers, and each depends only on the one below:

- `file_load_index` erases one file's inline annotations, keeping every line break so line numbers
  match the file on disk, and records each item's position in that text. `file_items` then gives
  each item its content hash (of its erased text), whether it is a function definition, the names
  it writes and reads while it runs (`item_immediate_reads`, which excludes reads inside the
  functions it defines), and the S4, R6 and S3 registrations it makes. Both read one file only.
- `file_binds` reduces one file to the names it binds and how, and `package_bindings` joins those
  into which names are bound lazily: a name whose one writer is a function definition. A body edit
  leaves a file's binds equal, so salsa backdates them and the join, which is the firewall that
  keeps an edit from reaching every file's bindings.
- `package_index` joins the files and the bindings into what a plan diffs.

A reload depends on all of a package's files, so salsa would revisit every file's memos after any
edit. The loader avoids that with durability: a file starts at high durability and drops to low on
its first edit, and salsa skips verifying memos that depend only on inputs at a durability nothing
changed at. The diff then looks inside only the files whose hash changed, plus the eager code of
every file, which is all an edit can reach. Lowering a file's durability makes its first edit verify
every memo once (about 20 ms at 326,000 lines), and later edits of that file skip the check.
`test_incremental.rs` pins the per-file contract.

Before the console or `ry load` starts R for a package, it calls the loader's `prepare`, which reads
the sources and derives this whole index on its own threads while base R loads. The first
`load_all()` then waits for that to finish and only diffs.

Deriving the index means parsing every file, which is most of a cold start, so the first plan of a
package in a process caches the index across processes. The entry lives in the user's cache
directory (`$XDG_CACHE_HOME/ry/load-index`, one file per package root), and its key hashes the
`DESCRIPTION`, `NAMESPACE` and `R/` texts, the strict setting, and a fingerprint of the parser, the
analysis, the shipped stubs and the loader that `crates/loader/build.rs` computes. On a hit the
loader syncs the texts into the database without parsing them and hands the backend each file's text
instead of its tree; ry's runtime then parses only the code it lowers, one top-level item at a time,
which gives the same tree as the whole file would. Meanwhile a detached thread parses every file on
a database handle of its own, so the first reload finds the parses memoized, and an edit that reaches
the database first cancels it.

A fresh load on ry's runtime may also hold back the package's S4 definitions until first use. The
loader decides which (`DeferredS4` in `crates/loader/src/loader.rs` states the rules), and the
methods package's class and generic lookups in `crates/interp/src/methods.rs` force them on a miss;
[package loading](/reference/package-loading.md#s4-definitions) has the contract. The first unit holds
the classes and the methods on other packages' generics, and each generic the package defines is a
unit of its own that a placeholder binding runs on first use (`.ry_force_s4` in `runtime.R`). Nothing
about the definitions is cached across processes.

Running the definitions and using the classes goes through the methods package's native core
(`crates/interp/src/methods_core.rs`). The methods package keeps GNU R's R code and its metadata
objects (class representations, the `.__C__` and `.__T__` tables, the method tables), which stay the
only source of truth; the core reads and writes those same objects. When the methods namespace
loads, `install_fast_paths` puts a native prologue from `methods_core.R` in front of the body of each
function it covers, but only if the function's hash matches the GNU R version the prologue was
written against. A prologue either does the function's whole work, with the same result and
effects, or returns without observable effect and lets GNU R's body run, so a case the core does
not handle stays exact. When the methods package's version is the bundled one, every prologue must
install, so a mismatch stops start-up with an error; on any other version a mismatch only leaves
GNU R's body in place.
The covered functions are the object operations (`new`, the default `initialize`, `validObject`,
`is`, `extends`), the definitions (`setGeneric`, `setMethod`, `setClass`, `setClassUnion`) and
`cacheMetaData`. A prologue may also do part of the work and hand the rest to GNU R's code one item
at a time: `cacheMetaData` handles the classes and generics that need nothing or only a table merge,
and runs GNU R's loop body for each of the others, so the order of effects is GNU R's.

### Diagnostics

Diagnostics come in two layers over one source of truth:

- `parse_stage_diagnostics(file)` holds the syntax errors, the typing-directive errors, the
  refusals of malformed `#:` blocks and inline annotations, and, while inline annotations are turned
  off, a warning at each one. All of them are pure functions of the parse and `AnalysisSettings`.
- `file_diagnostics(file)` adds the naming findings (unresolved and unused names) and the type
  errors. The strict-mode findings are rendered separately by `strict_diagnostics(file)`, which is
  empty unless `file_is_strict` holds (`[check] strict` or the file's own directive).

The host, meaning the server or the CLI, then gates diagnostic classes by configuration, applies
per-file `# typing:` overrides, escalates unresolved names to errors under strict mode, appends the
lints, and honors `# ry: allow(...)` suppression comments. The server and the CLI share this
assembly in `crates/ry`, so they cannot disagree.

## The language server

The server runs on two threads. The async-lsp frontend, on the tokio thread, turns every request and
notification into a job for a single dedicated worker thread. The worker owns the salsa database
and all the documents, and the database handle deliberately never leaves it.

- **The latest edit wins.** A notification that changes an input first cancels the worker's salsa
  cancellation token and then enqueues its job, so a query in flight unwinds cooperatively. Every
  later job starts on a fresh, cheaply cloned storage handle, because a cancellation is consumed by
  whichever query it killed. A best-effort feature answers with nothing when cancelled, but an
  authoritative answer never degrades: pull diagnostics return a retryable `SERVER_CANCELLED` with
  `retriggerRequest: true`, and rename returns `CONTENT_MODIFIED`.
- **Diagnostics are pushed in two waves.** On every sync the server immediately publishes the
  parse-stage classes and the lints, tagged with the document version. Until the startup warm-up
  (below) ends, the first wave holds only the parse-stage classes, because the lints read the
  project-wide name maps. At idle time it publishes the settled full set, without a version.
  Without a check record, the settled set is a faithful superset of the first, so the second wave
  only ever adds findings.
- **The first wave can come from the check record.** At initialize the worker loads
  `.ry_cache/check.json`, computes the record's key exactly as `ry check` does
  (`check_cache::key`, with the server's own binary), and runs `check_cache::plan` over the
  project's files, reading the scripts it does not load from disk. A changed file the server did
  not load has no surface there, which plans a fresh run and restores nothing. Every loaded file
  the plan reuses gets its recorded findings, and the first wave shows them instead of the parse
  stage until the settled wave replaces them. The server analyses only the package files and the
  open documents while `ry check` analyses every file, so the two can differ in a finding that
  depends on a script, and the settled wave corrects it. Any input change (an edit, a watched-file
  event, a stub or `NAMESPACE` buffer, a file opened with other text) drops every restored finding,
  because a change to one file can change another's findings. Restoring the record for a
  300,000-line package costs about what a replaying `ry check` does, about 0.2 s. A client that supports pull diagnostics gets no pushes at
  all, and instead receives result ids derived from a content hash, with unchanged reports.
- **Idle work has an order.** At startup the worker first runs the warm-up: across all cores, it
  scans every file's `library()` and `require()` calls and applies the attached set once, then
  derives the per-file folds behind the project-wide maps (`warm_project_folds`). Every semantic
  wave reads those maps, and one thread would parse the whole project to derive them, so settled
  publishes wait for it. The warm-up names no item: a file's check names only the items its
  interface reaches, so a file opened at startup settles in about 200 ms on a 300,000-line
  package. After the warm-up the worker rescans the `library()` calls of any file whose scan a
  cancellation interrupted, because the edit that cancelled it may belong to another file and
  nothing else would rescan this one. Then it serves any settled publish it owes, most recently
  edited file first. Then it re-derives `project_global_names`, the set goto-definition and
  references consult on every library call, whenever the warm-up or a durability drop left it
  stale: salsa never backdates across a durability drop, so the fold re-runs once per opened file
  (about 20 ms at 300,000 lines) and would otherwise land on the first request. Next it lowers the
  durability of documents opened since (see below). With
  nothing else to do, it checks the rest of the workspace in parallel batches and never publishes
  that work. Idle work that gets cancelled is requeued.
- **Parallel idle work stops between files.** It runs on one cloned database handle per core, and
  each thread checks before every file whether a job is queued. Cancelling the threads' salsa
  tokens is not enough on its own, because salsa clears a handle's cancellation flag whenever a
  top-level query returns, so a flip that lands at the end of a query is lost.
- **A request preempts idle work.** A request is queued first and then cancels the idle unit in
  flight, if any, so a hover never waits for a background file to finish. A request never cancels
  another request or a sync.
- **Only package files stay loaded while closed.** At startup the server loads the files under `R/`,
  and a closed or disk-changed file stays in the project only if it is one of them. Test, helper and
  script files are analyzed while they are open, so a session never holds a file a fresh start would
  not.
- **The worker is deterministic and steppable.** Everything it sends the client goes through an
  `Outbox`, which is the socket in production and a recorder in the
  [simulator](/development/robustness.md#the-language-server-simulator), and `run_job` is one step of
  its loop. Collections whose order reaches the client are ordered.
- **A coherence failure panics.** A failed document sync is unrecoverable, and a panic that escapes a
  worker job terminates the process deterministically rather than going on to serve corrupted state.
  An IDE feature lookup, by contrast, must never panic.
- **Only edited files change at low durability.** A workspace file enters the database at
  `Durability::MEDIUM`, an open document is lowered to `LOW`, and closing it restores `MEDIUM`. The
  stubs, settings, package metadata and data-file inputs are `MEDIUM` too, because a memo is only
  as durable as its least durable input and nearly every memo reads the stubs. An edit then
  revalidates every memo that reads only other files by durability alone, without walking its
  dependencies. `test_incremental.rs` pins that an edit never revisits the other files' trees or
  items.
- **The project file list is `LOW` from the start.** Salsa never backdates a memo whose durability
  drops, so if the project-wide maps started at `MEDIUM`, a file's first `LOW` write would change
  them, then every `package_name`, then every item check in the project (about 2 s on a
  300,000-line project). They read `ProjectFiles`, so a `LOW` list keeps them `LOW` throughout, and
  the first edit re-derives only the edited file's own memos. `didOpen` leaves text that matches
  the disk untouched, which would otherwise be a `MEDIUM` write that invalidates the whole project,
  and idle time rewrites it at `LOW` so that the first keystroke does not pay for that file's share.
  A setter that names no durability keeps the field's current one, so every write to a source file
  names it.
- **Positions convert only at the protocol edge**, against the target document's own text, in the
  negotiated encoding. That is UTF-16 by default, and UTF-8 when the client offers it. Signature-help
  label offsets are always UTF-16, as the LSP specification requires.
- **Stub buffers and `NAMESPACE` files are served standalone** and never enter the database. An
  `.Rtypes` buffer gets loader-problem diagnostics, semantic tokens, and goto-definition for type
  names, and a `NAMESPACE` file gets import validation.

## The start-up image

A runtime session that is not interactive and sources no profile starts from an image of the
session as it stands after `.First.sys()`, which [the CLI reference](/reference/cli.md#start-up-image)
describes for users. `crates/heap/src/image.rs` writes and loads the heap part,
`crates/interp/src/image.rs` saves the interpreter state outside the heap and decides when an
image applies, and `crates/interp/src/session.rs` either loads the image or starts the slow way.

- **Build output, keyed by build.** `ry-rcmd build-base` writes the image as its last step, into
  the base tree at `library/base/R/ry-start.<build>.image`, by running `ry-rcmd write-start-image`
  with an emptied environment. `<build>` is the runtime's identity (`RY_RUNTIME_ID`), and
  `crates/rcmd/src/images.rs` owns the naming. A release embeds the tree with its own build's
  images only. A session never writes an image, and nothing keys it on the run-time environment.
  `ry-rcmd start-image BASE --libr FILE` (`just start-image`) writes a build's images over a built
  tree, and so does assembling or reusing an R home (`ry-rcmd assemble`) when the tree holds no
  current image of the given `libR`'s build: the home links that image, so a missing one would
  make every start in the home slow. `libR` exports
  `ry_start_image_current`, which applies the loader's own format and identity checks. A current
  image is only read, so a home assembles over a read-only base tree. A stale one is rewritten
  under a lock file in the base tree (`.start-image.lock`), so homes assembled at once take
  turns, and over a base tree that cannot be written, assembly fails and names `just start-image`.
- **Base code images.** Right after the start image, the same step starts a session from it,
  fetches every record of every lazy-load database of the base-priority packages by its key
  (start-up has already forced some bindings, so forcing bindings would miss their records), and
  writes the images (`R/<package>.<build>.ryc`, `R/sysdata.<build>.ryc`) with the uncompressed
  records alone, so those fetches read them in place instead of inflating them. They hold no
  compiled code: for untrained base code, decoding it costs more than compiling the closures a
  session calls. Every other package in the tree (the
  recommended packages, such as MASS and Matrix) then gets the image an install writes, untrained,
  so a build that did not install them still loads them from an image. All of them are written for
  the same build as the start image.
- **Several builds per tree.** Builds never overwrite or remove each other's images while they
  are in use, so two builds compared on one tree both start fast. After writing its images, a
  build prunes the tree under the lock to the images of `images::KEPT_BUILDS` (4) builds: its own
  and the three others that wrote theirs most recently, a build's images being removed whole. The
  order comes from a ledger in the tree (`library/base/R/ry-builds`, one build per line, oldest
  first), never from file times, and a build the ledger does not name counts as the oldest, in
  name order. A running session holds a shared lock on its build's start-up image, and the pruner
  removes a build only while it holds that file's exclusive lock, so a build in use is kept
  however old it is. The ledger is rewritten through a temporary file and a rename, and the writes
  and the prune all happen under `.start-image.lock`, so two builds starting at once take turns.
  Unkeyed images (`ry-start.image`, `<package>.ryc`) belong to runtimes that predate the keying
  and are left alone. One build's images take about 54 MiB in `target/base-r` (a 16 MiB start-up
  image and 38 MiB of code images).
- **One build per image.** The header carries the runtime's identity too (`RY_RUNTIME_ID`, a hash
  of the runtime's sources), and a loader skips an image whose header names another build, which
  `RY_IMAGE_LOG=1` reports. After a runtime edit, the new build's first start in a development
  checkout writes its images beside the old build's, which `RY_IMAGE_LOG=1` reports as
  `ry: image written:`. Installed packages are not rebuilt: when a session finds a package whose
  code image only another build wrote, it notes once per library and session which packages of
  that library to reinstall (`note_foreign` in `crates/interp/src/code_image.rs`).
- **The environment is read again on every start.** After the load, the start clears the options
  and runs the steps that read the process environment: the C-level options, `.Machine`,
  `.Platform` and the device variables, base R's profile, and the `.onLoad` hooks of utils,
  grDevices and stats, which set options from the environment only when they are unset. The
  profile's second `Autoloads` is detached, and utils' `osVersion` binding is unlocked around its
  hook, which assigns it. What the image holds is what those steps do not touch: base's
  namespace, the six default namespaces and their attachment, which read no variable outside
  `image::LOADING_VARIABLES`. A start that sets one of those runs the slow way.
- **Relocated paths.** The image records the R home it was written in (the base tree's staging
  directory) and its canonical form. The loader moves every string under either prefix to the
  directory the image file really lives in, which is where a slow start's resolved namespace paths
  and lazy-load databases point.
- **The heap's pages, used in place.** The first heap in a process reserves its small-object
  space at a fixed address (`heap::space::IMAGE_BASE`), and the writer moves every large object
  into a second fixed section, so every handle in the written pages is the address the object has
  in the loading process too. The loader maps the file's pages privately over the space and the
  large section (`fill_from` in `crates/interp/src/image.rs`; Windows reads them instead): nothing
  is decoded or patched per object, a page start-up never writes stays shared through the page
  cache by every process that loads the image, and a page is copied when it is first written.
  The file stores only the pages that hold an object, back to back. An empty page between them
  (left where compaction emptied a page below an environment or symbol that cannot move) is
  untouched zeroed memory in the loader, so it costs neither file bytes nor memory until an
  allocation claims it. The objects carry their tenured bits, so no minor or intermediate collection traces them and a
  major collection's marking leaves their headers alone.
- **Base R's closures arrive compiled.** Just before writing, the writer compiles afresh every
  closure bound in base's namespace and the six default namespaces
  (`bytecode::compile_namespaces_for_image`), and the image keeps that code. A lazy-load binding
  start-up never forced stays a promise. Every other code object ran during start-up and carries
  the writer's inline caches, type feedback and profile, so the image stores its source instead.
  A session therefore runs base R's functions as bytecode from their first call without
  compiling them, and the pages of code it never calls stay in the shared page cache.
- **Objects are laid out in touch order.** Before writing, the writer numbers every live object
  with a stable id and moves every movable small object into dense pages in a recorded order of
  first touch, so the pages a session reads at start-up are adjacent and the code it never runs
  sits together at the end. Fixed objects, symbols, environments, ALTREP objects and the objects
  that existed before start-up keep their addresses: frame filters, slot indexes, hash tables
  and layout tags hash symbol addresses, and the precondition digest covers the rest. Large
  objects go to the large section in the same order. Without an order the writer still
  compacts, in id order. Nothing is advised, because reading only the touched pages makes each
  page its own read and slows a cold start.
- **The touch profile.** The layout build (`tools/layout/optimize.sh`) writes the image twice,
  and three files in a layout directory carry the profile from the first write to the second.
  None of them is part of the image or the repository: the loader never reads them, and the
  layout build records a fresh profile for the binary it is building, so a change that
  renumbers the image's objects cannot leave a stale order behind. Every other image write
  (`just start-image`, the base tree's build, tests, CI, and the release builds other than
  x86-64 Linux, which have no recorder) passes no layout directory and lays the image out in id
  order, which removes the holes but not the scattering. Each step below refuses a file
  recorded for another image or numbering instead of using it.
  - `ids`, written by the first write (`ry-rcmd start-image BASE --libr FILE --layout DIR`), is
    little-endian `u64`s: the image's header checksum, the digest of its numbering (the object
    count and each object's size, by id), then each object's address and size, by id. Ids come
    from a depth-first walk over each object's handles (and an environment's bound symbols and
    values), seeded in this order: fixed objects, global roots, the precious set, the heap's
    globals, weak references, untracked objects, symbols sorted by name, interned strings sorted
    by bytes, then every other live object by address. The same start-up therefore numbers the
    same objects the same way in both writes.
  - A touch file comes from each training run of `tools/layout/train.py --image-touch DIR`,
    which sets `RY_IMAGE_TOUCH=FILE`. The runtime then protects the mapped image (x86-64 Linux
    only) and writes `image HEX`, the loaded image's header checksum, followed by one line per
    first touch of a 16-byte granule: the address in hex and a phase, 0 for start-up, 1 inside a
    namespace load, 2 otherwise. The mix adds an empty program to the training programs.
  - `order` comes from `tools/layout/image_order.py`, which refuses a touch file whose image
    line differs from the checksum in `ids`. It writes `numbering HEX`, copied from `ids`, then
    one decimal id per line, sorted by the earliest phase any run touched the object in, then by
    its earliest position relative to the run's length, then by id. The second write reads
    `order` and fails when the numbering differs from its own; ids it does not list follow in id
    order. An order only ranks objects, so one that names the wrong objects still gives a correct
    image, only a less compact one.
  - Writing the same base tree twice gives the same image bytes: the writer zeroes every byte
    the live objects do not determine (spare vector capacity, write-barrier cards, slot tails,
    and the process-local fields below) and places the intern table's entries in address order.
    Strings that name the base tree's path are part of the numbering, so training and the
    second write use the same tree at the same path.
- **The same starting state, or no load.** Fixed objects (the heap's well-known objects, the
  primitives by index, `package:ry` and libR's ALTREP classes) exist before start-up in both
  processes. The writer records a digest of its heap's allocation state at that moment
  (`Heap::image_precondition`), and the loader compares its own before it takes anything: equal
  digests mean every object the loading heap holds sits where the image holds the same object,
  so the mapped pages replace them without leaving a dangling handle. A different digest starts
  the slow way.
- **Process-local state lives beside the pages.** The few fields that hold a process's own
  addresses are cleared in the written pages and restored by the loader: environment frames whose
  bindings spilled into memory outside the heap, external pointers to builtin routines (carried
  as tokens), and the contents of the fixed primitives and ALTREP classes, which keep the
  loader's. The
  metadata carries the intern and symbol tables, the precious set, the global roots and the
  counters. Weak-pair lists load empty.
- **The writer refuses what it cannot carry.** Writing fails, and so does the base tree's build,
  when start-up left interpreter state behind (warnings, a call in progress, `.Random.seed` in the
  global environment, a new record layout, an open graphics device; `image::check_empty`), an
  external pointer other than a builtin routine, or a weak reference with a C finalizer.
- **The file is replaced, never rewritten.** The writer writes beside the image and renames, so
  a process that has the old image mapped keeps its pages.
- **Damage to the header fails before the heap changes.** The header (the heap metadata and the
  interpreter state) carries a checksum that is verified before the heap takes any of it, and a
  damaged header starts the slow way with a line on stderr. The pages are not checksummed, since
  reading them all would undo the sharing; like the runtime library beside it, the image is build
  output of the install. A failure past the precondition check (`LoadError::Partial`) would leave
  the heap half-built, so the start fails.
- **The base tree only.** Installed packages have no image. Lazy loading already fetches a small
  fraction of a namespace's objects, so their cost is setup work (S4 metadata, S3 registration,
  `.onLoad`), which runs natively behind hash-guarded prologues instead (`decisions/libr-rcmd.md`,
  "Installed packages load through native fast paths, not an image").

Interpreter state outside the heap is saved beside it: methods' dispatch globals, templates and
primitive method tables, the S4 extends and implicit-class tables, the loaded builtin libraries
and data.table's namespace. The host's environment beside base R (`tools:ry`, the loader's
runtime functions) is built when the image is written and carried in it, so a start attaches it
without parsing or evaluating its source. Per-process state, such as the command line, the temporary directory,
the RNG seed and the start time, is never in the image.

## Correctness and performance instruments

The fixture suites are the correctness contract. `crates/semantics/tests/test_incremental.rs` pins
the incrementality contracts by counting query executions after an edit, and
`crates/semantics/tests/test_perf.rs` asserts wall-time, memory, and keystroke-latency budgets
against a corpus of real files. [Testing](/development/testing.md) documents all of them, and
[Robustness](/development/robustness.md) documents the simulator, the fuzzers and the scenario tests
that check the whole toolchain against from-scratch runs and GNU R.
