Structure
This page lists what each source file in ry’s crates is responsible for. How the crates depend on each other, and where one phase hands over to the next, is in Architecture.
syntax
Section titled “syntax”The crate root is src/syntax.rs.
lexer.rs: the hand-written lexer. A#:annotation region comes out as structured trivia, so the parser can give it a grammar of its own.parser.rs: a recursive-descent parser with Pratt parsing for expressions. It also parses the annotation type grammar, and it recovers from errors at statement boundaries.tree.rs: the flat syntax tree the parser builds, its owned cursors (SyntaxNode,SyntaxToken) and borrowed ones (NodeRef), and the splice that a reparse uses.kind.rs:SyntaxKind, the enum naming every token and node kind, with a display name for each.ast.rs: typed views over the raw tree. Every accessor returns anOption, because a broken file can be missing any part.literate.rs: reads an.Rmd,.qmd, or.Rnwdocument as the R program its chunks contain. Every non-R character becomes a space, so the conversion preserves length and byte offsets need no translation.reparse.rs: incremental reparsing by splicing in a single statement. It is purely an optimization, and parse correctness never depends on it.native_values.rs: the facts about ry’s own records, tuples, tags, dictionaries and sets, and the R functions that accept them, which the checker and the runtime both read so the two cannot drift apart.testing.rs: the fixture harness every fixture suite shares, with the.testfile format, theRY_BLESSandFIXTURE_FILTERvariables, the rejection of duplicate case ids, and the#~~~~separator that splits one case into several files. It also reads the program corpus intests/programs/, about 2,000 R edge cases that run through the invariant batteries.
semantics
Section titled “semantics”The crate root is src/semantics.rs.
semantics.rs: the salsa database. It declares three of the database inputs,SourceFile,ProjectFilesandAnalysisSettings(the others,PackageMetadata,StubSourcesandPackageSources, live with the modules that own them). It also holds the item tree and its insertion-stable identities, the anchoring of each item to its syntax, the package interface computed by thescc_exportsfixpoint, and the item-span queries.hir.rs: lowering from the syntax tree to the per-item HIR, whose expression ranges are relative to their item.naming.rs: the variable model, in which a variable is a mutable slot. It resolves scopes, reaching-write flow, and captures, recognizes data-masked evaluation, and reports unused outputs.types.rs: the interned typesTyandTyKind, plus schemes, constraints, and union normalization.infer.rs: the inference table, with its union-find entries, unification, the directional compatibility relation, and memoized deep resolution.check.rs: the inference walk over one item. This is where the environment’s undo log, calls and overload probing, control flow, annotation enforcement, and strict origins live.annotations.rs: lowers a#:annotation node to an interned type, and holds the rules for annotation blocks.stubs.rs: the.Rtypescorpus. It parses the stubs, assembles them into a library of schemes, nominal types, masked verbs, and namespace exports, and reports loader problems.metadata.rs: thePackageMetadatainput. It parsesNAMESPACEandDESCRIPTIONand resolves imports and dependencies, and it reads the fullNAMESPACEdirective set a loader honors.load_index.rs: the package-load index behindry load, with its own input,PackageSources. Per item it derives a content hash, whether the item is a function definition, and the registrations it makes; per file, the text R reads (inline annotations erased) and every item’s position; per package, which names are bound lazily.exhaustiveness.rs: whether the arms of amatchcover the scrutinee’s type, by Maranget’s usefulness algorithm.top_level.rs: what a package file’s top level runs when the package loads, and which functions are declared@pure, which strict mode’s definitions-only rule and the lazy constants ofry loadboth read.sorted.rs: read-only sorted maps and sets for memoized results.lints.rs: the style lints and their configuration types.testing.rs: test support that other crates share. It holds the invariant battery for the semantic pipeline, which the fuzz harness and the coverage-guided targets run, the shared fixture database, andProbeDatabase, which logs query executions for the incrementality tests.diagnostics.rs: where diagnostics leave the crate. It produces the parse-stage set and the full per-file set, renders strict output, and holdsTypeRenderer, the one place types are turned into user-facing text.
The crate root is src/ide.rs, and src/testing.rs holds the editor features’ fuzz contract,
which the in-tree harness and the coverage-guided target share. Every editor feature in ide.rs is a read of semantics
queries: hover, goto-definition, references, rename, call hierarchy, inlay hints (including
parameter names), signature help, completion, code actions with their fixes (diagnostic_fixes,
fix_all), the outline, workspace symbols, folding, selection ranges, semantic tokens, and
navigation to annotation types and S4 definitions.
Definition, references, and rename share one occurrence engine, and completion uses the shared
smart-case matcher.
The crate root is src/rdb.rs. It reads installed R packages for editor navigation into packages
without ry type stubs, without an R runtime: the library directories, a package’s R code as
browsable source, and the help page behind a topic. ry links it.
unserialize.rs: R’s serialization format (versions 2 and 3, XDR and native binary) read into a plain tree, and the compressed containers of the lazy-load database (.rdx,.rdb).deparse.rs: values from that tree turned back into R source that parses.rd.rs: a parsed Rd help page rendered to compact Markdown for hovers.
format
Section titled “format”The whole crate is one file, src/format.rs: the formatter, which works on the syntax tree and
preserves your line breaks, together with its configuration types.
loader
Section titled “loader”The crate root is src/loader.rs.
loader.rs: the session that reads a package’s sources into the analysis, and the diff that turns the load index into a backend-neutralPlanof what to bind, run, remove, and register.runtime.R: the R code both backends share, which applies a plan to a session and providesload_all().r.rsturns a plan into an R list, encoding it as an RDS file for GNU R, andgnur.Randry.Rare the hooks of the two runtimes.testing.rs: the fuzz contracts for reloads and the load index, shared by the in-tree harness and the coverage-guided targets.
oracle
Section titled “oracle”The library root is src/oracle.rs, and the binary root is src/main.rs. It runs one R file under
GNU R and under ry’s engines and compares stdout, messages, warnings, errors and the last value;
driver.R is the script that produces each engine’s report. fuzz.rs generates the programs for
differential fuzzing, packages.rs runs the corpus packages’ own tests on both runtimes,
install.rs installs them from source on ry, and bench.rs compares the generated benchmark
packages’ tests test by test. Corpus, oracle and benchmark generator
describes how to run it.
The library root is src/ry.rs, and the binary root is src/main.rs.
main.rs: the dispatch of every subcommand and the exit-code contract.command_line.rsis the clap model, and resolvesry run,ry Rand thery replalias to one session shape.preflight.rs: the check that runs before ry executes R code. It prints the inline-annotation warnings and enforces strict mode, and leaves everything else tory check.cli.rs: the implementations ofcheck,fmt, andry debug ast.checkassembles the project, renders the output, and reportsNAMESPACEproblems and stub overrides.server.rs: the LSP server, all of it: the frontend and worker threads, document sync, push and pull diagnostics, every feature endpoint, semantic tokens, and the stub andNAMESPACEbuffers.server/simulation.rsruns that same server under a seeded scheduler (Robustness).config.rs: finds and parsesry.toml.diagnostics.rs: assembles diagnostics for the server and the CLI alike, applying configuration gating, strict escalation, and suppression comments.namespace.rs: parses and validatesNAMESPACEimports.position.rs: the line index, converting between byte offsets and line and column. A column can be counted in bytes, characters, or UTF-16 code units: the CLI reports characters, and LSP defaults to UTF-16.stats.rs: the performance diagnosis behindry debug analysis-stats.repl_completer.rs: completion for the interactive console.load.rs: package loading in R sessions: the.Callroutines and file watcher behind thery Rconsole, the plan emitter behind the hiddenry load --emit-plan, and the R code that installs the runtime, byte-compiled once and cached.
The crate root is src/repl.rs. This crate backs ry run and ry R, and it finds and loads the
system’s R, or ry’s runtime library, at runtime. That is why the rest of the workspace builds, and analyzes R, on a machine
with no R installed.
repl.rs: the session, with evaluation, the read-eval-print loop, and the exit codes.libr.rs: the binding to R’s shared library, resolved at runtime with no build-time link, including the registration of.Callroutines that live in the ry process.console.rs: the terminal front end.gnu_r_home.rs: where a GNU R installation lives, forry R, whenR_HOMEnames none.
repl.rs also finds ry’s base R (base_r) and extracts the copy built into ry into the cache.
The crate root is src/heap.rs. It holds R’s object model and the garbage collector, and knows
nothing about C.
heap.rs:Heap, with allocation, the barriered stores, environment bindings, the roots (thePROTECTstack, the precious multiset, the handle stack, globals andRootSource), and the collector itself: marking, ephemeron processing for weak references, and the young and full sweeps.object.rs:Sexp, the type codes, the header and payload layouts, and the raw accessors.space.rs: the small-object space: one reserved address range cut into 64 KiB pages of one size class each, with the mark, allocated and finalize bits in per-page bitmaps outside the objects, and the platform’s page reservation (mmaporVirtualAlloc). The range starts at a 2 MiB boundary and is committed 2 MiB at a time, advised for transparent huge pages on Linux.records.rs: the tag-name and layout interners behind records, tuples, tags, dictionaries and sets.strings.rs: the weakCHARSXPintern table and the encodings.env.rs: environment frames, which map symbols to binding cells.arena.rs: theR_allocarena and itsvmaxgetwatermarks.
numeric
Section titled “numeric”The crate root is src/numeric.rs. It holds the numeric kernels that take and return plain
numbers, so it sits below interp and an edit to the evaluator recompiles none of it. The root
answers which GNU R release is emulated (r_at_least), because a few results changed between
releases.
rmath.rs: R’s nmath library: the densities, distribution functions, quantiles and the special functions behind them.long_double.rs: x87 extended-precision arithmetic, for the sums GNU R accumulates in along double.vector_math.rs: the elementary functions over whole slices.fft.rs: Singleton’s mixed-radix FFT.linpack.rs: the LINPACK routines base R calls.c_format.rs: C’sprintfnumber formats.
interp
Section titled “interp”The crate root is src/interp.rs. It is the R interpreter: it evaluates R code on heap’s
objects and implements base R’s primitives and internals in Rust. It knows nothing about C;
libr installs it and supplies the native-code builtins. The root holds the interpreter state
(Interp, the frame stack, the host Hooks).
The engine links no table engine, rasterizer or HTTP client. The crates that do (stats,
graphics, frames and services) sit above it, and each has an install() that a host calls
after interp::install: it registers the crate’s builtins and native routine tables, and fills
the engine’s table of function pointers for the few calls the engine makes into that crate.
eval.rs: the AST evaluator: variable lookup, promises, calls and argument matching, closures andon.exit, the control-flow specials,substituteand complex assignment.bytecode.rs: the tier-0 compiler and register machine, its guards, inline caches and type feedback.unwind.rs:UnwindandJump, the non-local exits as Rust errors.conditions.rs: condition signalling, handler and restart stacks, the default error and warning handlers, and the traceback.dispatch.rs: S3 dispatch (UseMethod,NextMethod, group and internal generics) and the hooks S4 dispatch uses.methods.rs: the methods package’s native routines and S4 dispatch.methods_core.rswithmethods_core.R: the native core of the methods package, which runsnew(), the defaultinitialize(),validObject(),is()andextends()in Rust.envir.rs: environments, thesys.*functions, options, namespaces and the apply family.builtins.rs: the builtin registry (BuiltinDef,Call) and argument matching for builtins.values.rs: object constructors and accessors the rest of the crate shares, andGlobals.lower.rs: R source to R language objects, throughsyntax’s parser.deparse.rs,format.rs,print.rs: deparsing, number formatting and printing.arith.rs,coerce.rs,subset.rs,strings.rs,sort.rs: base’s vector internals: arithmetic and math, coercion, subsetting and assignment, strings and regular expressions, sorting and matching.linalg.rs: the matrix products (ry’s own kernels, in reference BLAS’s summation order), BLAS and LAPACK (opened at runtime), and base’s LINPACK routines.distributions.rs: the probability distributions.random.rs: the random number generators.connections.rs,serialize.rs,system.rs: the connection table (theRconnstruct native code sees, the built-in connection kinds, custom connections, sinks andcat), serialization and the lazy-load database, and the operating system (files, time, the environment).native.rs: the foreign-function builtins for a session without libR, and base’s routines.builtin_libraries.rs: the default packages’ routine tables, whichdyn.loadhands out instead of loading their shared libraries, andregister_routines, through which a crate above the engine adds a package’s table. The engine keeps five tables itself: methods, the distributions of stats, utils (utils.rs), tools (tools.rs) and parallel.extensions.rs: the function-pointer tables (Graphics,Tables,Viewer,Transport) the engine calls into the crates above it through. Each entry defaults to what a session without that crate does.http.rs: the request and error types of the HTTP transport, and the calls that go through its table.platform.rs: every operating-system call the interpreter makes, with a Unix and a Windows implementation of the same functions, so a libc use outside it is a Windows build error.session.rs: start-up (the base R treesystem.rsfinds, the profiles, the default packages) and running a script.parallel.rs: the parallel package’s native routines.rd.rs: the Rd parser behindtools::parse_Rd.annotations.rs: the run-time checks that annotations become at a function’s boundary.records.rs: records, tuples, tags andmatchat run time.arrays.rs:List<T>, an R vector with the heap’s list bit set, and the strict rules that go with it;numbers.rs: the native number types.standard_library.rs:Result,List,Str,Num,Bool,DictandSet, typed bytypes/std.Rtypes.testing.rs: the fuzz contracts of the interpreter, shared by the in-tree harness and the coverage-guided targets.
The crate root is src/stats.rs. It holds the stats package’s native routines, and install()
registers their table for the package stats.
routines.rs: the routine table, and the routines that need no file of their own.models.rs: model frames, model matrices and the fitting routines behindlmandglm.optimize.rs,nlm.rs,lbfgsb.rs,port.rswithport_minimizer.rs: the optimizers behindoptim,optimize,integrate,nlmandnlminb.arima.rs,loess.rs,spline.rs,cluster.rs,deriv.rs,hypothesis_tests.rs: time series, local regression, splines, clustering, symbolic derivatives, and the exact distributions of the test statistics.
graphics
Section titled “graphics”The crate root is src/graphics.rs. It is R’s graphics stack, and install() registers its
builtins and the routine tables of the packages graphics and grDevices.
engine.rs: R’s graphics engine, with the device table, the graphics systems registered with every device, and theDevDesclayout that device packages fill in.routines.rs: the native routines of the graphics and grDevices packages.plotmath.rs: mathematical annotation in plot text.render.rs: what a graphics device draws and the image files it writes.testing.rs: the fuzz contract of the renderer.
frames
Section titled “frames”The crate root is src/frames.rs. It binds the table engine to the interpreter as the
DataFrame type, and install() registers its builtins.
frame.rs: theDataFramevalue, its readers, writers and conversions.dataframe.rs: the Polars-styleDataFrameverbs and column expressions.viewer.rs: the paged table viewer the console server opens for a data frame.
services
Section titled “services”The crate root is src/services.rs. It holds the interpreter’s network client, and install()
makes it the engine’s HTTP transport.
http.rs:url()connections anddownload.fileover HTTP and HTTPS.
The crate root is src/jit.rs. It is the optimizing tier: it compiles hot tier-0 bytecode to
machine code with Cranelift, and hands the activation back to tier 0 when a speculation fails.
analysis.rs: the speculation plan, from an abstract interpretation of a body’s bytecode over R value types.codegen.rs: machine code for a plan.runtime.rs: the activation record compiled code runs on, and the helpers it calls for everything that allocates, runs R code or can fail.
The crate root is src/typed.rs: the entry point from jit’s native::call, the table of
compiled code per closure and argument types, and the conversion of R values to native values and
back.
lower.rs: a closure’s body as typed IR, with local functions inlined at their calls and every global function guarded by the binding it resolved to.ir.rs: the typed IR: types, instructions, blocks and the builder.ownership.rs: reference counts from liveness, in the style of Perceus.backend.rs: Cranelift code for the IR, and the trampoline the entry point calls.values.rs: the native values (Str,List,Dict) and the runtime functions compiled code calls.
The crate root is src/ssa.rs: the one mid-level SSA IR that jit and typed both lower to,
its builder, and the frame states an untyped speculation exits through.
lower.rs: the one lowering from the IR to machine code through Cranelift, and the code module every compiled function lives in.
The crate root is src/libr.rs, and the crate builds libR.so as well as an rlib. It holds the
runtime state and R’s C API. Each exported function that can fail runs its body through
boundary::enter.
libr.rs: the process-wideRuntimeand its start-up.boundary.rs: the native boundary:Unwind,enterandraise, the trampolines that call native code, the handle scope, finalizer runs, and the Rust half ofnative.c.native.c: the C half: the context stack, the trampolines,ry_jump, the variadic functions, and the non-local exits native code sets up (R_UnwindProtect,R_ToplevelExec,R_ExecWithCleanup,R_tryCatch).build.rsexports what it marksRY_EXPORT.memory.rs: allocation, duplication,PROTECT, the precious list and multisets,R_allocand theR_Callocfamily.vectors.rs: vector access, the type predicates, coercion and number formatting,identicalandmatch.lists.rs: pairlists, calls, closures and promises.attributes.rs: attributes, classes and the header bits.strings.rs:CHARSXPs, encodings, symbols andRiconv.environments.rs: environments, variable lookup, bindings, namespaces and options.loader_backend.rs: the package loader’s backend for ry’s own runtime, where the plan never leaves the process and a lazy definition is a promise over the analysis’s syntax tree.eval.rs: installs the interpreter and itsEvaluatorhook, the evaluation entry points of the C API (with a small built-in evaluator for a host without the interpreter), the foreign-call builtins anddyn.loadfamily;.Call,.External,.Cand.Fortran; conditions; and the deparser for messages.dynload.rs: loading package libraries, routine registration,NativeSymbolInfolookup and C callables.macho.rs: on macOS, rewrites a GNU R package binary’s load command for GNU R’slibRto name ry’s runtime.extptr.rs: external pointers, weak references and finalizers.altrep.rs: the ALTREP class registry, dispatch for every accessor, and base R’s own classes (compact sequences, deferred strings, wrappers).frame_altrep.rs: a table column as an ALTREP vector.serialize.rs: R’s serialization format in every variant, the stream API, RDS files and lazy-load databases.connections.rs: theR_ext/Connections.hentry points over the interpreter’s connection table, and the trampolines that call a package’s own connection methods.classes.rs: the implicit class and the S4 slot and class functions.hashtab.rs: R’s hash tables, kept in heap objects.legacy.rs: names R 4.6 no longer declares that binaries built against older R import (TRUELENGTH,SETLENGTH,FRAME,R_PPStackand the like).graphics.rs: the C entry points of R’s graphics engine over thegraphicscrate’sengine.rs, so grid and device packages such as ragg run unchanged.arith.rs: NA tests, sorting, and the random number generator, which shares its state with the interpreter’s through.Random.seed.rmath.rs:Rmath.h: the densities, distribution functions, quantiles and random variates of R’s distributions, and the special functions under them.applic.rs:R_ext/Applic.h: theoptim()andnlm()optimizers, QUADPACK integration,findInterval,R_max_colandR_pretty.linpack.rs: the LINPACK routines R uses, includingdqrdc2behindqr()andlm.fit(), and the PORT machine constants.utils.rs:R_ext/Utils.h: file and temporary names, string predicates, and the printing and signalling helpers Fortran code calls.globals.rs: GNU R’s data symbols.embed.rs: ry’s ownry_entry points for an embedding host, declared ininclude/ry_embed.h, and the embedding functions ofRembedded.h.include/: the header set, written independently and source-compatible with R 4.6.BLAS.handLapack.hdeclare the system’s BLAS and LAPACK, which packages link themselves.
The crate root is src/table.rs. It is ry’s built-in typed table, an engine over Polars.
table.rs:Table, a PolarsDataFramewhose columns are restricted to the types that have an R vector type, so every column crosses into R and back without loss.convert.rs: conversion between Arrow columns and R vectors, includingNA.query.rs: the query IR, and its lowering oftbl[i, j, by]to a Polars lazy plan with data.table’s semantics.lazy.rs: the Polars-style plan (Plan, its verbs, the optimizer andexplain).group.rs,join.rsandkernel.rs: kernels that beat Polars’ general plans for grouped aggregation, joins and row-wise expressions.io.rs:freadandfwrite, with the layout read bysniff.ops.rs: the data.table helpers that real code calls beside[(setkey,rbindlist,melt,dcast,shift,fifelseand the like).ffi.rs: the zero-copy interchange with the Arrow C data interface. A column reaches R and native code as an ALTREP vector, whichlibr’sframe_altrep.rsprovides.
The crate root is src/sniff.rs. It reads a delimited text file’s separator, header and column
types by data.table’s fread rules. The table engine’s reader and the checker share it, so a
literal fread("prices.csv") is typed as exactly the columns that read produces.
The library root is src/rcmd.rs, and the binary ry-rcmd (src/main.rs) is what bin/R,
bin/Rscript and bin/Rcmd in ry’s R home run. assemble creates the R home (headers,
libR.so, etc/Makeconf, share/make/shlib.mk, links to the base tree’s packages, share/,
etc/ and R CMD scripts, and the three front-end scripts); ry-rcmd assemble --base DIR is its
command line. The base tree is ry’s own or a GNU R installation. base.rs builds ry’s own
from GNU R’s source release (ry-rcmd build-base, behind just base-r): it replays GNU R’s
src/library/Makefile on ry’s interpreter, one ry-rcmd R process per step, writes the
lazy-load databases and help, and compiles grid’s and splines’ C code. runtime_home links that
tree into an R home as it does a GNU R installation.
install, shlib and config are R CMD INSTALL, R CMD SHLIB and R CMD config; run_make runs
make in parallel and retries a failed parallel build with one job.
r_command_line and rscript_command_line parse GNU R’s front-end arguments into a Session
that main.rs runs through ry_run in the home’s libR.so. src/install.R is the R-level
half of R CMD INSTALL, embedded and run on ry after the compile. Platform::of_home reads
the home’s etc/Makeconf to choose the Unix or Windows (Rtools) conventions; on Windows the
front ends are copies of ry-rcmd.exe named R.exe, Rscript.exe, Rcmd.exe and Rterm.exe,
and because Windows has no bin/ scripts, the front end itself starts the R session behind
R CMD build, check, Rprof, Rdconv, Sweave and Stangle, as GNU R’s Rcmd.exe does. The package
uses crates/ry/build.rs as its build script, for the Windows stack size and manifest.
The whole crate is one file, crash.rs: the report a fatal signal (SIGSEGV, SIGBUS or SIGABRT)
prints to standard error before the process dies by the same signal, with the ry version, the
signal, the R call stack when a runtime registered its printer (set_traceback), and the native
stack. ry and repl install it.
features
Section titled “features”A manifest-only member that nothing links. Its normal dependencies carry the features that only
dev-dependencies turn on, so that workspace-wide feature unification (.cargo/config.toml) gives
cargo build and cargo test one compiled dependency tree.