# Package loading

`ry load`, `ry test`, `ry document`, the `ry run` and `ry R` consoles and `load_all()` all load a
package the same way. This page is the exact behavior.

In RStudio or another R session, `source(pipe("ry load --r-entry"))` defines a `load_all()` that
loads the same way; put the line in `~/.Rprofile`. It takes `path`, `export_all` (default `TRUE`),
`attach`, `full`, `quiet` and `helpers` (default `TRUE`; `FALSE` leaves the testthat helpers
unsourced, as in pkgload), and reloads only when you call it, as devtools' does. In a console,
`load_all(full = TRUE)` rebuilds the namespace from scratch.

## A load

- **A function definition**, `name <- function(...) ...`, becomes a promise that parses and
  evaluates only that definition the first time anything uses `name`. When another top-level
  statement writes the same name, as `f <- memoise(f)` does, every writer of it runs in order
  instead.
- **A constant of a strict file**, `name <- value` in a file under [strict mode](/guide.md#strict-mode)
  whose value calls only pure functions, becomes a promise the same way, unless another statement
  writes the same name. Strict mode guarantees the value has no effect, so evaluating it on first use
  gives what a load would have computed
  ([lazy constants](/reference/type-system.md#lazy-constants)).
- **An R6 class generator**, `Name <- R6Class(...)` or `Name <- R6::R6Class(...)` in any file,
  becomes a promise too when its arguments are built only from literals, names, functions, and
  `list()` or `c()` of them, and every package name it reads outside its methods (an `inherit =`
  class, say) is written once, earlier in collation order. Such a generator depends on nothing but
  its arguments, and its methods read the namespace only when they run, so building it on first
  use gives the same class. An argument that calls anything else, as in
  `public = list(at = Sys.time())`, keeps the statement eager.
- **Every other top-level statement** runs at load time, in collation order. The code files are
  those of `R/` and then those of `R/unix/` (`R/windows/` on Windows), as R CMD INSTALL finds them:
  names that start with a letter or digit and end in `.R`, `.r`, `.S`, `.s` or `.q`. Collation
  follows `Collate.unix` (`Collate.windows`) when `DESCRIPTION` has it, else `Collate`, whose entries
  name subdirectory files as `unix/a.R`. Files the field leaves out follow, `R/` before `R/unix/` and
  each in C-locale order; R CMD INSTALL would refuse such a package, but `load_all()` appends them
  too.
- **`NAMESPACE`** works as in an installed package: imports, `S3method()` registrations, and exports.
  A declared method whose function the package does not define refuses the load, as R does.
  With `export_all = TRUE` every object in the namespace is attached as `package:<name>`, as with
  `load_all()`.
- **Datasets in `data/`** are read into the namespace's lazy data the way `data()` reads them:
  `.rda` and `.RData` files with `load()`, `.R` files with `sys.source(chdir = TRUE)`, and tables
  with `read.table(header = TRUE, as.is = FALSE)`, which splits `.csv` files on `;` and `.tab` and
  `.txt` files on whitespace. `pkg::name` finds a dataset there when no export has that name, and
  the attached `package:<name>` holds a copy of each, which wins over an export of the same name.
  Unlike an installed package, every dataset is read at load time whatever `LazyData` says, as with
  `load_all()`.
- **`if` in `NAMESPACE`** works as in GNU R: at every load, the condition is evaluated in the
  global environment, and the directives on the branch it takes apply, so
  `if (.Platform$OS.type == "windows") export(f)` exports `f` only on Windows. This holds inside
  a top-level `{ }` block too. A condition that fails stops the load with its error. `ry check` cannot evaluate R, so it reads the directives
  of both branches: an import on either one resolves the names it brings in.
- `Depends` packages are attached, `R/sysdata.rda` is loaded, `src/` is compiled and loaded,
  and `.onLoad` and `.onAttach` run. On ry's runtime, `src/` is compiled by ry itself, against
  ry's own headers and `libR`, with ry's own `R CMD` front end; GNU R and pkgbuild are not
  involved. The R home this needs is created once per build of ry in ry's cache
  directory (`~/.cache/ry/home` on Linux). On GNU R, `src/` is compiled with
  [pkgbuild](https://pkgbuild.r-lib.org), as `load_all()` does.
- **`system.file()`** finds a source package's files the way `load_all()` maps them: a path under
  `inst/` is found as if the package were installed, so `system.file("extdata", "a.txt", package =
  "mypkg")` returns `inst/extdata/a.txt`, and other paths are looked up in the package root. Code
  of a package loaded from source, `.onLoad` included, and code run from the console or a script
  see this mapping. Code inside an installed package calls base R's `system.file()`, which does
  not know the source package.
- **testthat** is attached when the package uses it, as `load_all()` attaches it: the package has a
  `tests/testthat` or `inst/tests` directory and testthat is installed. It is attached after the
  code and before `.onLoad`, so it sits on the search path right behind `package:<name>`, and test
  code or console code that calls `test_that()` or `expect_equal()` without `testthat::` works.
- **testthat helpers**, the `helper*.R` files of `tests/testthat` (or of `inst/tests` when there is
  no `tests/testthat`), are sourced into the attached `package:<name>` environment after the
  package is attached, when the package uses testthat and `export_all = TRUE`, the default. They run
  with `NOT_CRAN=true`, as `load_all()` runs them. Tests and the console can call them, but they
  are not part of the namespace, so `pkg:::helper` does not find them. A reload
  sources them again when it changed any code, or when a helper was added, changed or deleted, so a
  helper's value computed from the package's functions stays current. A helper that fails fails
  the load, and the next load sources it again.
- **A file in latin1** loads when `DESCRIPTION` says `Encoding: latin1` (or `ISO-8859-1`): its text
  is read as latin1, and its strings arrive in R as UTF-8. Without that field, a file that is not
  valid UTF-8 refuses the load with a message naming the field.
- **A syntax error stops the load before anything runs**, and names the file and line. A statement
  that fails at load time is reported with its file and line too.
- **With [strict mode](/guide.md#strict-mode) on**, in `ry.toml` or in a file's
  `# typing: strict` comment, `ry load` checks the package's `R/` files. If any has a strict-mode
  error, it prints the errors and exits with status 1 before anything with an effect runs. On GNU
  R the check finishes before the load starts. On ry's runtime it runs while the runtime starts
  and the package's definitions bind. Imported packages may load in that time, but the load waits
  for the check before it runs a top-level statement whose arguments could have an effect,
  `data/*.R`, compiled code, `.onLoad`, or your `-e` and `-f` code. The
  `R/` files are checked together, so a name one file defines resolves in the others, and test
  files are not part of the check.
  Every later load checks too: `load_all()`, a reload, and the load a console makes at start, on
  either runtime. When the check refuses one of them, the load raises
  `<package> was not loaded, because strict mode refused it` before anything runs, so the session
  keeps the code it had. A load that changes nothing skips the check.
  The first check in a process shares [`ry check`'s record](/reference/cli.md#the-check-cache) in
  `.ry_cache/`, so a load after `ry check`, the editor or an earlier load reuses the findings of
  every file that has not changed and checks only the rest. The first reload builds an analysis
  that the process keeps, and every reload after it redoes only what the edit affects.

### S4 definitions

On ry's runtime, a fresh load holds back a package's S4 definitions and runs them the first time
anything needs one, so a session that never touches its classes never pays for registering them. The
package behaves as if they had run in place: each name they bind (a class generator, a generic, the
`.__C__` and `.__T__` metadata) is a promise that runs them, and so is looking up one of their classes
or generics by name, as `new()`, `is()`, `getClass()` and method dispatch do. When a statement needs
one during the load itself, the held-back definitions placed before that statement run then, and the
rest run in their own places.

After the load, the definitions run in units. The first unit holds the classes, unions, validity
methods, coercions and methods for other packages' generics, and runs as a whole. Each of the
package's own generics whose methods name only classes defined before them, and whose names no other
definition reads, is a unit of its own: its `setGeneric()` and all its methods run the first time the
generic is called or looked up, after the first unit. A session that creates objects of the package's
classes but calls only a few of its generics registers only those generics' methods.

A load holds them back only when all of these hold, and otherwise runs every one of them in place:

- Every S4 registration in the package is `setClass()` (alone or as `Gen <- setClass(...)`),
  `setGeneric()` with a function literal as its definition, `setMethod()` or `setReplaceMethod()` with
  a literal signature, `setValidity()`, or `setClassUnion()` with literal members. Any other one, such
  as `setRefClass()`, keeps them all in place.
- Every method is for a generic the package itself defines, or its signature names one of the
  package's own classes, as `setMethod("show", "Quantity", ...)` does. A method of the second kind
  may be for a generic function such as `show` or `initialize`, for a primitive such as `length` or
  `[`, or for a function that is not yet generic, such as `as.list`, which the method turns into a
  generic in the namespace. The held-back definitions then also run the first time that generic is looked up or
  dispatches, the first time the primitive dispatches on an S4 object, or the first time code reads
  the function's name in the namespace. A method for a group generic such as `Arith` keeps them in
  place, because its members dispatch to it without a lookup ry can see. Every validity method must
  be for one of the package's own classes.
- A coercion, `setAs("Quantity", "character", ...)` or `setAs("numeric", "Quantity", ...)`, is a
  method for the `coerce` generic (and for `coerce<-` with `replace =`), so it is held back like the
  methods above when it names one of the package's classes, and `as()` runs the held-back
  definitions when it looks for it.
- A class union is held back too, even when it names other packages' classes, as
  `setClassUnion("QuantityOrNull", c("Quantity", "NULL"))` and
  `setClassUnion("NumOrNull", c("numeric", "NULL"))` do. Asking whether `NULL` belongs to the
  union, with `is(NULL, "NumOrNull")`, `extends()` or a method for the union, names the union and so
  runs the held-back definitions. A method for such a union on another package's generic, such as
  `show`, keeps every definition in place, because dispatch on `NULL` would not look for it.
- A coercion that names only other packages' classes, such as `setAs("numeric", "character", ...)`,
  is not held back but runs in place.
- Every name the definitions read is written once, by a statement placed before them, so running them
  later reads the same values.
- Every name the definitions bind is written by them alone.
- No generic they define already exists as a function the namespace can see.

The load reads each S4 call's arguments as R matches them, by exact name, then by unique partial
name, then by position, so `setAs(from = "numeric", to = "Quantity", function(from) ...)` and
`setClassUnion("NumOrNull", mem = c("numeric", "NULL"))` are recognized like their positional forms.

`options(ry.load.lazy_s4 = FALSE)` turns this off. The load's summary counts the definitions it held
back, as in `Loaded shapes: 1 definition bound lazily, 1 top-level statement run, 6 S4 definitions held
back until first use, in 5 ms.` A reload that changes one of them, or a name they bind, runs them all
first.

### Source references

Functions keep their source references on both runtimes, so `print()` shows them as written and
`getSrcLocation()`, tracebacks and the debugger point into your files, as long as
`options(keep.source = TRUE)` is set, which is the default in an interactive session. On ry's runtime
a lazily bound definition gets its source reference when it is first used, and a file's `srcfile` is
built only then. Columns count characters and byte offsets count bytes on both runtimes, so in
`m <- function() "é"; n <- function() 3` the definition of `n` starts at column 27 and byte 28.

## The load-index cache

A load reads its analysis of the package from a cache when an earlier process analyzed exactly the
same sources, so a cold start parses nothing until code runs. The cache lives in
`$XDG_CACHE_HOME/ry/load-index` (`~/.cache/ry/load-index` by default), holds one entry per package
root, and is safe to delete. An entry is used only for the same `DESCRIPTION`, `NAMESPACE` and `R/`
texts, the same strict setting, and the same build of ry.

## A reload

A reload works out which top-level items changed and touches only those:

- A changed function is rebound, a new one is bound, and a deleted one is removed. A function that
  only moved, because you edited something above it, is rebound to keep its source reference right.
- A top-level statement runs again when you changed it, or when it read something that changed while
  it ran, including anything the functions it called use. `cached <- helper()` reruns after you edit
  `helper`, or a function `helper` calls, while a function that calls `helper()` does not need to,
  because it looks `helper` up when it runs.
- A lazy constant is bound again when you changed it, or when anything its value reaches changed,
  such as the body of a pure function it calls, so its next use computes the new value.
- When two statements write the same name, the last one in collation order wins again, and a
  statement like `f <- memoise(f)` sees `f` as it was at load time rather than wrapping it twice.
- A `NAMESPACE` change rebuilds the imports or resyncs the exports, `.onLoad` runs again when you
  changed it, and a deleted S4 method or class is removed.
- When a dataset file was added, changed or deleted, judged by the names, sizes and modification
  times of the files in `data/`, every dataset is read again, and a deleted one is gone.
  `R/sysdata.rda` is loaded again when its content changed.

A console reloads before a command when `DESCRIPTION`, `NAMESPACE`, or anything under `R/`,
`data/` or `src/` changed since the last load, and an edit under `src/` rebuilds the compiled code
first. It learns of edits from the operating system's file notifications, so a command costs
nothing when nothing changed. Where notifications are unavailable, it compares the names, sizes
and modification times of the code files, `R/sysdata.rda` and the datasets instead, which misses
`src/`. A package whose load failed, or was refused by strict mode, is watched by those stamps, so
fixing the file loads it before the next command.

When a package another development package imports reloads, the importer's imports are rebuilt
too, so it calls the new definitions without reloading itself. If the importer's `NAMESPACE` no
longer fits, because it imports a name the reload removed, the reload still succeeds, and the
importer keeps its old imports with a warning until you reload it.

## Speed

A load parses a function only when something first calls it, and a reload touches only the
definitions you changed. On a generated package of about 300,000 lines in 500 files, against
`devtools::load_all()` and [loadfast](https://github.com/finccam/loadfast):

| | `load_all()` | `load_fast()` | `ry R` | `ry load` |
| --- | --- | --- | --- | --- |
| cold load | 9.6 s | 4.0 s | 1.3 s | 1.7 s |
| reload after a body edit | 8.6 s | 125 ms | 79 ms | 83 ms |

These are medians of several runs of `tools/loader-bench` on a shared machine, with ry built without
link-time optimization, so the ratios are more reliable than the milliseconds.

## Timing a load

`options(ry.load.verbose = TRUE)` prints how long each phase of a load took: the plan, the imports,
the top-level statements, the S4 caches and so on. `RY_LOAD_VERBOSE=true` in the environment sets
the option's default, which reaches the load `ry load` makes before any of your code runs. The first
load of a process also prints `process start`, the time the process took to reach the load.

On ry's runtime, `setClass()`, `setClassUnion()`, `setGeneric()` and `setMethod()` run natively
when the result is identical to what the methods package's R code builds, and that R code runs
otherwise. With the option on, each definition that takes the R code says which one and why:

```
ry: setClass("Num") runs the methods package's R code: a superclass is a basic class or comes from another package (methods_core.rs:2682)
```

A fresh load whose S4 definitions run in place, because one of the conditions above fails, says so
with the first definition that keeps them there:

```
  S4 definitions run at load: R/a.R:5: a method for another package's generic `show` whose signature names none of the package's classes
```

The R code gives the same classes and methods, only more slowly (a few milliseconds per
definition against tens of microseconds), so the report is a guide to load time, not a warning.

On ry's runtime, process start and the imports run GNU R's own R code: base, `methods` and the
default packages at start, then each imported namespace. A warm load spends most of its time
there. The runtime registers each namespace's S3 methods natively, in place of the loops in GNU R
4.3.3's `registerS3methods()`, and keeps the installed function wherever it differs from that one.
The same step gives a few package functions a native implementation, such as rlang's
`trace_back()` and the helpers behind withr's `local_envvar()` and `local_options()`. Each one applies only while the function's formals and body hash to the release it
was written from, so another release of the package runs its own R code.

## Limits

- Until held-back S4 definitions run, listing everything another package's class extends, as
  `extends("NULL")`, `extends("numeric")` or `is(NULL)` do, leaves out a held-back union that names
  that class. Asking about the union itself is exact, and so is the listing once the definitions
  have run.
- Compressed dataset tables (`.csv.gz`, `.tab.bz2`, `.txt.xz` and so on) are not read. When one
  dataset name has files in several formats, such as `x.R` and `x.rda`, every file is read in name
  order, where `data()` reads only one.
- A reload that leaves a declared S3 method without its function is refused before it changes
  anything, unless it reruns a top-level statement that may bind a name it computes, such as
  `assign(paste0("print.", class), ...)`, which could define the function. Then the refusal comes
  after the statements ran, and the namespace holds the new code. In a file that is not strict,
  every statement other than a function definition counts as such a statement.
- Rebuilding compiled code unloads the old library first, as `load_all()` does, so an object that
  still points into it must not be used afterwards. On ry's runtime the old library stays mapped, and
  the rebuilt one loads from a fresh copy of the file, because the system loader would otherwise
  hand back the mapping it already has.
- A registered routine whose R name (after `.fixes`) is already defined in the namespace is not
  bound, and the load warns, as GNU R's `loadNamespace` does.
- A top-level statement that reads a name before the statement defining it runs sees, on a reload,
  the value from the previous load. A fresh load would fail on such code anyway.
