Package loading
ry load, ry test, ry document, the ry run and ry R consoles and load_all() all load a
package the same way. This page is the exact behavior.
In RStudio or another R session, source(pipe("ry load --r-entry")) defines a load_all() that
loads the same way; put the line in ~/.Rprofile. It takes path, export_all (default TRUE),
attach, full, quiet and helpers (default TRUE; FALSE leaves the testthat helpers
unsourced, as in pkgload), and reloads only when you call it, as devtools’ does. In a console,
load_all(full = TRUE) rebuilds the namespace from scratch.
A load
Section titled “A load”- A function definition,
name <- function(...) ..., becomes a promise that parses and evaluates only that definition the first time anything usesname. When another top-level statement writes the same name, asf <- memoise(f)does, every writer of it runs in order instead. - A constant of a strict file,
name <- valuein a file under strict mode whose value calls only pure functions, becomes a promise the same way, unless another statement writes the same name. Strict mode guarantees the value has no effect, so evaluating it on first use gives what a load would have computed (lazy constants). - An R6 class generator,
Name <- R6Class(...)orName <- R6::R6Class(...)in any file, becomes a promise too when its arguments are built only from literals, names, functions, andlist()orc()of them, and every package name it reads outside its methods (aninherit =class, say) is written once, earlier in collation order. Such a generator depends on nothing but its arguments, and its methods read the namespace only when they run, so building it on first use gives the same class. An argument that calls anything else, as inpublic = list(at = Sys.time()), keeps the statement eager. - Every other top-level statement runs at load time, in collation order. The code files are
those of
R/and then those ofR/unix/(R/windows/on Windows), as R CMD INSTALL finds them: names that start with a letter or digit and end in.R,.r,.S,.sor.q. Collation followsCollate.unix(Collate.windows) whenDESCRIPTIONhas it, elseCollate, whose entries name subdirectory files asunix/a.R. Files the field leaves out follow,R/beforeR/unix/and each in C-locale order; R CMD INSTALL would refuse such a package, butload_all()appends them too. NAMESPACEworks as in an installed package: imports,S3method()registrations, and exports. A declared method whose function the package does not define refuses the load, as R does. Withexport_all = TRUEevery object in the namespace is attached aspackage:<name>, as withload_all().- Datasets in
data/are read into the namespace’s lazy data the waydata()reads them:.rdaand.RDatafiles withload(),.Rfiles withsys.source(chdir = TRUE), and tables withread.table(header = TRUE, as.is = FALSE), which splits.csvfiles on;and.taband.txtfiles on whitespace.pkg::namefinds a dataset there when no export has that name, and the attachedpackage:<name>holds a copy of each, which wins over an export of the same name. Unlike an installed package, every dataset is read at load time whateverLazyDatasays, as withload_all(). ifinNAMESPACEworks as in GNU R: at every load, the condition is evaluated in the global environment, and the directives on the branch it takes apply, soif (.Platform$OS.type == "windows") export(f)exportsfonly on Windows. This holds inside a top-level{ }block too. A condition that fails stops the load with its error.ry checkcannot evaluate R, so it reads the directives of both branches: an import on either one resolves the names it brings in.Dependspackages are attached,R/sysdata.rdais loaded,src/is compiled and loaded, and.onLoadand.onAttachrun. On ry’s runtime,src/is compiled by ry itself, against ry’s own headers andlibR, with ry’s ownR CMDfront end; GNU R and pkgbuild are not involved. The R home this needs is created once per build of ry in ry’s cache directory (~/.cache/ry/homeon Linux). On GNU R,src/is compiled with pkgbuild, asload_all()does.system.file()finds a source package’s files the wayload_all()maps them: a path underinst/is found as if the package were installed, sosystem.file("extdata", "a.txt", package = "mypkg")returnsinst/extdata/a.txt, and other paths are looked up in the package root. Code of a package loaded from source,.onLoadincluded, and code run from the console or a script see this mapping. Code inside an installed package calls base R’ssystem.file(), which does not know the source package.- testthat is attached when the package uses it, as
load_all()attaches it: the package has atests/testthatorinst/testsdirectory and testthat is installed. It is attached after the code and before.onLoad, so it sits on the search path right behindpackage:<name>, and test code or console code that callstest_that()orexpect_equal()withouttestthat::works. - testthat helpers, the
helper*.Rfiles oftests/testthat(or ofinst/testswhen there is notests/testthat), are sourced into the attachedpackage:<name>environment after the package is attached, when the package uses testthat andexport_all = TRUE, the default. They run withNOT_CRAN=true, asload_all()runs them. Tests and the console can call them, but they are not part of the namespace, sopkg:::helperdoes not find them. A reload sources them again when it changed any code, or when a helper was added, changed or deleted, so a helper’s value computed from the package’s functions stays current. A helper that fails fails the load, and the next load sources it again. - A file in latin1 loads when
DESCRIPTIONsaysEncoding: latin1(orISO-8859-1): its text is read as latin1, and its strings arrive in R as UTF-8. Without that field, a file that is not valid UTF-8 refuses the load with a message naming the field. - A syntax error stops the load before anything runs, and names the file and line. A statement that fails at load time is reported with its file and line too.
- With strict mode on, in
ry.tomlor in a file’s# typing: strictcomment,ry loadchecks the package’sR/files. If any has a strict-mode error, it prints the errors and exits with status 1 before anything with an effect runs. On GNU R the check finishes before the load starts. On ry’s runtime it runs while the runtime starts and the package’s definitions bind. Imported packages may load in that time, but the load waits for the check before it runs a top-level statement whose arguments could have an effect,data/*.R, compiled code,.onLoad, or your-eand-fcode. TheR/files are checked together, so a name one file defines resolves in the others, and test files are not part of the check. Every later load checks too:load_all(), a reload, and the load a console makes at start, on either runtime. When the check refuses one of them, the load raises<package> was not loaded, because strict mode refused itbefore anything runs, so the session keeps the code it had. A load that changes nothing skips the check. The first check in a process sharesry check’s record in.ry_cache/, so a load afterry check, the editor or an earlier load reuses the findings of every file that has not changed and checks only the rest. The first reload builds an analysis that the process keeps, and every reload after it redoes only what the edit affects.
S4 definitions
Section titled “S4 definitions”On ry’s runtime, a fresh load holds back a package’s S4 definitions and runs them the first time
anything needs one, so a session that never touches its classes never pays for registering them. The
package behaves as if they had run in place: each name they bind (a class generator, a generic, the
.__C__ and .__T__ metadata) is a promise that runs them, and so is looking up one of their classes
or generics by name, as new(), is(), getClass() and method dispatch do. When a statement needs
one during the load itself, the held-back definitions placed before that statement run then, and the
rest run in their own places.
After the load, the definitions run in units. The first unit holds the classes, unions, validity
methods, coercions and methods for other packages’ generics, and runs as a whole. Each of the
package’s own generics whose methods name only classes defined before them, and whose names no other
definition reads, is a unit of its own: its setGeneric() and all its methods run the first time the
generic is called or looked up, after the first unit. A session that creates objects of the package’s
classes but calls only a few of its generics registers only those generics’ methods.
A load holds them back only when all of these hold, and otherwise runs every one of them in place:
- Every S4 registration in the package is
setClass()(alone or asGen <- setClass(...)),setGeneric()with a function literal as its definition,setMethod()orsetReplaceMethod()with a literal signature,setValidity(), orsetClassUnion()with literal members. Any other one, such assetRefClass(), keeps them all in place. - Every method is for a generic the package itself defines, or its signature names one of the
package’s own classes, as
setMethod("show", "Quantity", ...)does. A method of the second kind may be for a generic function such asshoworinitialize, for a primitive such aslengthor[, or for a function that is not yet generic, such asas.list, which the method turns into a generic in the namespace. The held-back definitions then also run the first time that generic is looked up or dispatches, the first time the primitive dispatches on an S4 object, or the first time code reads the function’s name in the namespace. A method for a group generic such asArithkeeps them in place, because its members dispatch to it without a lookup ry can see. Every validity method must be for one of the package’s own classes. - A coercion,
setAs("Quantity", "character", ...)orsetAs("numeric", "Quantity", ...), is a method for thecoercegeneric (and forcoerce<-withreplace =), so it is held back like the methods above when it names one of the package’s classes, andas()runs the held-back definitions when it looks for it. - A class union is held back too, even when it names other packages’ classes, as
setClassUnion("QuantityOrNull", c("Quantity", "NULL"))andsetClassUnion("NumOrNull", c("numeric", "NULL"))do. Asking whetherNULLbelongs to the union, withis(NULL, "NumOrNull"),extends()or a method for the union, names the union and so runs the held-back definitions. A method for such a union on another package’s generic, such asshow, keeps every definition in place, because dispatch onNULLwould not look for it. - A coercion that names only other packages’ classes, such as
setAs("numeric", "character", ...), is not held back but runs in place. - Every name the definitions read is written once, by a statement placed before them, so running them later reads the same values.
- Every name the definitions bind is written by them alone.
- No generic they define already exists as a function the namespace can see.
The load reads each S4 call’s arguments as R matches them, by exact name, then by unique partial
name, then by position, so setAs(from = "numeric", to = "Quantity", function(from) ...) and
setClassUnion("NumOrNull", mem = c("numeric", "NULL")) are recognized like their positional forms.
options(ry.load.lazy_s4 = FALSE) turns this off. The load’s summary counts the definitions it held
back, as in Loaded shapes: 1 definition bound lazily, 1 top-level statement run, 6 S4 definitions held back until first use, in 5 ms. A reload that changes one of them, or a name they bind, runs them all
first.
Source references
Section titled “Source references”Functions keep their source references on both runtimes, so print() shows them as written and
getSrcLocation(), tracebacks and the debugger point into your files, as long as
options(keep.source = TRUE) is set, which is the default in an interactive session. On ry’s runtime
a lazily bound definition gets its source reference when it is first used, and a file’s srcfile is
built only then. Columns count characters and byte offsets count bytes on both runtimes, so in
m <- function() "é"; n <- function() 3 the definition of n starts at column 27 and byte 28.
The load-index cache
Section titled “The load-index cache”A load reads its analysis of the package from a cache when an earlier process analyzed exactly the
same sources, so a cold start parses nothing until code runs. The cache lives in
$XDG_CACHE_HOME/ry/load-index (~/.cache/ry/load-index by default), holds one entry per package
root, and is safe to delete. An entry is used only for the same DESCRIPTION, NAMESPACE and R/
texts, the same strict setting, and the same build of ry.
A reload
Section titled “A reload”A reload works out which top-level items changed and touches only those:
- A changed function is rebound, a new one is bound, and a deleted one is removed. A function that only moved, because you edited something above it, is rebound to keep its source reference right.
- A top-level statement runs again when you changed it, or when it read something that changed while
it ran, including anything the functions it called use.
cached <- helper()reruns after you edithelper, or a functionhelpercalls, while a function that callshelper()does not need to, because it lookshelperup when it runs. - A lazy constant is bound again when you changed it, or when anything its value reaches changed, such as the body of a pure function it calls, so its next use computes the new value.
- When two statements write the same name, the last one in collation order wins again, and a
statement like
f <- memoise(f)seesfas it was at load time rather than wrapping it twice. - A
NAMESPACEchange rebuilds the imports or resyncs the exports,.onLoadruns again when you changed it, and a deleted S4 method or class is removed. - When a dataset file was added, changed or deleted, judged by the names, sizes and modification
times of the files in
data/, every dataset is read again, and a deleted one is gone.R/sysdata.rdais loaded again when its content changed.
A console reloads before a command when DESCRIPTION, NAMESPACE, or anything under R/,
data/ or src/ changed since the last load, and an edit under src/ rebuilds the compiled code
first. It learns of edits from the operating system’s file notifications, so a command costs
nothing when nothing changed. Where notifications are unavailable, it compares the names, sizes
and modification times of the code files, R/sysdata.rda and the datasets instead, which misses
src/. A package whose load failed, or was refused by strict mode, is watched by those stamps, so
fixing the file loads it before the next command.
When a package another development package imports reloads, the importer’s imports are rebuilt
too, so it calls the new definitions without reloading itself. If the importer’s NAMESPACE no
longer fits, because it imports a name the reload removed, the reload still succeeds, and the
importer keeps its old imports with a warning until you reload it.
A load parses a function only when something first calls it, and a reload touches only the
definitions you changed. On a generated package of about 300,000 lines in 500 files, against
devtools::load_all() and loadfast:
load_all() |
load_fast() |
ry R |
ry load |
|
|---|---|---|---|---|
| cold load | 9.6 s | 4.0 s | 1.3 s | 1.7 s |
| reload after a body edit | 8.6 s | 125 ms | 79 ms | 83 ms |
These are medians of several runs of tools/loader-bench on a shared machine, with ry built without
link-time optimization, so the ratios are more reliable than the milliseconds.
Timing a load
Section titled “Timing a load”options(ry.load.verbose = TRUE) prints how long each phase of a load took: the plan, the imports,
the top-level statements, the S4 caches and so on. RY_LOAD_VERBOSE=true in the environment sets
the option’s default, which reaches the load ry load makes before any of your code runs. The first
load of a process also prints process start, the time the process took to reach the load.
On ry’s runtime, setClass(), setClassUnion(), setGeneric() and setMethod() run natively
when the result is identical to what the methods package’s R code builds, and that R code runs
otherwise. With the option on, each definition that takes the R code says which one and why:
ry: setClass("Num") runs the methods package's R code: a superclass is a basic class or comes from another package (methods_core.rs:2682)A fresh load whose S4 definitions run in place, because one of the conditions above fails, says so with the first definition that keeps them there:
S4 definitions run at load: R/a.R:5: a method for another package's generic `show` whose signature names none of the package's classesThe R code gives the same classes and methods, only more slowly (a few milliseconds per definition against tens of microseconds), so the report is a guide to load time, not a warning.
On ry’s runtime, process start and the imports run GNU R’s own R code: base, methods and the
default packages at start, then each imported namespace. A warm load spends most of its time
there. The runtime registers each namespace’s S3 methods natively, in place of the loops in GNU R
4.3.3’s registerS3methods(), and keeps the installed function wherever it differs from that one.
The same step gives a few package functions a native implementation, such as rlang’s
trace_back() and the helpers behind withr’s local_envvar() and local_options(). Each one applies only while the function’s formals and body hash to the release it
was written from, so another release of the package runs its own R code.
Limits
Section titled “Limits”- Until held-back S4 definitions run, listing everything another package’s class extends, as
extends("NULL"),extends("numeric")oris(NULL)do, leaves out a held-back union that names that class. Asking about the union itself is exact, and so is the listing once the definitions have run. - Compressed dataset tables (
.csv.gz,.tab.bz2,.txt.xzand so on) are not read. When one dataset name has files in several formats, such asx.Randx.rda, every file is read in name order, wheredata()reads only one. - A reload that leaves a declared S3 method without its function is refused before it changes
anything, unless it reruns a top-level statement that may bind a name it computes, such as
assign(paste0("print.", class), ...), which could define the function. Then the refusal comes after the statements ran, and the namespace holds the new code. In a file that is not strict, every statement other than a function definition counts as such a statement. - Rebuilding compiled code unloads the old library first, as
load_all()does, so an object that still points into it must not be used afterwards. On ry’s runtime the old library stays mapped, and the rebuilt one loads from a fresh copy of the file, because the system loader would otherwise hand back the mapping it already has. - A registered routine whose R name (after
.fixes) is already defined in the namespace is not bound, and the load warns, as GNU R’sloadNamespacedoes. - A top-level statement that reads a name before the statement defining it runs sees, on a reload, the value from the previous load. A fresh load would fail on such code anyway.