Big Picture
The success of the R language can at least in part be attributed to the many thousands of packages that extend the functionality of base R. There is a mature system of ‘Depends/Imports’ for hard dependencies and ‘Suggests/Enhances’ for soft or conditional dependencies. In this way, the current ecosystem of interlinked packages has developed over time.
However, this has only been the case for R packages importing or calling exported R functions from other packages.
Many packages provide interfaces at a lower level via the R_RegisterCCallable / R_GetCCallable mechanism, and these are almost always more performant where using them is possible (just avoiding re-entry back into R itself can be significant for fairly atomic operations). In other cases, it may only make sense to interface at the C level.
If this is the case, then there is only the ‘LinkingTo’ approach described in ‘Writing R Extensions’ 5.4.3 Linking to native routines in other packages, which also requires the use of ‘Imports’ to load the dependent package for R_GetCCallable to work. In other words, there is seemingly only the option to add a hard dependency.
In many cases this is simply not feasible. Taking for example code that provides a binding to a C library - this can be expected to have many uses in any number of contexts. Even if there is a compelling use case in any one such context, it would rarely make sense for this (perhaps zero-dependency) package to add a dependency, as this encumbers all downstream users of the package.
The unfortunate result of this state of affairs is that many such packages remain siloed. Sometimes package authors attempt to ‘vendor’ or copy across the source code of another to avoid a dependency.
The Specific Case
mirai is a minimalist asynchronous evaluation framework for R. It is an async launcher of Shiny ExtendedTasks (the hot new feature of 2024), highlighted by Joe Cheng (CTO, Posit and creator of Shiny) himself.
As part of its integration with Shiny, mirai goes one step further - as a collaborative effort, it implements next-generation promises that are completely event-driven and non-polling - the first of its kind anywhere in R.
This is only available through mirai as it is powered by a tight integration of nanonext’s own concurrency framework with the later event loop at the C level. This results in zero-latency promise resolution, enabling more responsive, and also massively-scalable Shiny apps. It allows crazy possibilities, such as firing off hundreds of thousands of mirai promises.
The Shiny use case was compelling, hence leading to its adoption in the first place. However, nanonext/mirai is also used in many other scientific contexts in industry, where it acts as the High Performance Computing back-end for targets through crew. nanonext is so incredibly lightweight that adding later as a dependency doubled its load time, which in turn affected the performance of short-lived mirai processes.
A Solution
nanonext linked to later in the usual way in its 1.0.0 release. However as of v1.1.1, ‘conditional LinkingTo’ has been achieved, with later only appearing as a ‘suggests’ dependency of nanonext.
The method employed by nanonext consists only of the following steps:
- Define a C function pointer with the correct signature, along with a dummy function ->
- Assign the function pointer to the dummy function during package initialization ->
- At the point where the function is called, check if the function pointer points to the dummy function ->
- If so, construct and evaluate a call to load the dependent package ->
- Then assign the function pointer to the real function obtained by
R_GetCCallable->
The later package / shared object is hence only loaded if it is actually used. This has the happy consequence of reducing the load time of nanonext by up to half. Furthermore, later (with its dependencies) is now not required at all for the installation of nanonext itself, which remains a zero-dependency package.
Concluding Remarks
The above presents a viable method for constructing ‘conditional LinkingTo’.
It should be noted that for packages to pass ‘R CMD check’, care should be taken to write tests conditional upon the presence of the soft dependency. Also, where relevant, the failure of step 4 (for example if the package is not installed) should also be considered and handled.
We hope that this sparks some new ideas for package authors, opening up avenues for innovation and collaboration, with positive external benefits to the R ecosystem as a whole.