Skip to main content

R interop

The process at the other end of a channel does not have to be Python. mizu is pymizu’s R counterpart, built on the same shared-memory core, and a channel or pool connects the two languages as directly as two Python processes.

R channel peers

A channel peer can be an R process with the mizu package installed: pass the peer program as R source and set the launcher to pymizu.r_launcher(). Here, an R process doubles the array you send:

import numpy as np
import pymizu

ch = pymizu.Channel.create(
    """
library(mizu)
repeat {
  x <- mizu_recv(ch, timeout = 30)
  if (inherits(x, "mizu_sentinel")) break
  mizu_send(ch, x * 2)
}
""",
    launcher=pymizu.r_launcher(),
)

ch.send(np.array([1.5, 2.5, 3.5]))
print(ch.recv(timeout=5))            # [3. 5. 7.]
ch.close()

r_launcher() needs R and the mizu package installed; if either is missing it raises MizuError before the channel exists. Pass rscript= to use a specific R installation.

The reverse direction works too — an R session spawns a Python peer with mizu::mizu_py_launcher().

Large data crosses without copying

Above a size threshold, pymizu writes the data to shared memory once and both sides read it in place — nothing is serialized, parsed, or copied again. R reads a large value you send directly over the shared pages; a large value you receive arrives as a view over those pages, usually wrapped as a numpy array. The peer below echoes back whatever it receives; this channel stays open for the next few examples:

ch = pymizu.Channel.create(
    """
library(mizu)
repeat {
  x <- mizu_recv(ch, timeout = 30)
  if (inherits(x, "mizu_sentinel")) break
  mizu_send(ch, x)
}
""",
    launcher=pymizu.r_launcher(),
)

big = np.random.default_rng(0).normal(size=1_250_000)  # ten megabytes
ch.send(big)
back = ch.recv(timeout=5)
np.array_equal(back, big)                              # True

Apart from the one write into the shared region, no data moved: R’s echo came back as a reference, with no payload bytes. Send a received view onward and it crosses the same way, as the region’s name alone. Zero-copy covers every large layout: a flat array of an exact-match dtype (float64, int32, int64, complex128, uint8), an array of any shape and memory order, a nested tree of lists, dicts, arrays, and strings, a datetime64 array (NaT crosses as NA), or a list[str | None] (None crosses as NA). A large data frame crosses as one shared region in both directions. The floor is 256 KB for a flat contiguous array on a channel, 32 KB everywhere else.1

Arrays of other numeric dtypes (booleans, the narrow ints, uint64, float32, …) cross as a converted copy at any size; the dtype matrix has the per-dtype rule. Without numpy, a received view is a plain buffer: memoryview(view) works, and pa.array(view) or pl.from_arrow(view) wrap the same pages without copying. One exception: R’s ALTREP vectors (such as 1:1e9, stored as a recipe rather than data) cross by value, values exact. A polars or pyarrow object that wraps a received frame keeps the frame’s region alive until garbage collection, so delete these wrappers when you no longer need them.

What crosses

Small values round-trip exactly too. Each form below comes back from R unchanged:

def echo(x):
    ch.send(x)
    return ch.recv(timeout=5)

x = np.array([1.5, 2.5, 3.5])
np.array_equal(echo(x), x)     # a float64 vector: True

m = np.arange(6, dtype=np.int32).reshape(2, 3)
np.array_equal(echo(m), m)     # an int32 matrix, dim included: True

echo("hello") == "hello"       # a string: True

d = np.array(["2024-01-01", "2024-01-03"], dtype="datetime64[D]")
np.array_equal(echo(d), d)     # dates keep their meaning: True

A data frame crosses as well, with its column types intact. Here a polars frame with a categorical column; pandas and pyarrow frames work the same way:

import polars as pl
df = pl.DataFrame({
    "total_bill": [16.99, 10.34, 21.01],
    "day": pl.Series(["Sun", "Sun", "Mon"], dtype=pl.Categorical),
})
ch.send(df)        # arrives in R as a data.frame, the categorical column as a factor
back = ch.recv(timeout=5)
back.names         # ('total_bill', 'day')

R sees the natural equivalent throughout: a vector for an array, a factor for a category column, a named list for a dict, a Date or POSIXct for a datetime64 array. An Arrow array arrives as an R vector, with Arrow nulls as R missing values: ch.send(pa.array([1, None, 3])). An ordered categorical — an R ordered factor, a polars Enum, an ordered pyarrow dictionary — crosses with its ordering intact. Labels cross with the values they name: a named R vector arrives as a DimArray carrying its names, and labels you attach in Python cross to R as names or dimnames.

What doesn’t cross

Some values have no equivalent on the other side. pymizu declines them at send time with a DeclinedError (a subclass of TypeError) that names the offending value on its path and reason attributes. The channel stays usable:

import decimal
import pyarrow as pa

tbl = pa.table({"price": pa.array([decimal.Decimal("1.5")])})
try:
    ch.send(tbl)
except pymizu.DeclinedError as e:
    print(e)
    # pymizu: value is not portable to the peer (column 'price': an Arrow type
    # with no portable home (format 'd:2,1'))
ch.close()

From Python, also declined: Arrow nested, date64, time, and interval columns. An Arrow uint8 column with nulls is a TypeError: an R raw vector has no NA.

From R: a POSIXlt time, a character matrix, and S4 objects stay on R’s side. So does a named vector with a class or of character type (a named character vector, factor, Date, or POSIXct), since names travel only on the plain atomic vectors. Among data frames: a data.table, a groupedData-style frame (a formula attribute), and a column that is itself a labeled matrix.

The counterpart of the send-time decline is a value that arrives in a language-private format no send gate caught (an attach race, or a misbehaving or version-skewed peer). The read raises ForeignPayloadError and the message is consumed, so one bad payload never blocks the channel; under recv_async(), only that message’s waiter rejects.

pymizu.Frame

A pymizu.Frame is Python’s home for an R data.frame: named columns with a row count. frame.to_dict() returns the columns as a dict: numpy arrays when numpy is installed, memoryviews otherwise, strings and factors as list[str | None]. frame.names and frame.row_names carry the metadata.

Any Arrow consumer takes a frame in one line: pl.from_arrow(f), pa.table(f), or pd.DataFrame.from_arrow(f). (With polars >= 2.0, a streaming producer gives a struct Series; .struct.unnest() restores the columns.) Factors become dictionary columns, logicals Arrow bool, Date date32, and POSIXct timestamp[us].

A Frame pickles, so it crosses Python-to-Python channels. Sent to R, it becomes a data.frame again.

Frame metadata crosses unchanged: a data.frame’s full class vector (a tibble’s c("tbl_df", "tbl", "data.frame"), any class ending in "data.frame") and extra attributes travel as-is, stored on the Frame uninterpreted. An unmodified echo restores them in R; a frame rebuilt through pyarrow, polars, pandas, or from_arrow arrives in R as a plain data.frame.

Mixed-language pools

A pool’s workers can be R processes: pymizu.r_launcher() spawns them, and pymizu.call() builds a language-neutral task — a qualified "pkg::fn" name for R or "mod.fn" for Python, or a source= string, plus constant arguments:

import numpy as np
import pymizu

pool = pymizu.Pool.create(4, launcher=pymizu.r_launcher())
t = pool.submit(pymizu.call("stats::quantile", np.arange(101.0),
                            probs=np.array([0.25, 0.5, 0.75]), names=False))
print(t.collect())                  # [ 25.  50.  75.]
t2 = pool.submit(pymizu.call(source="y <- x * 2\ny + 1", x=20))
print(t2.collect())                 # 41
pool.stop()

Unnamed arguments become positional, keyword arguments named, matching R’s calling convention. A source= task runs in a fresh environment with keyword arguments bound as names and positional arguments as _1, _2, …; the result is the value of the last expression.

On an R-worker pool, submitting a plain Python callable raises an error naming pymizu.call(). Task errors cross as TaskError with the R error type preserved in remote_type. Arguments follow the same rules as channel payloads, zero-copy included: a large array crosses as one shared view, and an array that is already a received view crosses as its name alone. A pool’s workers all run one language — mixing R and Python workers in one pool is not supported. The reverse direction works the same way: mizu::mizu_pool() with mizu::mizu_py_launcher(), and mizu::mizu_submit_call() with mizu::mizu_call() specs.

Mapping over a foreign pool

To map an R function over Python data, pass a pymizu.call() spec as the fn of Pool.map() or Pool.map_prepare() on an R-worker pool. mizu::mizu_map() mirrors this in R:

import numpy as np

rng = np.random.default_rng(0)
x = np.arange(50.0)
y = 1 + 2 * x + rng.normal(scale=5, size=50)
folds = [
    {"x": xf, "y": yf}
    for xf, yf in zip(x.reshape(5, 10), y.reshape(5, 10))
]

pool = pymizu.Pool.create(4, launcher=pymizu.r_launcher())
pool.map(pymizu.call("stats::median"), list(y.reshape(5, 10))).collect()
pool.map(pymizu.call(source="summary(lm(y ~ x, as.data.frame(x)))$r.squared"),
         folds, template=np.empty(1)).collect()
pool.stop()

Each element becomes the spec’s first positional argument (or binds as x for a source= spec). Put constant arguments in the spec itself; the map’s own args= and kwargs= stay empty. template= and collect= work unchanged, and a per-element error crosses with its index. Seeds must be integers in R’s 32-bit range. Each worker language derives its own random streams from the seed, so results are reproducible per worker language but not identical across languages.

The dtype matrix

The exact correspondence between Python and R dtypes. Conversion happens once, at send time. The exact-match rows (float64, int32, int64, complex128, uint8) are a plain memory copy.

Python sends R receives Notes
bytes / uint8 raw uint8 with nulls: TypeError
int8 / int16 / uint16 integer widened
int32 integer
uint32 double widened
int64 integer64 (bit64’s layout) INT64_MIN reads as NA
uint64 integer64 (NA past 2^63 - 1, warned) frame column bit-exact
float16 / float32 / float64 double float16/32 widened
bool logical
complex64 / complex128 complex buffer protocol only
Arrow bool / numeric with nulls logical / numeric with NA validity bitmap honored
Arrow strings (utf8 / large_utf8 / string_view) character Series as a string list
Arrow date32 / timestamp[u] Date / POSIXct zone name rides frame metadata
Arrow duration[u] difftime (secs) exports back as duration[us]
timedelta64 (any non-calendar unit) difftime (secs) NaT reads as NA
Arrow dictionary-encoded factor ordered dictionary → ordered factor
Arrow ChunkedArray / Table / Series vector / data.frame chunks concatenated
Arrow nested, decimal, time32/64, interval — DeclinedError

Missing values cross as sentinels inside the data, and the read rules depend on the accessor.2

Round trips are stable after the first hop, and a pass-through echo is bit-exact — even the NA_real_ value survives a relay through a received view. A slice or a dtype view of a received view crosses by value.

Python sends R sees Back in Python Notes
uint8 raw uint8 exact
int8 / int16 / uint16 integer int32 widened
int32 integer int32 exact
uint32 double float64 exact
int64 integer64 int64 exact
uint64 integer64 (NA past 2^63 - 1, warned) uint64 frame column bit-exact
float16 double float64 widened
float32 double float64 widened
float64 double float64 exact
bool logical bool_ (int32 with NAs) dtype lost
complex64 / complex128 complex complex128 exact
timedelta64 difftime timedelta64[us] exact
R sends Python sees Back in R Notes
integer int32 (float64 with NAs) integer exact
integer64 (bit64) int64 (warns on NA) integer64 Python -2^63 reads as NA
double float64 double exact
raw uint8 raw exact
complex complex128 complex exact
logical bool_ (int32 with NAs) integer tag lost; whole view stays logical
difftime timedelta64[us] difftime (secs) NA crosses as NaT

Between two Python processes none of this applies: the exact-match dtypes cross unchanged and everything else pickles.

Footnotes

  1. Below the floor, a value crosses as one quick copy. At any size, these types cross: Python scalars (bool, int, float, complex, str, None), lists, dicts with str keys (a tuple arrives as a list in R), bytes (an R raw vector), and datetime.date / datetime.datetime scalars (length-1 dates and times in R). A few round trips change the container but not the values: a standalone R factor comes back as strings, a length-1 NA arrives as None and sent back is R’s NULL, and a standalone POSIXct normalizes to UTC (the instant is exact).↩︎

  2. R marks missing values with sentinels inside the data: INT_MIN for integer and logical, INT64_MIN for integer64, and a special NaN value (NA_real_) for double; Arrow nulls convert to these sentinels, so R sees correct NAs. Read from Python, the result depends on the accessor:

    • Copied reads (list(...), Frame.to_dict()) give exact dtypes: a logical arrives as bool_ when it has no NA, else int32. An integer arrives as int32 when NA-free, else float64 with NaN (every int32 value is exact in float64).
    • Zero-copy views keep the sentinel values in the buffer. .to_numpy() applies the same rule and checks the region’s validity marks for the presence of NAs before scanning any data.
    • int64 keeps its dtype in every numpy form and warns when an NA is detected on a copied read or .to_numpy(). Use .to_arrow() for an NA-exact read. A genuine Python -2^63 value round-trips without a warning.
    • Every Arrow export of a logical is Arrow bool with a validity bitmap, whatever the path. Integer exports are int32 or int64 with a validity bitmap. A double exports as float64 with NA_real_ slots as Arrow nulls (a genuine NaN stays a valid value).

    A genuine int32 value of -2147483648 collides with the NA sentinel and reads as NA in R, and an int64 value of -9223372036854775808 reads as NA_integer64_. numpy sends stay silent; an Arrow int32 or int64 send with a validity bitmap warns once, to an R peer, while a Python peer reads the identical columns exactly and stays silent. Python compute treats NA_real_ as an ordinary NaN, and whether the exact NaN value survives arithmetic is platform-dependent — do not rely on it.↩︎