R interop
The process at the other end of a channel does not have to be Python. mizu is pymizu’s R counterpart, built on the same shared-memory core, and a channel or pool connects the two languages as directly as two Python processes.
R channel peers
A channel peer can be an R process with the mizu package installed: pass the peer program as R source and set the launcher to pymizu.r_launcher(). Here, an R process doubles the array you send:
import numpy as np
import pymizu
ch = pymizu.Channel.create(
"""
library(mizu)
repeat {
x <- mizu_recv(ch, timeout = 30)
if (inherits(x, "mizu_sentinel")) break
mizu_send(ch, x * 2)
}
""",
launcher=pymizu.r_launcher(),
)
ch.send(np.array([1.5, 2.5, 3.5]))
print(ch.recv(timeout=5)) # [3. 5. 7.]
ch.close()r_launcher() needs R and the mizu package installed; if either is missing it raises MizuError before the channel exists. Pass rscript= to use a specific R installation.
The reverse direction works too — an R session spawns a Python peer with mizu::mizu_py_launcher().
Large data crosses without copying
Above a size threshold, pymizu writes the data to shared memory once and both sides read it in place — nothing is serialized, parsed, or copied again. R reads a large value you send directly over the shared pages; a large value you receive arrives as a view over those pages, usually wrapped as a numpy array. The peer below echoes back whatever it receives; this channel stays open for the next few examples:
ch = pymizu.Channel.create(
"""
library(mizu)
repeat {
x <- mizu_recv(ch, timeout = 30)
if (inherits(x, "mizu_sentinel")) break
mizu_send(ch, x)
}
""",
launcher=pymizu.r_launcher(),
)
big = np.random.default_rng(0).normal(size=1_250_000) # ten megabytes
ch.send(big)
back = ch.recv(timeout=5)
np.array_equal(back, big) # TrueApart from the one write into the shared region, no data moved: R’s echo came back as a reference, with no payload bytes. Send a received view onward and it crosses the same way, as the region’s name alone. Zero-copy covers every large layout: a flat array of an exact-match dtype (float64, int32, int64, complex128, uint8), an array of any shape and memory order, a nested tree of lists, dicts, arrays, and strings, a datetime64 array (NaT crosses as NA), or a list[str | None] (None crosses as NA). A large data frame crosses as one shared region in both directions. The floor is 256 KB for a flat contiguous array on a channel, 32 KB everywhere else.1
Arrays of other numeric dtypes (booleans, the narrow ints, uint64, float32, …) cross as a converted copy at any size; the dtype matrix has the per-dtype rule. Without numpy, a received view is a plain buffer: memoryview(view) works, and pa.array(view) or pl.from_arrow(view) wrap the same pages without copying. One exception: R’s ALTREP vectors (such as 1:1e9, stored as a recipe rather than data) cross by value, values exact. A polars or pyarrow object that wraps a received frame keeps the frame’s region alive until garbage collection, so delete these wrappers when you no longer need them.
What crosses
Small values round-trip exactly too. Each form below comes back from R unchanged:
def echo(x):
ch.send(x)
return ch.recv(timeout=5)
x = np.array([1.5, 2.5, 3.5])
np.array_equal(echo(x), x) # a float64 vector: True
m = np.arange(6, dtype=np.int32).reshape(2, 3)
np.array_equal(echo(m), m) # an int32 matrix, dim included: True
echo("hello") == "hello" # a string: True
d = np.array(["2024-01-01", "2024-01-03"], dtype="datetime64[D]")
np.array_equal(echo(d), d) # dates keep their meaning: TrueA data frame crosses as well, with its column types intact. Here a polars frame with a categorical column; pandas and pyarrow frames work the same way:
import polars as pl
df = pl.DataFrame({
"total_bill": [16.99, 10.34, 21.01],
"day": pl.Series(["Sun", "Sun", "Mon"], dtype=pl.Categorical),
})
ch.send(df) # arrives in R as a data.frame, the categorical column as a factor
back = ch.recv(timeout=5)
back.names # ('total_bill', 'day')R sees the natural equivalent throughout: a vector for an array, a factor for a category column, a named list for a dict, a Date or POSIXct for a datetime64 array. An Arrow array arrives as an R vector, with Arrow nulls as R missing values: ch.send(pa.array([1, None, 3])). An ordered categorical — an R ordered factor, a polars Enum, an ordered pyarrow dictionary — crosses with its ordering intact. Labels cross with the values they name: a named R vector arrives as a DimArray carrying its names, and labels you attach in Python cross to R as names or dimnames.
What doesn’t cross
Some values have no equivalent on the other side. pymizu declines them at send time with a DeclinedError (a subclass of TypeError) that names the offending value on its path and reason attributes. The channel stays usable:
import decimal
import pyarrow as pa
tbl = pa.table({"price": pa.array([decimal.Decimal("1.5")])})
try:
ch.send(tbl)
except pymizu.DeclinedError as e:
print(e)
# pymizu: value is not portable to the peer (column 'price': an Arrow type
# with no portable home (format 'd:2,1'))
ch.close()From Python, also declined: Arrow nested, date64, time, and interval columns. An Arrow uint8 column with nulls is a TypeError: an R raw vector has no NA.
From R: a POSIXlt time, a character matrix, and S4 objects stay on R’s side. So does a named vector with a class or of character type (a named character vector, factor, Date, or POSIXct), since names travel only on the plain atomic vectors. Among data frames: a data.table, a groupedData-style frame (a formula attribute), and a column that is itself a labeled matrix.
The counterpart of the send-time decline is a value that arrives in a language-private format no send gate caught (an attach race, or a misbehaving or version-skewed peer). The read raises ForeignPayloadError and the message is consumed, so one bad payload never blocks the channel; under recv_async(), only that message’s waiter rejects.
pymizu.Frame
A pymizu.Frame is Python’s home for an R data.frame: named columns with a row count. frame.to_dict() returns the columns as a dict: numpy arrays when numpy is installed, memoryviews otherwise, strings and factors as list[str | None]. frame.names and frame.row_names carry the metadata.
Any Arrow consumer takes a frame in one line: pl.from_arrow(f), pa.table(f), or pd.DataFrame.from_arrow(f). (With polars >= 2.0, a streaming producer gives a struct Series; .struct.unnest() restores the columns.) Factors become dictionary columns, logicals Arrow bool, Date date32, and POSIXct timestamp[us].
A Frame pickles, so it crosses Python-to-Python channels. Sent to R, it becomes a data.frame again.
Frame metadata crosses unchanged: a data.frame’s full class vector (a tibble’s c("tbl_df", "tbl", "data.frame"), any class ending in "data.frame") and extra attributes travel as-is, stored on the Frame uninterpreted. An unmodified echo restores them in R; a frame rebuilt through pyarrow, polars, pandas, or from_arrow arrives in R as a plain data.frame.
Mixed-language pools
A pool’s workers can be R processes: pymizu.r_launcher() spawns them, and pymizu.call() builds a language-neutral task — a qualified "pkg::fn" name for R or "mod.fn" for Python, or a source= string, plus constant arguments:
import numpy as np
import pymizu
pool = pymizu.Pool.create(4, launcher=pymizu.r_launcher())
t = pool.submit(pymizu.call("stats::quantile", np.arange(101.0),
probs=np.array([0.25, 0.5, 0.75]), names=False))
print(t.collect()) # [ 25. 50. 75.]
t2 = pool.submit(pymizu.call(source="y <- x * 2\ny + 1", x=20))
print(t2.collect()) # 41
pool.stop()Unnamed arguments become positional, keyword arguments named, matching R’s calling convention. A source= task runs in a fresh environment with keyword arguments bound as names and positional arguments as _1, _2, …; the result is the value of the last expression.
On an R-worker pool, submitting a plain Python callable raises an error naming pymizu.call(). Task errors cross as TaskError with the R error type preserved in remote_type. Arguments follow the same rules as channel payloads, zero-copy included: a large array crosses as one shared view, and an array that is already a received view crosses as its name alone. A pool’s workers all run one language — mixing R and Python workers in one pool is not supported. The reverse direction works the same way: mizu::mizu_pool() with mizu::mizu_py_launcher(), and mizu::mizu_submit_call() with mizu::mizu_call() specs.
Mapping over a foreign pool
To map an R function over Python data, pass a pymizu.call() spec as the fn of Pool.map() or Pool.map_prepare() on an R-worker pool. mizu::mizu_map() mirrors this in R:
import numpy as np
rng = np.random.default_rng(0)
x = np.arange(50.0)
y = 1 + 2 * x + rng.normal(scale=5, size=50)
folds = [
{"x": xf, "y": yf}
for xf, yf in zip(x.reshape(5, 10), y.reshape(5, 10))
]
pool = pymizu.Pool.create(4, launcher=pymizu.r_launcher())
pool.map(pymizu.call("stats::median"), list(y.reshape(5, 10))).collect()
pool.map(pymizu.call(source="summary(lm(y ~ x, as.data.frame(x)))$r.squared"),
folds, template=np.empty(1)).collect()
pool.stop()Each element becomes the spec’s first positional argument (or binds as x for a source= spec). Put constant arguments in the spec itself; the map’s own args= and kwargs= stay empty. template= and collect= work unchanged, and a per-element error crosses with its index. Seeds must be integers in R’s 32-bit range. Each worker language derives its own random streams from the seed, so results are reproducible per worker language but not identical across languages.
The dtype matrix
The exact correspondence between Python and R dtypes. Conversion happens once, at send time. The exact-match rows (float64, int32, int64, complex128, uint8) are a plain memory copy.
| Python sends | R receives | Notes |
|---|---|---|
bytes / uint8 |
raw | uint8 with nulls: TypeError |
| int8 / int16 / uint16 | integer | widened |
| int32 | integer | |
| uint32 | double | widened |
| int64 | integer64 (bit64’s layout) | INT64_MIN reads as NA |
| uint64 | integer64 (NA past 2^63 - 1, warned) |
frame column bit-exact |
| float16 / float32 / float64 | double | float16/32 widened |
| bool | logical | |
| complex64 / complex128 | complex | buffer protocol only |
| Arrow bool / numeric with nulls | logical / numeric with NA |
validity bitmap honored |
| Arrow strings (utf8 / large_utf8 / string_view) | character | Series as a string list |
Arrow date32 / timestamp[u] |
Date / POSIXct |
zone name rides frame metadata |
Arrow duration[u] |
difftime (secs) |
exports back as duration[us] |
| timedelta64 (any non-calendar unit) | difftime (secs) |
NaT reads as NA |
| Arrow dictionary-encoded | factor | ordered dictionary → ordered factor |
| Arrow ChunkedArray / Table / Series | vector / data.frame |
chunks concatenated |
| Arrow nested, decimal, time32/64, interval | — | DeclinedError |
Missing values cross as sentinels inside the data, and the read rules depend on the accessor.2
Round trips are stable after the first hop, and a pass-through echo is bit-exact — even the NA_real_ value survives a relay through a received view. A slice or a dtype view of a received view crosses by value.
| Python sends | R sees | Back in Python | Notes |
|---|---|---|---|
| uint8 | raw | uint8 | exact |
| int8 / int16 / uint16 | integer | int32 | widened |
| int32 | integer | int32 | exact |
| uint32 | double | float64 | exact |
| int64 | integer64 | int64 | exact |
| uint64 | integer64 (NA past 2^63 - 1, warned) |
uint64 | frame column bit-exact |
| float16 | double | float64 | widened |
| float32 | double | float64 | widened |
| float64 | double | float64 | exact |
| bool | logical | bool_ (int32 with NAs) |
dtype lost |
| complex64 / complex128 | complex | complex128 | exact |
| timedelta64 | difftime |
timedelta64[us] | exact |
| R sends | Python sees | Back in R | Notes |
|---|---|---|---|
| integer | int32 (float64 with NAs) | integer | exact |
| integer64 (bit64) | int64 (warns on NA) |
integer64 | Python -2^63 reads as NA |
| double | float64 | double | exact |
| raw | uint8 | raw | exact |
| complex | complex128 | complex | exact |
| logical | bool_ (int32 with NAs) |
integer | tag lost; whole view stays logical |
| difftime | timedelta64[us] | difftime (secs) | NA crosses as NaT |
Between two Python processes none of this applies: the exact-match dtypes cross unchanged and everything else pickles.
Footnotes
Below the floor, a value crosses as one quick copy. At any size, these types cross: Python scalars (
bool,int,float,complex,str,None), lists, dicts withstrkeys (a tuple arrives as alistin R),bytes(an R raw vector), anddatetime.date/datetime.datetimescalars (length-1 dates and times in R). A few round trips change the container but not the values: a standalone R factor comes back as strings, a length-1NAarrives asNoneand sent back is R’sNULL, and a standalonePOSIXctnormalizes to UTC (the instant is exact).↩︎R marks missing values with sentinels inside the data:
INT_MINfor integer and logical,INT64_MINfor integer64, and a special NaN value (NA_real_) for double; Arrow nulls convert to these sentinels, so R sees correctNAs. Read from Python, the result depends on the accessor:- Copied reads (
list(...), Frame.to_dict()) give exact dtypes: a logical arrives asbool_when it has noNA, else int32. An integer arrives as int32 whenNA-free, else float64 with NaN (every int32 value is exact in float64). - Zero-copy views keep the sentinel values in the buffer.
.to_numpy()applies the same rule and checks the region’s validity marks for the presence of NAs before scanning any data. - int64 keeps its dtype in every numpy form and warns when an
NAis detected on a copied read or.to_numpy(). Use.to_arrow()for an NA-exact read. A genuine Python-2^63value round-trips without a warning. - Every Arrow export of a logical is Arrow
boolwith a validity bitmap, whatever the path. Integer exports are int32 or int64 with a validity bitmap. A double exports as float64 withNA_real_slots as Arrow nulls (a genuine NaN stays a valid value).
A genuine int32 value of
-2147483648collides with theNAsentinel and reads asNAin R, and an int64 value of-9223372036854775808reads asNA_integer64_. numpy sends stay silent; an Arrow int32 or int64 send with a validity bitmap warns once, to an R peer, while a Python peer reads the identical columns exactly and stays silent. Python compute treatsNA_real_as an ordinary NaN, and whether the exact NaN value survives arithmetic is platform-dependent — do not rely on it.↩︎- Copied reads (