R interop

The process at the other end of a channel does not have to be Python. mizu is the R counterpart of pymizu, built on the same shared-memory core. A channel or pool connects the two languages as directly as two Python processes.

R channel peers

A channel peer can be an R process with the mizu package installed. Pass the peer program as R source, and set the launcher to pymizu.r_launcher(). Here, an R process doubles the array you send:

import numpy as np
import pymizu

ch = pymizu.Channel.create(
    """
library(mizu)
repeat {
  x <- mizu_recv(ch, timeout = 30)
  if (inherits(x, "mizu_sentinel")) break
  mizu_send(ch, x * 2)
}
""",
    launcher=pymizu.r_launcher(),
)

ch.send(np.array([1.5, 2.5, 3.5]))
print(ch.recv(timeout=5))            # [3. 5. 7.]
ch.close()

r_launcher() needs R and the mizu R package installed. If either is missing, it raises MizuError before the channel is created. Pass rscript= to use a specific R installation.

To spawn the peer differently, write your own launcher: one callable that takes the channel’s token and starts the peer process. The reverse direction works too: an R session spawns a Python peer with mizu::mizu_py_launcher().

Large data crosses without copying

Past a size floor, the data itself never moves. pymizu writes it to shared memory once, and both sides read it in place: no serialization, no parsing, no further copies. A large value you send is read by R directly over the shared pages. A large value you receive arrives as a view over those pages, usually wrapped as a numpy array. The peer below echoes back whatever it receives. This channel stays open for the next few examples:

ch = pymizu.Channel.create(
    """
library(mizu)
repeat {
  x <- mizu_recv(ch, timeout = 30)
  if (inherits(x, "mizu_sentinel")) break
  mizu_send(ch, x)
}
""",
    launcher=pymizu.r_launcher(),
)

big = np.random.default_rng(0).normal(size=1_250_000)  # ten megabytes
ch.send(big)
back = ch.recv(timeout=5)
np.array_equal(back, big)                              # True

Apart from that one write into the shared region, nothing moved at all. R’s echo came back as a reference with zero payload bytes. If you send a received view onward, it crosses the same way: as the region’s name alone. The zero-copy tier covers every large layout: a flat array of an identity dtype (float64, int32, int64, complex128, uint8), an array of any shape and memory order, or a nested tree of lists, dicts, arrays, and strings. A datetime64 array crosses the same way, with NaT as NA, and a list[str | None] with None as NA. A large data frame crosses as one shared region in both directions. The floor is low: 256 KB for a flat contiguous array on a channel, and 32 KB everywhere else.¹

Arrays of other numeric dtypes (booleans, the narrow ints, uint64, float32, …) cross as a converted copy at any size. The dtype matrix has the per-dtype rule. Without numpy, a received view is a plain buffer: memoryview(view) works, and pa.array(view) or pl.from_arrow(view) wrap the same pages without copying. One exception: R’s ALTREP vectors (say, 1:1e9, stored as a recipe rather than data) cross by value, values exact. A polars or pyarrow object that wraps a received frame keeps the frame’s region alive until garbage collection, so delete these wrappers when you no longer need them.

What crosses

Small values round-trip just as exactly. Each form below comes back from R unchanged:

def echo(x):
    ch.send(x)
    return ch.recv(timeout=5)

x = np.array([1.5, 2.5, 3.5])
np.array_equal(echo(x), x)     # a float64 vector: True

m = np.arange(6, dtype=np.int32).reshape(2, 3)
np.array_equal(echo(m), m)     # an int32 matrix, dim included: True

echo("hello") == "hello"       # a string: True

d = np.array(["2024-01-01", "2024-01-03"], dtype="datetime64[D]")
np.array_equal(echo(d), d)     # dates keep their meaning: True

A data frame crosses as well, with its column types intact. Here a polars frame with a categorical column; pandas and pyarrow frames work the same way:

import polars as pl
df = pl.DataFrame({
    "total_bill": [16.99, 10.34, 21.01],
    "day": pl.Series(["Sun", "Sun", "Mon"], dtype=pl.Categorical),
})
ch.send(df)        # arrives in R as a data.frame, the categorical column as a factor
back = ch.recv(timeout=5)
back.names         # ('total_bill', 'day')

R sees the natural equivalent throughout: a vector for an array, a factor for a category column, a named list for a dict, a Date or POSIXct for a datetime64 array. An Arrow array arrives as an R vector, with Arrow nulls as R missing values: ch.send(pa.array([1, None, 3])). An ordered categorical (an R ordered factor, a polars Enum, an ordered pyarrow dictionary) crosses as an ordered factor in R, and the Frame’s Arrow export keeps the ordered flag, so pandas and polars see an ordered categorical and a round trip is identical(). Labels cross with the values they name: a named R vector arrives as a DimArray carrying its names, and labels you attach in Python ride to R as names or dimnames.

What doesn’t cross

Some values have no portable home on the other side. pymizu declines these at send time with a DeclinedError (a subclass of TypeError) that names the offending value, carried on its path and reason attributes. The channel stays usable:

import decimal
import pyarrow as pa

tbl = pa.table({"price": pa.array([decimal.Decimal("1.5")])})
try:
    ch.send(tbl)
except pymizu.DeclinedError as e:
    print(e)
    # pymizu: value is not portable to the peer (column 'price': an Arrow type
    # with no portable home (format 'd:2,1'))
ch.close()

From Python, also declined: Arrow nested, date64, time, and interval columns. An Arrow uint8 column with nulls is a TypeError: an R raw vector has no NA.

From R: a POSIXlt time, a character matrix, and S4 objects stay on R’s side. So does a named vector with a class or of character type (a named character vector, factor, Date, or POSIXct), since names ride only on the plain atomic vectors. Among data frames: a data.table, a groupedData-style frame (a formula attribute), and a column that is itself a labeled matrix.

pymizu.Frame

A pymizu.Frame is Python’s home for an R data.frame: named columns with a row count. frame.to_dict() returns the columns as a dict: numpy arrays when numpy is installed, memoryviews otherwise, strings and factors as list[str | None]. frame.names and frame.row_names carry the metadata.

Any Arrow consumer takes a frame in one line: pl.from_arrow(f), pa.table(f), or pd.DataFrame.from_arrow(f). (With polars >= 2.0, a streaming producer gives a struct Series; .struct.unnest() restores the columns.) Factors become dictionary columns, logicals Arrow bool, Date date32, and POSIXct timestamp[us].

A Frame pickles, so it crosses Python-to-Python channels. Sent to R, it becomes a data.frame again.

Frame metadata rides along verbatim: the full class vector of a data.frame (a tibble’s c("tbl_df", "tbl", "data.frame"), any class ending in "data.frame") and extra attributes cross opaquely, stashed undecoded on the Frame. An unmodified echo restores them in R. A frame rebuilt through pyarrow, polars, pandas, or from_arrow arrives in R as a plain data.frame.

Mixed-language pools

A pool’s workers can be R processes. pymizu.r_pool_launcher() spawns the workers. pymizu.call() builds a language-neutral task: a qualified "pkg::fn" name for R or "mod.fn" for Python, or a source= string, plus constant arguments:

import numpy as np
import pymizu

pool = pymizu.Pool.create(4, launcher=pymizu.r_pool_launcher())
t = pool.submit(pymizu.call("stats::quantile", np.arange(101.0),
                            probs=np.array([0.25, 0.5, 0.75]), names=False))
print(t.collect())                  # [ 25.  50.  75.]
t2 = pool.submit(pymizu.call(source="y <- x * 2\ny + 1", x=20))
print(t2.collect())                 # 41
pool.stop()

Unnamed arguments become positional arguments, and keyword arguments become named ones, matching R’s calling convention. A source= task runs in a fresh environment with the keyword arguments bound as names and the positional arguments bound as _1, _2, … The result is the value of the last expression.

On an R-worker pool, submitting a plain Python callable raises an error naming pymizu.call(). Task errors cross as TaskError with the R error type preserved in remote_type. Arguments follow the same rules as channel payloads, zero-copy included: a large array crosses as one shared view, and an array that is already a received view crosses as its name alone. A pool’s workers all run one language: mixing R and Python workers in one pool is not supported. The reverse direction mirrors this: mizu::mizu_pool() with mizu::mizu_py_pool_launcher(), and mizu::mizu_submit_call() with mizu::mizu_call() specs.

Mapping over a foreign pool

To map an R function over Python data, pass a pymizu.call() spec as the fn of Pool.map() or Pool.map_prepare() on an R-worker pool. mizu::mizu_map() mirrors this in R:

pool = pymizu.Pool.create(4, launcher=pymizu.r_pool_launcher())
pool.map(pymizu.call("stats::median"), x).collect()
pool.map(pymizu.call(source="summary(mgcv::gam(x$y ~ s(x$x)))$r.sq"),
         folds, template=0.0).collect()   # each fold a dict; the element binds as x
pool.stop()

Each element becomes the spec’s first positional argument, or binds as x for a source= spec. Put constant arguments in the spec itself: the map’s own args= and kwargs= stay empty. template= and collect= work unchanged, and a per-element error crosses with its index. Seeds must be integers within R’s 32-bit range. Each worker language derives its own random streams from the seed, so results are reproducible for a given worker language but not identical across languages.

The dtype matrix

The exact correspondence between Python and R dtypes. Conversion happens once, at send time. The identity rows (float64, int32, int64, complex128, uint8) are a plain memcpy.

Python sends	R receives	Notes
`bytes` / uint8	raw	Arrow uint8 with nulls: `TypeError` (R raw has no NA)
int8 / int16 / uint16	integer	widened, exact
int32	integer
uint32	double	widened, exact
int64	integer64 (bit64’s layout)	bit-exact; `INT64_MIN` reads as `NA`
uint64	integer64 (`NA` past 2^63 - 1, warned)	the frame column and `mizu_as_arrow()` are bit-exact
float16 / float32 / float64	double	float16 and float32 widened, exact
bool	logical
complex64 / complex128	complex	(buffer protocol only; Arrow has no standard complex)
Arrow bool / numeric with nulls	logical / numeric with `NA`	the validity bitmap is honored, slices included
Arrow strings (utf8 / large_utf8 / string_view)	character	a Series as `list[str \\| None]`; a column as a frame column
Arrow `date32` / `timestamp[u]`	`Date` / `POSIXct`	the zone name rides a frame column’s metadata
Arrow `duration[u]`	`difftime` (secs)	the frame column exports back as `duration[us]`
timedelta64 (any non-calendar unit)	`difftime` (secs)	`NaT` reads as `NA`; the unit converts to seconds
Arrow dictionary-encoded	factor	an ordered dictionary (polars `Enum`) arrives as an ordered factor
Arrow ChunkedArray / Table / Series	vector / `data.frame`	chunks are concatenated
Arrow nested, decimal, time32/64, interval	—	DeclinedError at send time

Missing values:

R marks missing values with sentinels inside the data: INT_MIN for integer and logical, INT64_MIN for integer64, and a special NaN value (NA_real_) for double. Arrow nulls convert to these sentinels, so R sees correct NAs.
Read from Python, the result depends on the accessor:
- Copied reads (list(...), Frame.to_dict()) give exact dtypes: a logical arrives as bool_ when it has no NA, else int32. An integer arrives as int32 when NA-free, else float64 with NaN (every int32 value is exact in float64).
- Zero-copy views keep the sentinel values in the buffer. .to_numpy() applies the same rule and reads NA-freeness off the region’s validity section before any data scan.
- int64 keeps its dtype in every numpy form and warns when an NA is detected on a copied read or .to_numpy(). Use .to_arrow() for an NA-exact read. A genuine Python -2^63 value round-trips without a warning.
- Every Arrow export of a logical is Arrow bool with a validity bitmap, whatever the path. Integer exports are int32 or int64 with a validity bitmap. A double exports as float64 with NA_real_ slots as Arrow nulls (a genuine NaN stays a valid value).
A genuine int32 value of -2147483648 collides with the NA sentinel and reads as NA in R. Likewise an int64 value of -9223372036854775808 reads as NA_integer64_. numpy sends stay silent. An Arrow int32 or int64 send with a validity bitmap warns once, to an R peer. A Python peer reads the identical columns exactly and stays silent.
Python compute treats NA_real_ as an ordinary NaN. Whether the exact NaN value survives arithmetic is platform-dependent. Do not rely on it.

Round trips are stable after the first hop, and a pass-through echo is bit-exact. An untouched received view sent back crosses as a reference to its region, so no data moves and even the NA_real_ value survives a relay. A slice or a dtype view of a received view crosses by value.

Python sends	R sees	Back in Python	Notes
uint8	raw	uint8	exact
int8 / int16 / uint16	integer	int32	widened, values exact
int32	integer	int32	exact
uint32	double	float64	exact
int64	integer64	int64	exact
uint64	integer64 (`NA` past 2^63 - 1, warned)	uint64	the column and R’s Arrow export are bit-exact
float16	double	float64	widened, values exact
float32	double	float64	widened, values exact
float64	double	float64	exact
bool	logical	`bool_` (int32 with NAs)	dtype lost
complex64 / complex128	complex	complex128	exact
timedelta64	`difftime`	timedelta64[us]	unit normalizes to us, values exact

R sends	Python sees	Back in R	Notes
integer	int32 (float64 with NAs)	integer	exact
integer64 (bit64)	int64 (warns on `NA`)	integer64	exact; a genuine Python `-2^63` arrives as `NA`
double	float64	double	exact
raw	uint8	raw	exact
complex	complex128	complex	exact
logical	`bool_` (int32 with NAs)	integer	tag lost on a Python round trip (values kept); a whole view sent back stays logical
difftime	timedelta64[us]	difftime (secs)	units normalize to seconds, values exact; `NA` crosses as `NaT`

Between two Python processes, on a channel or a pool, the conversions on this page never run. The identity dtypes cross unchanged, and everything else pickles. An exact Python-to-Python send of a non-identity dtype needs no work. They apply only when the other end runs R.

Footnotes

Below the floor, a value crosses as one quick copy. At any size, these types cross: Python scalars (bool, int, float, complex, str, None), lists, dicts with str keys (a tuple arrives as a list in R), bytes (an R raw vector), and datetime.date / datetime.datetime scalars (length-1 dates and times in R). A few round trips change the container but never the values. A standalone R factor comes back as strings, a length-1 NA arrives as None and sent back is R’s NULL, and an R logical vector with NAs comes back as integer. A POSIXct timezone normalizes to UTC (the instant is exact), and difftime units normalize to seconds. The bit patterns -2147483648 (int32) and -9223372036854775808 (int64) collide with R’s own NA sentinels and read as NA in R. A uint64 up to 2^63 - 1 converts exactly to integer64; past that it crosses as NA with a warning at send.↩︎