Channels

A channel is a two-way message link between a Python process and a helper process that it spawns: one lock-free ring per direction, over shared memory. One process writes data, and the other reads it in place, never copied through a socket, pipe, or file.

Creating a channel

Channel.create() spawns the peer process and connects both ends. The peer program is a Python source string, evaluated in the peer with ch bound to its side of the channel:

import pymizu

ch = pymizu.Channel.create("""
import pymizu
while True:
    x = ch.recv()
    if x is pymizu.CLOSED:
        break
    ch.send(x)
""")

The peer is a full Python process: it imports anything importable in the spawning interpreter’s environment. Channel.create() returns after the peer connects. It waits up to startup_timeout seconds and raises StartupError on expiry.

A launcher changes how pymizu spawns the peer. It is one callable that takes the channel’s token and arranges for the peer process to start. pymizu.r_launcher() is the built-in example: it spawns an R peer instead (see R interop). User code spawns peers through a launcher. Channel.attach(token) is the low-level entry for a peer that attaches itself.

Sending and receiving

ch.send() publishes a value, visible to the peer the moment the call returns. Sends never block for ring space. ch.recv() returns the next value, waiting up to timeout seconds (indefinitely when omitted):

ch.send([1, "a", None])
ch.recv(timeout=5)

[1, 'a', None]

The outcomes that end a conversation come back as sentinel singletons, never raised:

Sentinel	Returned when
`pymizu.FULL`	a send finds the ring full
`pymizu.TIMEOUT`	a receive’s timeout expires
`pymizu.CLOSED`	the peer has closed and the ring is drained
`pymizu.PEER_GONE`	the peer process has died and the ring is drained

Test them by identity (x is pymizu.TIMEOUT) or with pymizu.is_sentinel(x). No call needs an error handler. If the peer dies, receives first drain what it already sent, then report PEER_GONE.

send_batch() and recv_batch() move several values in one call. Use them at rates where the per-call overhead starts to matter:

ch.send_batch(range(4))
ch.recv_batch(4, timeout=5)

[0, 1, 2, 3]

ch.alive() reports whether the peer still runs, without touching the rings. ch.info() returns a read-only snapshot of the channel’s state.

asyncio

ch.recv_async() is the event-driven receive: an asyncio.Future for the next value, resolved off the channel’s notify endpoint — no blocking, no polling, and no timeout argument (asyncio.timeout / asyncio.wait_for are the idiom). Terminal states stay values: the Future resolves with CLOSED / PEER_GONE. Several outstanding calls resolve in FIFO order, one message each. Cancelling the Future detaches the waiter, passing the next message to the one after.

ch.send(1)
assert await ch.recv_async() == 1
ch.send(2)
await ch.recv_async()

async for x in ch is the stream form: it awaits each value through recv_async() and ends on CLOSED or PEER_GONE, exactly as the sync iterator ends.

Closing

ch.close() is the orderly shutdown: it signals the peer and waits for it to finish draining before releasing the shared resources:

ch.close()

True

A channel is also a context manager: the with block closes it on exit. close_signal() signals without waiting. destroy() tears the handle down immediately, without the close handshake. The peer observes the channel as closed (PEER_GONE is the hard-death outcome). A destroy is safe from any thread. A verb in flight on another thread pins the handle, and teardown completes when the last one returns. An indefinite wait (recv(), recv_batch(), ready_wait(), close()) breaks out within a bounded slice, so the teardown and the peer’s observation of it follow promptly.

What crosses a channel

pymizu moves each value the cheapest way it can:

None crosses as a marker: no data moves.
Buffer-protocol objects (bytes, 1-D C-contiguous numpy arrays) cross without serialization. float64, int32, int64, complex128, and uint8 keep their exact values and arrive as arrays (bytes arrives as a uint8 array). numpy scalars and 0-d arrays keep their exact type and shape. A large buffer arrives as a read-only zero-copy view over the shared pages: no copy, no parse. A contiguous buffer below 256 KB instead crosses as one copy into the channel’s spill arena — cheaper than region machinery at those sizes; from 256 KB the receive is a view.
A str within the inline budget crosses as raw UTF-8.
Booleans, numbers, and flat lists or dicts of them cross in a compact binary form.
Everything else crosses as a pickle (protocol 4).

Each end of a channel knows the peer’s language. Between two Python processes that is the whole story: arrays of the dtypes in this list cross unchanged, and everything else pickles. With an R peer, values must fit a portable subset that both languages share. A value outside it raises pymizu.DeclinedError at send time, naming the value and the reason. The channel is unharmed. See R interop.

An uncaught error in the peer crosses as a value, not a raised exception: a pymizu.TaskError carrying the original exception’s class name and traceback text. pymizu.is_remote_error(x) tests a received value. Raise it to propagate the error.

Frames without pickling

Between two Python processes, a pyarrow Table, polars DataFrame, or duckdb relation pickles by default: container-exact, but a full serialize/parse round trip. pymizu.Frame.from_arrow() opts an Arrow producer into the frame path instead: one shared-memory region on the wire, read in place at receive:

import pyarrow as pa

with pymizu.Channel.create("""
import pymizu
while True:
    x = ch.recv()
    if x is pymizu.CLOSED:
        break
    ch.send(x)
""") as ch2:
    f = pymizu.Frame.from_arrow(
        pa.table({"x": pa.array([1.5, None] * 100_000)})
    )
    ch2.send(f)
    echoed = pa.table(ch2.recv(timeout=5))
echoed.column("x").null_count

Construction adopts the producer’s own buffers wherever the frame’s column layout is the Arrow layout: a single-batch producer on offset-0 boundaries, nulls included (the validity bitmap adopts with the values). It copies nothing it does not have to:

Stage	Copies
producer → Frame.from_arrow()	0 per adopted column; a per-column copy only where its type forces it
`ch.send(f)`	1 — the layout write into the shared region (the region is the wire)
`ch.recv()`	0 — the frame reads the region in place
`pa.table(f)`	0 — the frame’s __arrow_c_stream__ exports borrowed buffers

Values cross exactly, but types normalize. A column also copies, never declines, where its type forces it:

In	Out (the received frame’s Arrow export)
`string`	`large_string`
`string_view`	`large_string` (copies)
`bool`	`bool` (copies: bit-packed)
int8 / int16 / uint16	int32 (copies)
uint32 / half_float / float32	float64 (copies)
uint64	uint64 (adopts)
timestamp[s / ms / us / ns]	timestamp[s / ms / us / ns] (adopts)
duration[s / ms / us / ns]	duration[s / ms / us / ns] (adopts)
dictionary<i32, utf8>	dictionary<i32, string> (adopts; other index widths copy)
chunked or sliced columns	concatenated or rebased (copies)

Chunked pyarrow tables adopt after table.combine_chunks(). Timestamps and durations keep their counts and unit verbatim: a timestamp[ns] round trip is exact to the nanosecond, tz included. Nanosecond fidelity needs pymizu from this version on both ends. A peer without the capability (an older pymizu, or R) receives the microsecond form instead: values rounded to the double-second wire’s precision, never an error.

Adopted buffers stay alive for the frame’s lifetime: a frame from a 4 GB table pins it, like a numpy view. to_dict()’s masked reads copy where the Arrow export does not. When you want the exact producer type back, keep the default: a plain pyarrow or polars object pickles, container-exact. On a pool, a Frame task argument and a frame returned from a task both ride the shared-memory region as on a channel (see the pools guide). Pool.map does not take a frame as its element source.

Sizing

Channel.create() takes capacity (ring slots, a power of two), slot_size (bytes per slot, so the inline payload budget is slot_size - 16), and arena_size (the channel’s spill arena). A value past the inline budget spills to the arena or a fresh shared-memory region. spin=True selects pure-spin waiting: receivers never park to the OS. This gives the lowest latency, at full CPU use while waiting.