pymizu 水

AI / Agents

Skills
llms.txt
llms-full.txt

Developers

Charlie Gao

Author

Posit Software, PBC

Community

Contributing guide
Code of conduct
Full license MIT
Citing pymizu

Meta

Requires: Python >=3.10
Provides-Extra: numpy, cloudpickle, dev, docs, all
Package Info

      ________
     /\       \
    /  \ pymizu\
    \  /  水   /
     \/_______/

pymizu uses shared memory to make communication between Python processes fast enough to parallelize work at thread granularity.

Task round trips drop from around 100 µs over sockets to under a microsecond — two orders of magnitude (see Benchmarks). The hot path never enters the kernel and data is never copied through a system socket.

In the default CPython build, the GIL runs CPU-bound threads on one core at a time, so compute parallelism in Python usually means multiple processes. Processes also isolate failures: a worker that crashes does not take the host down with it.

Channels and work-stealing task pools run over POSIX shared memory (Linux, macOS) or Win32 file mappings (Windows): lock-free single-producer single-consumer rings with batched publication, spin-then-park waiting, and peer death delivered as an OS event.

pymizu is built on libmizu, a C library for lock-free shared-memory IPC. mizu binds the same core for R, so Python and R can talk to each other.

Pre-release. The API is not stable and may change at any time before a release.

Installation

pip install git+https://github.com/shikokuchuo/pymizu

Requires Python 3.10 or later on a 64-bit platform. See Installation for platform notes, optional extras, and deployment on Linux.

Quickstart

import pymizu

with pymizu.Pool.create(4) as pool:
    task = pool.submit(pow, 2, 16)
    result = task.collect(timeout=5)

result
65536

Benchmarks

Communication overhead against the stdlib concurrent.futures pools (Apple M4 Pro, from benchmarks/mizu-stdlib-bench.py).

Against ProcessPoolExecutor (tasks run in separate processes, with pickled payloads):

Benchmark pymizu ProcessPoolExecutor Speedup
Trivial task round trip 0.7 µs 94.2 µs 135x
Pipelined throughput, 1 worker 2,390,000 tasks/s 18,500 tasks/s 129x
Parallel map overhead, trivial function, 4 workers 0.4 µs/elt* 70.3 µs/elt 176x
Parallel map of 2,000 ~5 µs tasks, 4 workers 4.3 ms 129 ms 30x

Against ThreadPoolExecutor (tasks share one process, so the GIL caps CPU-bound work at a single core):

Benchmark pymizu ThreadPoolExecutor Speedup
Trivial task round trip 0.7 µs 9.5 µs 14x
Pipelined throughput, 1 worker 2,390,000 tasks/s 405,000 tasks/s 5.9x
Parallel map overhead, trivial function, 4 workers 0.4 µs/elt* 2.6 µs/elt 6.5x
Parallel map of 2,000 ~5 µs tasks, 4 workers 4.3 ms 50 ms 12x

* elt = element; microseconds of wall time per map element.

See Benchmarks for how pymizu compares with other frameworks, and for the benchmark scripts.

Drop-in concurrent.futures

PoolExecutor adapts a pool to the stdlib concurrent.futures.Executor interface — code written against ProcessPoolExecutor drops in, and submit returns real Future objects:

with pymizu.PoolExecutor.create(4) as ex:
    futures = [ex.submit(pow, 2, i) for i in range(4)]
    result = [f.result() for f in futures]

result
[1, 2, 4, 8]

See Task pools for wrapping an existing pool and the cancellation semantics.

Parallel map

Pool.map(fn, x) maps fn over x on the pool and returns the run’s handle immediately. The function and data cross into shared memory once. Workers divide the elements among themselves, claiming ranges from a shared cursor. handle.collect() returns the results in order — or await handle in asyncio code.

A map also nests — the mapped function can map over its own pool with pymizu.current_pool(), deadlock-free.

with pymizu.Pool.create(4) as pool:
    result = pool.map(abs, range(-5, 5)).collect()

result
[5, 4, 3, 2, 1, 0, 1, 2, 3, 4]

See Parallel map.

R interop

Pool workers can be R processes that run the mizu package, the R binding of the same core. pymizu.r_pool_launcher() spawns them and pymizu.call() describes the task — here R’s built-in mtcars summarized by R itself, read back into polars:

import polars as pl
import pymizu

with pymizu.Pool.create(4, launcher=pymizu.r_pool_launcher()) as pool:
    task = pool.submit(
        pymizu.call(source="aggregate(mpg ~ cyl, data = mtcars, FUN = mean)")
    )
    print(pl.DataFrame(task.collect()))
  • A numpy array arrives in R as a numeric vector, and a polars, pyarrow, or pandas frame as a data frame
  • A vector or data frame arrives back as a numpy array or a pymizu.Frame
  • Arrow data (pyarrow tables and record batch readers, or any __arrow_c_stream__ producer) crosses the same way

The mizu README shows the reverse direction: an R host that drives Python workers with mizu_py_pool_launcher(). See R interop.

Where next

  • Benchmarks — overhead against the stdlib pools, and how pymizu compares with other frameworks.
  • Channels — sending and receiving, batch operations, and what crosses a channel.
  • Task pools — outcomes, nested tasks, growing and observing a pool.
  • Parallel map — scheduling, reproducible randomness, templates, prepared maps.
  • R interop — R peers and workers, what crosses the language boundary, and the dtype matrix.
  • API reference — the generated function and class reference.