Benchmarks
These tables measure task-communication overhead against the stdlib concurrent.futures pools, on an Apple M4 Pro (benchmarks/mizu-stdlib-bench.py).
First, ProcessPoolExecutor: tasks run in separate processes, and every payload crosses as a pickle.
| Benchmark | pymizu | ProcessPoolExecutor | Speedup |
|---|---|---|---|
| Trivial task round trip | 0.6 µs | 94.2 µs | 157x |
| Pipelined throughput, 1 worker | 2,200,000 tasks/s | 19,500 tasks/s | 113x |
| 8 KB vector round trip | 2.0 µs | 100.7 µs | 50x |
| Parallel map of 2,000 ~5 µs tasks, 4 workers | 2.8 ms | 123 ms | 44x |
Next, ThreadPoolExecutor: tasks share one process, so the GIL caps CPU-bound work at one core.
| Benchmark | pymizu | ThreadPoolExecutor | Speedup |
|---|---|---|---|
| Trivial task round trip | 0.6 µs | 9.5 µs | 16x |
| Pipelined throughput, 1 worker | 2,200,000 tasks/s | 405,000 tasks/s | 5.4x |
| 8 KB vector round trip | 2.0 µs | 9.3 µs | 4.7x |
| Parallel map of 2,000 ~5 µs tasks, 4 workers | 2.8 ms | 47 ms | 17x |
How pymizu compares
pymizu runs processes in parallel on one machine and moves data through shared memory instead of pickled copies. The tools below overlap at the edges; they differ in what has to run, how data moves, and what survives a crash.
| Tool | What runs | How data moves | Best for |
|---|---|---|---|
multiprocessingconcurrent.futures |
Stdlib only | Pickle copies per task | Coarse tasks, maximum portability |
joblib |
Reusable worker pool (loky) | Pickle; large arrays memmapped to disk | Coarse parallel loops anywhere |
ipyparallel |
Controller + engines | Pickle over ZeroMQ | Interactive use from Jupyter |
| Ray | Scheduler, object store, dashboard | Zero-copy object store (one node) | Cluster scaling, actors, GPU placement |
| Dask | Scheduler (+ distributed workers) |
Process copies; threads share heap | Out-of-core dataframes and arrays |
| Free-threaded CPython | Threads | Zero copies — shared heap | CPU work, no isolation needed |
concurrent.interpreters |
Stdlib only (3.14+) | Builtins copy, buffers share; no numpy | Process-free isolation |
| pymizu | A library | Shared-memory views, µs handoff | Fine-grained tasks, large arrays, R interop |
Process pools: multiprocessing, concurrent.futures, joblib, ipyparallel
The stdlib pools are the baseline in the tables above.
joblib’s Parallel/delayed is the standard tool for embarrassingly parallel loops over numpy data. Its loky backend is the most similar to pymizu: a reusable process pool that memmaps large arrays to a temp folder instead of pickling them. That avoids the pickle copy, but the array still round-trips through the filesystem, and a task round trip stays in the millisecond range.
ipyparallel brings process pools to Jupyter: a controller process brokers pickled messages over ZeroMQ sockets to engine processes. That generality costs a controller in the loop and a socket copy per message.
multiprocessing.shared_memory gives you raw shared buffers and leaves synchronization, lifetime tracking, and crash cleanup to you. pymizu builds on the same idea — the same pages — and adds the waiting, framing, and peer-death detection.
Distributed frameworks: Ray and Dask
Ray and Dask spread work across machines. On one machine, that design has a cost: a local Ray instance still starts a scheduler, an object store, and a dashboard, and a task round trip costs on the order of a millisecond.
Dask’s distributed scheduler is similar; its threaded scheduler shares memory, but the GIL caps it for Python-level work. Ray’s object store does read numpy arrays without copying on one node. The difference on one machine is overhead and footprint.
Modern CPython: free-threading and subinterpreters
Python 3.14 officially supports a free-threaded build with the GIL removed: threads share one heap and exchange data with no copies at all. If you do not need isolation, it is the best tool for single-language CPU parallelism. The tradeoffs: a crashing thread takes the process down, C extensions must be built and validated for it, and it does not cross into R. pymizu targets the standard GIL build and does not yet support free-threading. The two approaches solve different problems: threads trade isolation for sharing; pymizu keeps the isolation and makes the sharing cheap.
Python 3.14 also added concurrent.interpreters to the stdlib (PEP 734): multiple interpreters in one process, each with its own GIL, so CPU work parallelizes without new processes on the standard build. Objects that cross between interpreters must be shareable: builtin scalars and plain containers copy across, buffer-protocol objects share their memory (synchronization is on you), and everything else — numpy arrays included — is rejected. The isolation is real but not OS-level: a fatal error in one interpreter ends the process, and you build the surrounding machinery (queues, lifetimes, crash handling) yourself. concurrent.interpreters requires Python 3.14+; pymizu runs on 3.11+.
R workers
Only pymizu can run a pool of R workers, exchanging numpy arrays, polars DataFrames, and Arrow tables as shared-memory views (see R interop). No other tool here crosses the language boundary.
- Choose pymizu when the work fits one machine and tasks are fine-grained, data is large, or you need R.
- Choose Ray or Dask when you outgrow one machine.
- Choose
concurrent.interpretersfor stdlib-only isolation when you can require Python 3.14. - Choose free-threading when one language and shared-everything is acceptable.
To reproduce these numbers, run benchmarks/mizu-bench.py (pymizu standalone), benchmarks/mizu-stdlib-bench.py (the comparison above), or benchmarks/crosslang-map-bench.py (the cross-language map comparison). The scripts report timings but assert nothing about them — CI runner timing is too variable — and dated records live in benchmarks/notes.md.