Skip to main content

Benchmarks

These tables measure task-communication overhead against the stdlib concurrent.futures pools, on an Apple M4 Pro (benchmarks/mizu-stdlib-bench.py).

First, ProcessPoolExecutor: tasks run in separate processes, and every payload crosses as a pickle.

Benchmark pymizu ProcessPoolExecutor Speedup
Trivial task round trip 0.6 µs 94.2 µs 157x
Pipelined throughput, 1 worker 2,200,000 tasks/s 19,500 tasks/s 113x
8 KB vector round trip 2.0 µs 100.7 µs 50x
Parallel map of 2,000 ~5 µs tasks, 4 workers 2.8 ms 123 ms 44x

Next, ThreadPoolExecutor: tasks share one process, so the GIL caps CPU-bound work at one core.

Benchmark pymizu ThreadPoolExecutor Speedup
Trivial task round trip 0.6 µs 9.5 µs 16x
Pipelined throughput, 1 worker 2,200,000 tasks/s 405,000 tasks/s 5.4x
8 KB vector round trip 2.0 µs 9.3 µs 4.7x
Parallel map of 2,000 ~5 µs tasks, 4 workers 2.8 ms 47 ms 17x

How pymizu compares

pymizu runs processes in parallel on one machine and moves data through shared memory instead of pickled copies. The tools below overlap at the edges; they differ in what has to run, how data moves, and what survives a crash.

Tool What runs How data moves Best for
multiprocessing
concurrent.futures
Stdlib only Pickle copies per task Coarse tasks, maximum portability
joblib Reusable worker pool (loky) Pickle; large arrays memmapped to disk Coarse parallel loops anywhere
ipyparallel Controller + engines Pickle over ZeroMQ Interactive use from Jupyter
Ray Scheduler, object store, dashboard Zero-copy object store (one node) Cluster scaling, actors, GPU placement
Dask Scheduler (+ distributed workers) Process copies; threads share heap Out-of-core dataframes and arrays
Free-threaded CPython Threads Zero copies — shared heap CPU work, no isolation needed
concurrent.interpreters Stdlib only (3.14+) Builtins copy, buffers share; no numpy Process-free isolation
pymizu A library Shared-memory views, µs handoff Fine-grained tasks, large arrays, R interop

Process pools: multiprocessing, concurrent.futures, joblib, ipyparallel

The stdlib pools are the baseline in the tables above.

joblib’s Parallel/delayed is the standard tool for embarrassingly parallel loops over numpy data. Its loky backend is the most similar to pymizu: a reusable process pool that memmaps large arrays to a temp folder instead of pickling them. That avoids the pickle copy, but the array still round-trips through the filesystem, and a task round trip stays in the millisecond range.

ipyparallel brings process pools to Jupyter: a controller process brokers pickled messages over ZeroMQ sockets to engine processes. That generality costs a controller in the loop and a socket copy per message.

multiprocessing.shared_memory gives you raw shared buffers and leaves synchronization, lifetime tracking, and crash cleanup to you. pymizu builds on the same idea — the same pages — and adds the waiting, framing, and peer-death detection.

Distributed frameworks: Ray and Dask

Ray and Dask spread work across machines. On one machine, that design has a cost: a local Ray instance still starts a scheduler, an object store, and a dashboard, and a task round trip costs on the order of a millisecond.

Dask’s distributed scheduler is similar; its threaded scheduler shares memory, but the GIL caps it for Python-level work. Ray’s object store does read numpy arrays without copying on one node. The difference on one machine is overhead and footprint.

Modern CPython: free-threading and subinterpreters

Python 3.14 officially supports a free-threaded build with the GIL removed: threads share one heap and exchange data with no copies at all. If you do not need isolation, it is the best tool for single-language CPU parallelism. The tradeoffs: a crashing thread takes the process down, C extensions must be built and validated for it, and it does not cross into R. pymizu targets the standard GIL build and does not yet support free-threading. The two approaches solve different problems: threads trade isolation for sharing; pymizu keeps the isolation and makes the sharing cheap.

Python 3.14 also added concurrent.interpreters to the stdlib (PEP 734): multiple interpreters in one process, each with its own GIL, so CPU work parallelizes without new processes on the standard build. Objects that cross between interpreters must be shareable: builtin scalars and plain containers copy across, buffer-protocol objects share their memory (synchronization is on you), and everything else — numpy arrays included — is rejected. The isolation is real but not OS-level: a fatal error in one interpreter ends the process, and you build the surrounding machinery (queues, lifetimes, crash handling) yourself. concurrent.interpreters requires Python 3.14+; pymizu runs on 3.11+.

R workers

Only pymizu can run a pool of R workers, exchanging numpy arrays, polars DataFrames, and Arrow tables as shared-memory views (see R interop). No other tool here crosses the language boundary.

  • Choose pymizu when the work fits one machine and tasks are fine-grained, data is large, or you need R.
  • Choose Ray or Dask when you outgrow one machine.
  • Choose concurrent.interpreters for stdlib-only isolation when you can require Python 3.14.
  • Choose free-threading when one language and shared-everything is acceptable.

To reproduce these numbers, run benchmarks/mizu-bench.py (pymizu standalone), benchmarks/mizu-stdlib-bench.py (the comparison above), or benchmarks/crosslang-map-bench.py (the cross-language map comparison). The scripts report timings but assert nothing about them — CI runner timing is too variable — and dated records live in benchmarks/notes.md.