import pymizu
with pymizu.Pool.create(4) as pool:
result = pool.map(abs, range(-5, 5)).collect()
result[5, 4, 3, 2, 1, 0, 1, 2, 3, 4]
Pool.map(fn, x) maps fn over x on the pool and returns a Map handle immediately: setup and submission happen at the call, and you collect results from the handle. A map is not a loop over pool.submit(): the function, its constant args=/kwargs=, and the data are written to shared memory once, and one runner task per worker divides up the elements:
import pymizu
with pymizu.Pool.create(4) as pool:
result = pool.map(abs, range(-5, 5)).collect()
result[5, 4, 3, 2, 1, 0, 1, 2, 3, 4]
A map collects exactly once, through the handle:
handle.collect(timeout=None) blocks for the results: a list in input order, or with template= the gathered output area (a view with collect="view"). On timeout expiry the outstanding work is cancelled and you get the TIMEOUT sentinel, not an exception.handle.collect_async(), or await handle, resolves one asyncio.Future with the same results when they arrive, with no blocking or polling. Cancelling the Future only detaches the waiter: the handle can be collected blocking again, and the map is never cancelled. A streaming map (stream=True) can only be collected blocking, because its sliding task window advances in the blocking collect loop.handle.cancel() stops the run: workers halt within about one batch, and a pending collect, blocking or async, ends with CancelledError.Pool.map()’s own timeout= bounds submission only: on its expiry the tasks already in are cancelled and Pool.map() itself returns the TIMEOUT sentinel (no handle). A dropped uncollected handle cancels at garbage collection, so no work is wasted.
Each runner claims batches of elements from a shared counter, and the batch size adapts: a cheap fn runs in large batches at near-zero scheduling cost, while an expensive or uneven one takes smaller batches to keep workers balanced. n_chunks= sets the batch size directly.
A 1-D C-contiguous numpy array of float64, int32, int64, complex128, or uint8 crosses as raw bytes. Each worker views the same array in shared memory and reads its elements directly, so x is never serialized:
import numpy as np
with pymizu.Pool.create(4) as pool:
result = pool.map(np.negative, np.arange(-5.0, 5.0)).collect()
result[np.float64(5.0),
np.float64(4.0),
np.float64(3.0),
np.float64(2.0),
np.float64(1.0),
np.float64(-0.0),
np.float64(-1.0),
np.float64(-2.0),
np.float64(-3.0),
np.float64(-4.0)]
pool.starmap(fn, x) unpacks each element of x as the call’s positional arguments: fn(*element, *args, **kwargs). This is the multiprocessing.Pool.starmap convention:
[1024, 27]
For the multi-iterable shape of concurrent.futures.Executor.map, zip the iterables first: pool.starmap(fn, zip(xs, ys)). Starmap needs Python workers: a pymizu.call spec raises TypeError. Everything else (seeding, templates, streaming, the outcomes) works as in Pool.map.
seed= (an int or bytes) gives every element its own deterministic stream of the stdlib random module. Element i is seeded from SHA-256 of the seed and i. Results are identical for any chunking, worker count, or steal order:
import random
def draw(i):
return random.random()
with pymizu.Pool.create(4) as pool:
a = pool.map(draw, range(100), seed=123).collect()
b = pool.map(draw, range(100), seed=123, n_chunks=10).collect()
print(a == b)True
Pass seed=(seed, offset) to shift every element’s stream by offset positions, for maps split across runs or processes.
seed= covers the stdlib random module only. To draw from numpy, call pymizu.current_rng() inside the task: the element’s own numpy.random.Generator, derived from the same seed and created once per element:
def draw_np(i):
return pymizu.current_rng().random()
with pymizu.Pool.create(4) as pool:
a = pool.map(draw_np, range(100), seed=123).collect()
b = pool.map(draw_np, range(100), seed=123, n_chunks=10).collect()
print(a == b)True
The legacy np.random.* functions draw from the worker’s shared global RandomState; a seeded map does not make them deterministic, and other RNG libraries (torch, jax) are out of scope. Also, a seeded map element must not submit and collect nested tasks: while a worker waits for nested results it runs other work — possibly another map’s batches mid-element — which can wipe the element’s current_rng() stash. The stdlib streams survive.
A template= example buffer (for example np.empty(m, dtype=...)) declares that every fn result is m values of that dtype. Workers write results directly into a shared n x m output area, so each result crosses the process boundary once, with no serialization. Each result must be a matching buffer; when m == 1, a plain Python scalar also works:
import math
with pymizu.Pool.create(4) as pool:
result = pool.map(math.log1p, [1.0, 2.0, 3.0], template=np.empty(1)).collect()
resultarray([0.69314718, 1.09861229, 1.38629436])
With a template, collect="copy" (the default) returns the output as one numpy array of shape (n, m) — (n,) when m == 1 — or a memoryview when numpy is not installed. collect="view" returns the shared output area itself as a read-only zero-copy view, released when the view is garbage-collected. The view exports __arrow_c_array__, so an Arrow consumer such as pyarrow or polars wraps it without numpy. With numpy, reach the view through the result’s .base (arr.base when m == 1, one link deeper when m > 1); without numpy, it is mv.obj on the memoryview.
With stream=True, the map does not write the whole of x to shared memory; it streams slices to workers as they take work. Slices travel as ordinary chunk tasks under a sliding submit/collect window: at most min(n_chunks, 2 * live workers, free result slots) tasks stay outstanding, so shared-memory use is bounded by window x slice instead of sizeof(x). The chunk count defaults to min(len(x), 32 * live workers); n_chunks= overrides it.
with pymizu.Pool.create(4) as pool:
result = pool.map(abs, range(-5, 5), stream=True).collect()
result[5, 4, 3, 2, 1, 0, 1, 2, 3, 4]
Everything else (ordering, template= and collect=, seed= reproducibility, the errors) works as in the non-streaming map. Fixed chunks have two costs: the adaptive batch sizing is lost (raise n_chunks= to reduce skew), and slices are serialized like ordinary task payloads, so each chunk pays a submit’s serialization cost instead of the zero-copy path a raw array gets. A streaming map with Python workers shares one description of the map with every worker, each read at most once. A pymizu.call spec as fn streams too, over workers of any language. Each chunk in the window writes its own small shared-memory region — the slice plus the task that runs it, one runner per chunk — removed as soon as the chunk completes, so shared-memory use stays bounded by the window plus a small per-worker cache rather than sizeof(x). Each chunk also writes its results separately, so they cannot combine into one view: collect="view" raises TypeError on a streaming spec map, as does preparing one with pool.map_prepare().
pool.map_prepare() stages a map once for repeated runs: the data is written to shared memory once, and each worker sets up the map once. Each m.run() then costs only the task submissions and the collection, reuses each worker’s cached setup, and returns the run’s handle:
with pymizu.Pool.create(4) as pool:
with pool.map_prepare(abs, range(-5, 5)) as m:
print(m.run().collect())
print(m.run().collect())[5, 4, 3, 2, 1, 0, 1, 2, 3, 4]
[5, 4, 3, 2, 1, 0, 1, 2, 3, 4]
m.run(x=...) replaces the data for that and later runs: a raw-array replacement of the same dtype and length copies over the old data in place; anything else is written out fresh. A streaming prepared map keeps x on the submitter, so a replacement of any shape re-slices. Only a length change under template= uses fresh shared memory, as does the run after one collected with collect="view" (the previous output area belongs to the returned view). m.run(seed=...) reseeds the run’s element streams and m.run(collect=...) chooses the result form — both per-run options, because the seed travels with the run’s tasks rather than the staged data.
Maps nest: a task can start a map on its worker’s own pool handle, via pymizu.current_pool(). The runner submissions land on the worker’s own queue, and while the worker waits it runs its own runners while idle workers steal the rest — a nested map cannot deadlock the pool.
def nested_total(parts):
return sum(pymizu.current_pool().map(sum, parts).collect())
parts = [range(0, 250), range(250, 500), range(500, 750), range(750, 1000)]
with pymizu.Pool.create(4) as pool:
total = pool.submit(nested_total, parts).collect()
total499500
(This example needs cloudpickle: nested_total is not an importable reference.)
A worker’s first nested map claims a submitter slot. The default max_submitters = 8 holds one slot for the creating process, so at most 7 workers can nest concurrently; raise max_submitters for wider fan-outs. Do not nest inside the element of a seeded map — while a worker waits for nested results it runs other work, possibly another map’s batches mid-element, which can wipe the element’s current_rng() stash. See Reproducible randomness.
fn re-raises as pymizu.TaskError carrying the failing element’s 0-based index. The map fails fast: the workers stop within about one batch.lost (0-based half-open (lo, hi) pairs, possibly wider than what was lost).timeout= expiry the outstanding work is cancelled and the pymizu.TIMEOUT sentinel is returned, not raised.A map started inside a task runs on the worker’s own pool handle via pymizu.current_pool(); a pymizu.call spec as fn is the cross-language map (see R interop).