Pool
A shared-memory work-stealing task pool handle (process-private).
Usage
Pool()Create the controller side with Pool.create(), which spawns the workers (python -m pymizu.worker). Other processes join as submitters with Pool.attach(). The pool exists only as long as the process that created it: dropping the handle shuts the pool down as Pool.stop() does, but without waiting for the workers. Handles do not survive fork().
Task callables travel by pickle. With the standard pickle a submitted callable must be an importable reference (as with multiprocessing), and installing cloudpickle lifts that restriction.
Threading: a handle has two single-threaded halves. The producing half (submit(), submit_batch(), map()) belongs to one thread, and a concurrent second producer raises ConcurrentUseError. The waiting half (the blocking collects and stop()) likewise belongs to one thread, which may be a different one from the producer’s. To produce from several threads, attach() one handle per thread (each is its own producer, with its own submission queue), or use PoolExecutor, whose submits are thread-safe. Teardown is the exception: destroy() is safe from any thread.
Attributes
| Name | Description |
|---|---|
| token | The join token for worker/submitter attach. |
Functions
| Name | Description |
|---|---|
| attach() | Attach to a live pool as a submitter, by its join token. |
| collect_all() | Wait until every task is terminal. |
| collect_all_async() | Return an asyncio.Future that resolves when every task finishes. |
| collect_any() | Wait on several tasks. |
| collect_any_async() | Return an asyncio.Future for the first finished task. |
| create() | Create a pool and spawn its worker processes. |
| destroy() | Tear down the pool handle immediately, without waiting. |
| dump() | A read-only debugging snapshot of the whole pool region (dict). |
| info() | A read-only snapshot of the pool’s state (dict). |
| map() | Map fn over x on the pool. |
| map_prepare() | Stage a map for repeated runs and return a PreparedMap. |
| retire() | Ask the worker in slot to exit cleanly. |
| spawn_workers() | Spawn n additional workers and wait for them to join. |
| starmap() | Map fn over x, unpacking each element as call arguments. |
| stats() | Cumulative per-worker and per-submitter counters. |
| stop() | Orderly shutdown (controller only). |
| submit() | Submit fn(*args, **kwargs) as a task; return a Task handle. |
| submit_batch() | Submit one task per zero-arg callable in fns in one call. |
| trace() | Register a hook called as fn(event, id) at each task event. |