|
libmizu 0.2.0
Lock-free shared-memory IPC at thread speed in C11
|
Lock-free shared-memory IPC at thread speed in C11.
libmizu is for runtimes that parallelize with processes rather than threads. A message round trip drops from tens of microseconds over sockets to < 1 µs, so you can divide work as finely as you would with threads (see the Performance section below).
Channels and work-stealing task pools run over POSIX shared memory (Linux, macOS) or Win32 file mappings (Windows): lock-free single-producer single-consumer rings with batched publication, spin-then-park waiting that stays out of the kernel on the hot path, and OS notification when a peer dies.
Bindings share one wire format, so processes in different languages can exchange data over the same channel. The first-party bindings are mizu for R and pymizu for Python. The Writing a binding section covers other languages.
Lifecycle: experimental. The API is not yet stable and we may make changes in response to feedback.
Representative figures from the C bench suite (make bench), measured on an Apple M4 Pro (macOS, arm64).
| Benchmark | Best observed |
|---|---|
| Channel round trip (nil / 64 B payload) | 0.2–0.3 µs |
| Channel round trip (64 KiB, arena tier) | 4.0 µs |
| Channel round trip (1 MiB, spill tier) | 50 µs |
| Channel batch throughput (64 B, batches of 32) | 52.8 M msgs/s |
| Pool submit + collect (nil payload) | 0.2 µs |
| Pool submit + collect (4 KiB payload) | 0.8 µs |
| Pool task throughput (tiny echo tasks) | ~18 M tasks/s |
See bench/notes.md for the full dated records, including parker wake latency and spill-region costs.
This example makes a channel between two processes with the built-in bytes binding. The bytes binding needs no callbacks, so the example compiles as written. The host:
The peer fills its binding with mizu_binding_bytes() too and attaches with mizu_channel_attach(&ch, token, &binding). It reads the bootstrap payload with mizu_channel_drop() and signals mizu_channel_ready_set().
To move a language's own objects rather than byte buffers, register callbacks on the binding. This is the seam that the Writing a binding section covers:
include/mizu.h documents the full contract for each declaration: ownership, threading, and the mizu_status values that each verb returns. DESIGN.md gives the invariants. The rendered API reference is at shikokuchuo.net/libmizu.
A binding tells libmizu how to turn the language's values into frames and back. The stage and read callbacks from the quickstart do the conversion, and exec runs tasks on pool workers. libmizu does the rest: the rings, waiting and waking, peer-death detection, and work-stealing. The built-in bytes binding, mizu_binding_bytes(), needs no callbacks and moves byte buffers, as the quickstart shows.
Every operation is an exported function. A language with a foreign function interface can call libmizu from the shared library, with no C of its own. Every handle is opaque, so libmizu's internals can change without breaking the binding. Bindings that compile C can add the amalgamation to their own build instead, as described under Building below.
The headers form two tiers:
A third header, src/internal.h, is private. It is never installed, and bindings never include it.
libmizu needs a C11 compiler and a 64-bit platform: the wire formats depend on lock-free 64-bit atomics. On Linux, kernel 5.3 or later is required (for pidfd_open).
The Makefile is the only build system:
On Windows, run tools\build-win.bat instead. The script builds with clang-cl, since MSVC's C11 atomics are experimental, so it needs LLVM and the Visual Studio C++ build tools. GitHub Windows runners have both preinstalled.
Static builds need no defines, because MIZU_API is empty by default. That covers vendored and amalgamated builds, where the first-party packages compile the sources into their own modules. If you build libmizu as a shared library, define MIZU_SHARED. If you compile the library itself on Windows, also define MIZU_BUILDING. The Makefile and build-win.bat set these defines for you.
To generate the amalgamation, run:
If you compile the amalgamation into your own C program, you only need mizu.c. The helpers in mizu_ext.h are inline functions, so your code gets them from the header.
If you want a shared library that other languages can call through an FFI, compile mizu.c and mizu_ext.c together with -DMIZU_SHARED -fvisibility=hidden. mizu_ext.c holds the exported copies of those helpers, which can't live in the same file as the inline versions. The result exports the same symbols as a library built with make.
On Linux, link the amalgamation (or the static library) with -pthread, and with -lrt too on glibc releases before 2.34, where shm_open lives in librt. make install records both in the pkg-config file. Pass the same PREFIX to make and to make install (or let the stamp relink): on macOS the shared library's install name embeds it.
See CONTRIBUTING.md for the repository layout and contributor notes.
MIT. LICENSE.note has the third-party attributions (RngStreams jump matrices, vendored doxygen stylesheet).