Benchmarking¶

postc ships a benchmark harness that measures its native backends against each other, against the postpyc reference baseline, and against CPython and NumPy. Its purpose is to track the postc-vs-postpyc trend over time (requirement 8): postc began as a lift of postpyc, and each recorded session makes any divergence visible as the backends evolve.

The harness is dev tooling — it lives in the top-level benchmarks/ package, outside the installed compiler. Run it with python -m benchmarks (or make bench).

Quick start¶

python -m benchmarks run                    # run the suite, print a summary
python -m benchmarks run --save             # also record a session to benchmarks.db
python -m benchmarks run --save --label wip # tag the session
python -m benchmarks list                   # list recorded sessions (newest first)
python -m benchmarks compare                # diff the two most recent sessions

run exits non-zero if any runtime produced an incorrect result, so it doubles as a coarse correctness gate in CI.

Commands¶

The database defaults to benchmarks.db in the working directory; override it with --db.

The runtimes¶

Each kernel is run through six contenders:

postc-c and postpyc both lower to C and hand it to cc -O2, so under this suite they compile byte-for-byte the same program (see below) and their times are expected to be equal. postc-llvm also runs at -O2, so it too lands at parity with postc-c. cpython and numpy serve as order-of-magnitude references.

If a runtime's build or setup fails — for example QBE is not installed — it is skipped with a note and the rest of the suite proceeds.

The kernels¶

Four kernels exercise different bottlenecks. Each is a POST @vectorize / @guvectorize source with a {pkg} placeholder for the compiler package, so the same source is templated for both postc and postpyc — the two compile identical programs.

Each kernel is compiled to a <name>_ufunc_loop symbol and driven over a large NumPy batch through the generalized-ufunc ABI (byte strides come straight from NumPy), so the entire inner computation runs in compiled code — the Python-side call overhead is amortized across the whole batch.

matmul is included deliberately: square is memory-bandwidth-bound and gaussian is exp-bound, so neither reflects codegen quality. matmul's nested reduction is where the QBE backend's gap versus cc -O2 actually shows.

Methodology¶

• Round-robin adjacent timing. The compiled backends (postc-c, postc-qbe, postc-llvm, postpyc) are timed with their calls adjacent within each round (40 rounds, after 5 warmup passes). Timing them in adjacent slots means slow-machine drift hits all of them together and cancels in the ratio, so the backend-vs-backend comparison is trustworthy. The interpreted references are timed separately with fewer rounds — only their magnitude matters.
• Min as the headline. The reported number per (kernel, runtime) is the minimum observed time, the measurement least polluted by scheduler noise. Mean and coefficient of variation (CV) are also recorded.
• Correctness-checked. Before timing, each runtime's output is compared against a NumPy reference with np.allclose(rtol=1e-6, atol=1e-12). Incorrect results are flagged in the output and make run exit non-zero.

The summary prints a per-kernel min-time table followed by three geomean ratios: postc-c vs postpyc, postc-qbe vs postc-c, and postc-llvm vs postc-c (in each, <1 means the first is faster).

Current results¶

Measured numbers as of this writing:

• postc-c == postpyc — geomean of the min-time ratio ≈ 1.0x. Both go through cc -O2 on the same generated C, so parity is expected and confirms postc has not regressed against its baseline.
• postc-qbe ≈ 1.15x slower than postc-c on geomean, up to ~1.3x on matmul. QBE's inner reduction is strength-reduced but not unrolled, which is exactly the gap matmul was added to expose.
• postc-llvm ≈ postc-c — geomean ≈ 1.01x, within measurement noise, including matmul. It emits naive IR and lets LLVM's -O2 do the LICM, strength reduction, and unrolling, so it lands where the C compiler does — closing the gap QBE's hand-rolled passes could not.

See QBE backend and LLVM backend for the codegen details and C backend for the baseline.

Cross-backend comparison (with WASM)¶

run tracks postc against postpyc over time. A second command, cross, answers a different question — how do all the backends compare head-to-head, WebAssembly included? — and renders a shareable report.

python -m benchmarks cross                     # colour terminal table
python -m benchmarks cross --html report.html  # + a self-contained HTML report
python -m benchmarks cross --quick             # tiny batches (smoke run)

To put WASM on equal footing with the native backends, cross uses whole-array kernels — a plain POST function that loops over the batch internally (def k(x: Array[Float64], out: Array[Float64]) -> None) — rather than the generalized-ufunc loop the native pipeline uses. (A wasm module is not a linkable .o and has no ufunc loop yet, so the ufunc-driven run suite can't include it.) Every backend runs the same compiled kernel in a single call over the same data:

Any runtime whose toolchain is absent (no wasm-tools, node, wasmtime, or postpyc) is skipped. The headline metric is throughput (elements/second); the report leads with an overall bar chart (each backend's geomean throughput vs C) and closes with a WASM engine head-to-head — wasmtime (Cranelift) vs V8 on the byte-identical module.

Why only two WASM engines?¶

The WASM backend emits memory64 modules (i64 addresses match the frontend's byte-index arithmetic and the all-i64 __pp_array layout). Benchmarking a kernel means pre-populating linear memory with the input array, so a runtime must both support memory64 and let the host write into its memory. Only wasmtime (Cranelift) and V8 (node) qualify here: bun (JavaScriptCore) rejects memory64 outright, the wasmer CLI can't marshal arrays into memory via --invoke (and its Python bindings are unmaintained), and deno is just V8 again. wasmtime's Winch baseline compiler isn't exposed by the Python wheel. Two genuinely different engines (an AOT Cranelift and a JIT V8) is the practical maximum on this setup.

The four kernels are square (bandwidth), sumsq (reduction), gaussian (libm exp), and logistic (a tight scalar inner loop — the codegen-quality probe). Typical observations:

• square — bandwidth-bound, and it is not SIMD: disassembly shows no backend auto-vectorizes here. All of C, LLVM, and postpyc saturate memory bandwidth with a 5-instruction post-increment pointer-walk inner loop; the win is -O2 hoisting the array base/offset out of the loop (LICM), strength-reducing the index, and CSEing x[i]*x[i] to a single load. QBE used to trail badly (~0.23×) — its hand-rolled passes neither CSEd the duplicate x[i] load nor strength-reduced the contiguous loop. Both were added (a local CSE pass, and constant-stride recognition in the strength-reduction pass so it unrolls like the ufunc kernels), lifting QBE to ~0.55×; the residual is QBE-the-tool's instruction selection — it emits ldr [x0]; add x0, 8 rather than a post-increment / scaled-index load, which the postc backend can't influence. This kernel also exposed an LLVM-backend gap — see the box below.
• sumsq — now at parity on every compiled backend (QBE included, once the contiguous loop strength-reduces and unrolls).
• logistic — LLVM tracks C; QBE trails ~1.5× on this scalar-heavy inner loop (no array strength reduction applies), the same class of gap the ufunc suite's matmul shows.
• postpyc — lands at parity with C (both lower to C at -O2), confirming postc has not regressed against the compiler it was lifted from.
• WASM — both engines land at a fraction of native throughput (a young backend plus the memory64 ABI), with V8 and Cranelift usually within ~1.1–1.2× of each other on pure-arithmetic kernels.
• gaussian sits out the WASM race: a libm call lowers to a host import, so its cost belongs to the embedding (node's native Math.exp vs a per-call ctypes trampoline for wasmtime-py). Timing it would measure the host bridge, which tells us nothing about the engine's codegen.

Correctness in cross is a checksum sanity gate against NumPy; the byte-identity oracle remains the test suite.

Tracking over time¶

run --save records a session — a git commit (git rev-parse --short HEAD), a timestamp, and an optional label — plus one row per (kernel, runtime) with min/mean time, CV, run count, and correctness, all in SQLite.

python -m benchmarks run --save --label "post qbe unroll"
python -m benchmarks list
#    12  2026-07-10T14:22:31   13bcf30   post qbe unroll
#    11  2026-07-09T09:05:12   e1972d5
python -m benchmarks compare        # newest vs previous, per (kernel, runtime)

compare reports new.min_time / old.min_time - 1 as a percentage per pair, so a positive number means the newer session is slower. That is the signal to watch: the goal is to keep postc-c pinned to postpyc while closing the postc-qbe gap without regressing anything.