Testing¶

postc's tests are organized as a pyramid, and every backend is held to two differential oracles: a compiled POST function must match CPython, and where the arithmetic is exact the C backend and every other available backend (QBE, LLVM) must agree byte-for-byte. Because POST Python is a typed subset of Python, the reference implementation of any POST function is the function itself, run under CPython — no golden values to hand-maintain.

On top of those sits a third, borrowed layer: LPython's test corpus, whose programs assert their own results and must hold both interpreted and compiled.

The test pyramid¶

Tests live in three tiers under tests/, fastest first:

The marker is applied automatically from the tier directory. tests/conftest.py inspects each collected item's path in pytest_collection_modifyitems and adds unit, integration, or e2e accordingly — you never decorate a test by hand. The three markers are declared in pyproject.toml under [tool.pytest.ini_options], and testpaths covers both tests and src (docstring tests).

make test                          # everything (pytest)
uv run pytest -m unit              # one tier by marker
uv run pytest -m "integration or e2e"
uv run pytest tests/a_unit/        # or by directory

The two oracles¶

Both oracles are provided by postc.testing.differential. Given POST source, a function name, and an argument tuple, it runs the function under CPython and under each backend and returns every result keyed by runtime:

from postc.testing import differential

SRC = """
from postyp import Int64
def m(a: Int64, b: Int64) -> Int64:
    return a % b
"""

def test_mod(tmp_path):
    results = differential(SRC, "m", (-7, 3), tmp_path)
    # results == {"cpython": 2, "c": 2, "qbe": 2}
    reference = results.pop("cpython")
    for name, value in results.items():
        assert value == reference, f"{name}={value} != cpython={reference}"

• Semantic oracle — each backend's result must equal results["cpython"]. This is the primary correctness check: the compiled code must mean the same thing as the Python it was compiled from.
• Parity oracle — the backends must agree with each other (byte-identical where the arithmetic is exact). This catches a backend that is self-consistently wrong.

The c_e2e golden-program corpus¶

tests/c_e2e/programs/ holds whole POST modules; tests/c_e2e/test_programs.py drives them. A PROBES table maps each program file to the (func, args) calls that exercise it, and every probe is run through CPython, C, and every other available backend — QBE when its binary is on PATH, LLVM when llvmlite is installed:

PROBES = {
    "floor_and_mod.py": [
        ("modulo", (-7, 3)), ("modulo", (7, -3)),
        ("fdiv",   (-7, 2)), ("fdiv",   (7, -2)),
    ],
    "int_pow.py": [("ipow", (2, 10)), ("ipow", (5, 3))],
    ...
}

The load-bearing probes are the mixed-sign cases — negative floor-division, modulo, and integer ** — exactly where a naive C lowering diverges from Python semantics. Float comparisons use math.isclose (rel_tol=abs_tol=1e-12); integers and bools must match exactly.

The ratchet¶

The corpus is designed to grow ahead of the compiler (the p2w — "parked to working" — pattern). WORKING = set(PROBES): a program with probes must pass on every available backend, and a BuildError there is a hard failure. Programs not yet supported are xfailed rather than deleted, so the corpus documents the target surface before the compiler reaches it. A second ratchet, test_program_compiles, parametrizes every *.py in programs/ (including the array/ufunc kernels that aren't scalar-probeable, like vec_square.py and gu_dot.py) across every available backend and asserts each one still compiles.

Adding a golden program¶

1. Drop a POST module into tests/c_e2e/programs/, e.g. saturating.py. It is picked up automatically by test_program_compiles and must compile on every available backend.
2. To assert its runtime behavior, add scalar probes to PROBES:

PROBES["saturating.py"] = [("clampi", (300,)), ("clampi", (-5,))]

That makes it a WORKING program: each probe must agree across CPython, C, and every other available backend (QBE, LLVM). No expected values are written down — CPython supplies them.

The shared scalar corpus¶

tests/corpus.py holds one list of scalar cases — a POST function named k and the inputs to run it on — and every backend runs all of it, against CPython. Two harnesses drive the same list: test_corpus_native.py goes through ctypes for C, LLVM and QBE, and test_corpus_wasm.py goes through wasm-tools and wasmtime for the WASM backend, which emits a .wasm module rather than a linkable object and so cannot use the ctypes path.

It exists because the three backends used to keep private corpora, and none was a superset of the others. Sub-word arithmetic ran only against QBE. while/for reductions ran only against LLVM. The inputs that catch a broken fmod lived only in the WASM file — which is precisely why QBE's mod_f case carried (7.5, 2.0) and (-7.0, 2.0), two values that are exact under the wrong formula and therefore proved nothing. Each backend was tested against the union of nobody's mistakes but its own.

Two rules keep it sound:

• diverges names, in a sentence, why a case does not match CPython — POST's int ** negative int is 0 where Python promotes to float and gives 0.5; a case that relies on integer wraparound is pinned to release, because postyp's arithmetic does not wrap. A case may only skip the CPython oracle by saying why, and the set of such cases is asserted, so a new silent divergence fails the suite.
• WASM_GAPS names what the WASM backend cannot lower, and a test asserts each gap still raises. If a gap closes, that test fails and tells you to delete the entry. A skip nobody has to look at is a coverage hole that reads as a pass.

The LPython corpus¶

tests/c_e2e/programs/lpython/ holds 40 of LPython's integration tests, translated from LPython's dialect into POST Python by tools/harvest_lpython.py and carrying LPython's own assertions unchanged. They are BSD-3-Clause (© 2019-2020 Triad National Security, LLC); the licence sits beside them and each file records that it was modified.

test_lpython_corpus.py runs each program two ways, and both must hold:

• under CPython, because POST Python is a subset of Python — the module is imported and its entry points called;
• on every backend, in a subprocess, because a violated assertion aborts the process and would take the test runner down with it rather than fail a test.

Each program names its entry points in its docstring (Entry points: test_loop_01, test_loop_02) — LPython drives its assertions from module-level calls, which POST has no place for, so the harvester lifts them out.

These programs are a different kind of oracle from the two above, and they earn their place by what they found. Between them they exposed eleven compiler defects that the whole existing suite had missed: for i in range(10, 0, -1) running zero times, a loop's else clause being dropped from the program entirely, round(0.5) returning 1.0, a ternary evaluating the arm it did not take (recursing forever), a libm call passing a Float32 to a symbol that takes a double, and fib written with a returning if/else failing to build at all on two backends.

Nearly every one survived a green parity oracle: backends agreeing with each other is not correctness, and three backends can — and did — agree on the same wrong answer. Programs that assert their own results, in shapes nobody on the project would have thought to write, are what find these.

To re-run the harvest (it needs an LPython checkout in sandbox/lpython/):

python tools/harvest_lpython.py            # classify, write nothing
python tools/harvest_lpython.py --write    # emit the ones that build and run clean

It translates mechanically and hands the result to the compiler: postc's own diagnostics say why each rejected program is out of reach, so there is no second definition of "what POST supports" to keep in sync.

Build modes in the suite¶

The suite runs in debug, by taking the default. A suite that does not exercise the shipped default is not testing the product — and here it is not a neutral choice, it is required by the one true oracle in the repo. The LPython corpus is driven almost entirely by assert, and its compiled driver checks nothing but the exit code. In a release build those assertions are elided (spec §5.1), so all 40 programs would compile to functions that do nothing observable and pass vacuously — the exact failure mode that made fixing assert worthwhile in the first place. test_the_corpus_is_built_in_a_mode_that_keeps_its_assertions makes a future flip of that default fail loudly rather than silently hollow the corpus out.

Some tests pin mode=BuildMode.RELEASE explicitly, and each says why in its docstring. There are two legitimate reasons to:

• The test asserts that arithmetic wraps. Wrapping is the release contract; a debug build diagnoses the same arithmetic as overflow and aborts. Both halves matter, so each of these has a debug twin in test_integer_overflow.py that asserts the abort, out of process.
• The test is about codegen. The QBE strength-reduction and unroll tests ask whether the shipped codegen fires, and the codegen that ships — and that the published benchmark numbers measure — is release. (They also assert that the passes survive a debug build, which they only do because the bounds checks are hoisted out of the loop body.)

Two things to know when writing a test that touches the checks:

• A failed check aborts. Compiled POST has no exceptions, so a violated assertion, an out-of-bounds index and an integer overflow all report on stderr and abort(). That takes the pytest worker down with it, so the failure cases must run in a subprocess and assert on the exit status.
• dlopen caches by path. The build mode is part of the artifact filename for exactly this reason: building the same function in debug and then in release into one directory would otherwise hand back the debug library both times — and pass, because the two answers usually agree.

Prerequisites and CI¶

End-to-end tests need a C compiler; the suite discovers cc/clang/gcc and skips (needs_cc) when none is found. QBE probes and QBE compile-checks are included only when the qbe binary is present, so a machine without QBE runs a correct C-only subset rather than failing. See Benchmarking for the performance side of the same corpus, and the IR reference and C / QBE backend pages for what the oracles are validating.