zoo-framework 0.9.0
benchmark

Run environment

Run environment (cont.)

dist_version (the distribution metadata) is the source of truth; module_version is only a footnote

data: Run environment

zoo-bench report

Measurement methodology

• the body duration is instrumented within the same run: the body measures itself internally and reports back; framework overhead = end-to-end - it. No cross-run subtraction - two runs differ in state (caches, frequencies, scheduling noise), so subtracting introduces systematic bias.
• warmups are excluded from statistics: each unit first runs 3 warmup rounds that are discarded, then only the following 30 measured rounds are counted; the sample count equals 'measured rounds x concurrency'.
• absolute durations are not comparable across runs: they only show the order of magnitude; across versions/machines read the relative quantities within the same run (overhead share, multiples against the baselines), which are insensitive to overall speed.

data: Measurement methodology

Conclusion summary

Conclusion summary (cont.)

• At concurrency 64, once the body exceeds about 40 microseconds, apscheduler is no longer faster than the framework under test.
• At concurrency 1, within the measured tiers asyncio_pool is always faster than the framework under test - tiers like these are exactly the openly published unfavourable data of this report.
• At concurrency 4, within the measured tiers asyncio_pool is always faster than the framework under test - tiers like these are exactly the openly published unfavourable data of this report.
• At concurrency 16, within the measured tiers asyncio_pool is always faster than the framework under test - tiers like these are exactly the openly published unfavourable data of this report.

data: Conclusion summary

Dimension: Latency percentiles and jitter

Latency percentiles and jitter at concurrency 1 (cont.)

concurrency 1 - end-to-end / concurrency

data: zoo-bench "Latency percentiles and jitter" at concurrency 1 - per-unit numbers and percentiles are in the same section of report.md

Dimension: Latency percentiles and jitter

Dimension: Latency percentiles and jitter (cont.)

per-unit values (percentiles, spread, raw per-round samples) are in 'Appendix: all values' at the end.

data: Dimension: Latency percentiles and jitter

Dimension: Framework overhead share

Framework overhead share at concurrency 1 (cont.)

concurrency 1 - framework overhead

data: zoo-bench "Framework overhead share" at concurrency 1 - per-unit numbers and percentiles are in the same section of report.md

Dimension: Framework overhead share

Dimension: Framework overhead share (cont.)

per-unit values (percentiles, spread, raw per-round samples) are in 'Appendix: all values' at the end.

data: Dimension: Framework overhead share

Dimension: Throughput and concurrency scaling

Throughput and concurrency scaling at concurrency 1 (cont.)

concurrency 1 - throughput

data: zoo-bench "Throughput and concurrency scaling" at concurrency 1 - per-unit numbers and percentiles are in the same section of report.md

Dimension: Throughput and concurrency scaling

Dimension: Throughput and concurrency scaling (cont.)

per-unit values (percentiles, spread, raw per-round samples) are in 'Appendix: all values' at the end.

data: Dimension: Throughput and concurrency scaling

Dimension: Overhead attribution

Dimension: Overhead attribution (cont.)

status: ok

this dimension is a diagnosis, not the main evidence: the sampled tiers and concurrencies are a fixed small batch, not the whole matrix. The concurrency ceiling is the machine's parallel capacity - beyond it, what is measured is contention;

sampled tiers (microseconds): 40, 10000; concurrency: [1, 4]; this machine's parallel capacity: 4

data: Dimension: Overhead attribution

zoo-bench report

Dimension: The cost of scheduling semantics

scope: this dimension uses an internal switch comparison (same workload, semantics all off vs. enabled one by one) and does not compare across schemes - a bare writeup has no counterpart for priority / timeout / retry, so it can only answer 'what is this framework's semantics worth', not 'how much more expensive than a bare writeup'

status: ok (the conclusion was reached on the 'current' drive-surface generation installed during this measurement)

data: Dimension: The cost of scheduling semantics

zoo-bench report

Dimension: Native task line: three execution forms of the same subject

scope: this line is measured on its own: it enters neither the main matrix's pivot tables nor its crossings, and its numbers must not be read against the baseline adapters listed there

verdict: not_applicable

the framework under test has no such capability (symbols not found: adaptive_params, dual_arm_worker, native_params, native_task_worker) - this line does not apply to this version, and that is not a failure

equivalence gate: not run (the probe did not reach it)

loaded task modbus_rtu.parse_response - Modbus RTU response-frame parsing: the input is a bare RTU ADU response frame (no transport framing), and the tier's frame bytes are its length

data: Dimension: Native task line: three execution forms of the same subject

zoo-bench report

tiers where the framework under test is faster

Tiers where the framework under test is ahead (the two sides differ by more than that tier's band). This comes from the same within-run comparison as the openly published unfavourable data - one side takes the faster tiers, the other the slower ones, so both cover the same ground and neither can pick its own tiers. This section being empty is not a failure: it should be empty when nothing is distinguishable or when the framework loses everywhere.

data: tiers where the framework under test is faster

Openly published unfavourable data

Openly published unfavourable data (cont.)

data: Openly published unfavourable data

Openly published unfavourable data

Openly published unfavourable data (cont.)

data: Openly published unfavourable data

zoo-bench report

tiers with no winner

Tiers where the two sides cannot be told apart: the difference is smaller than that tier's band. The band comes from the two sides' own across-round spread (stdev / median of the per-task medians) within the same run, so it varies with the sample and the reader can recompute it from the archived per-round samples. It also covers every measured tier and states a reason for each - this section is not a place to hide things: reporting noise-level differences as '1.00x faster' is.

criterion: band = the framework's own across-round relative spread plus the baseline's; a single side's across-round relative spread = stdev / median of that unit's per-round per-task medians

band this run: 1.4% to 30.1% (it varies with each tier's spread).

data: tiers with no winner

tiers with no winner

the rest of this section's details are in the full report

For one conclusion per slide, only the first 2 slides of this section are shown; the remaining 1 slides of per-unit numbers, percentiles and raw samples sit in the same section of the full report, verbatim checkable.

full report: the "tiers with no winner" section of report.md in the same directory

caliber limitations and sources of bias

caliber limitations and sources of bias (cont.)

• scope note (apscheduler): APScheduler's scheduled-job model, including job registration and trigger-decision cost; the library swallows exceptions raised inside jobs, so this tier re-raises them during drain. Caliber note: the library offers no job-completion notification, so the completion signal comes from this tier's Condition, and the cost of one notification per task counts toward its end-to-end
• scope note (asyncio_pool): the asyncio event loop dispatches synchronous bodies (equivalent to loop.run_in_executor), not native coroutine concurrency; the difference from a thread pool is the event loop's own cost

data: caliber limitations and sources of bias

zoo-bench report

bring your own baseline

the baselines of this harness are open: implement the four methods of zoo_bench.adapters.BaseAdapter (setup / submit / drain / teardown), register with zoo_bench.adapters.registry.register, or list your module path in extra_modules of matrix.yaml - no file in this repository needs to change. Your implementation goes through the same equivalence verification (submit N bodies, assert all ran and each exactly once);

failing it means the data is not trusted. We deliberately write this path into the report: a report written by the framework's own maintainer, measuring that framework, gives every reason to suspect a slowed baseline or a chosen caliber; letting anyone submit their own baseline is what makes this report challengeable and falsifiable.

data: bring your own baseline

self-check

self-check (cont.)

• 25, 56527, 56532, 56537, 56542, 56547, 56564, 56581, 56598, 56615, 56664, 56713, 56758], 'all_distinct': True, 'none_equals_parent': True}

data: self-check