zoo-framework 0.10.0
benchmark

zoo-bench report

Run environment

data: Run environment

zoo-bench report

Workload

• composition: one unit of work = JSON serialize + deserialize + string concatenation, repeated until the target duration is reached; the tiers are declared by body_tiers_us in matrix.yaml
• the workload is a stand-in, not a real trace: its shape resembles business actions such as 'take a frame of data -> serialize -> parse -> transform strings', but this repository holds no real trace, so every conclusion is conditional on that

data: Workload

zoo-bench report

Conclusion summary

• Once the framework under test zoo runs a body of about 2700 microseconds or more, its framework overhead drops below 15% of end-to-end; below that tier, the main price of choosing it is the framework overhead itself.
• At concurrency 1 and a 40 microsecond body, the framework under test is 2.11x faster than process_pool (the fastest measured tier at that concurrency).
• At concurrency 1, once the body exceeds about 40 microseconds, apscheduler is no longer faster than the framework under test.
• At concurrency 4, once the body exceeds about 40 microseconds, apscheduler is no longer faster than the framework under test.
• At concurrency 16, once the body exceeds about 40 microseconds, apscheduler is no longer faster than the framework under test.

data: Conclusion summary

Conclusion summary

the rest of this section's details are in the full report

For one conclusion per slide, only the first 2 slides of this section are shown; the remaining 8 slides of per-unit numbers, percentiles and raw samples sit in the same section of the full report, verbatim checkable.

full report: the "Conclusion summary" section of report.md in the same directory

Dimension: Latency percentiles and jitter

Latency percentiles and jitter at concurrency 1

concurrency 1 - end-to-end / concurrency

data: zoo-bench "Latency percentiles and jitter" at concurrency 1 - per-unit numbers and percentiles are in the same section of report.md

zoo-bench report

Dimension: Latency percentiles and jitter

end-to-end / concurrency; percentiles and relative spread are given in the same table - a metric that only gives the median is not acceptable

data: Dimension: Latency percentiles and jitter

Dimension: Framework overhead share

Framework overhead share at concurrency 1

concurrency 1 - framework overhead

data: zoo-bench "Framework overhead share" at concurrency 1 - per-unit numbers and percentiles are in the same section of report.md

zoo-bench report

Dimension: Framework overhead share

overhead = end-to-end / concurrency - the body duration measured within the same run (no cross-run subtraction). cells with -: that concurrency exceeds this machine's parallel capacity; the body's self-reported duration contains scheduling waits from oversubscription and is not comparable with the row's end-to-end per task, so no overhead figure is given (end-to-end and throughput live in their own dimensions)

data: Dimension: Framework overhead share

Dimension: Throughput and concurrency scaling

Throughput and concurrency scaling at concurrency 1

concurrency 1 - throughput

data: zoo-bench "Throughput and concurrency scaling" at concurrency 1 - per-unit numbers and percentiles are in the same section of report.md

zoo-bench report

Dimension: Throughput and concurrency scaling

throughput = concurrency / median end-to-end; whether it keeps rising with concurrency is the scaling behaviour

data: Dimension: Throughput and concurrency scaling

zoo-bench report

Dimension: Overhead attribution

scope: splits the per-task end-to-end into submit side / handoff / body / return and, at the same tier and concurrency, compares each segment against every baseline, answering 'which segment the excess falls on'. It is diagnostic, not the main evidence: it samples a small set of tiers, and does not compare total overhead against the baselines. A negative handoff is not an error: it means the submit call had not returned yet while the body was already running - schemes that count thread creation as submit side behave this way (measured: bare_thread handoff -343 microseconds). **The return segment includes waiting for th

e completion signal** (the last stretch before draining returns);

it shares the main measurement's scope

data: Dimension: Overhead attribution

Dimension: Overhead attribution

the rest of this section's details are in the full report

For one conclusion per slide, only the first 2 slides of this section are shown; the remaining 17 slides of per-unit numbers, percentiles and raw samples sit in the same section of the full report, verbatim checkable.

full report: the "Dimension: Overhead attribution" section of report.md in the same directory

Dimension: The cost of scheduling semantics

Dimension: The cost of scheduling semantics (cont.)

this conclusion was reached on the version under test and its coordinates hold for that version only; after appending a version to matrix.yaml, re-run this probe

data: Dimension: The cost of scheduling semantics

Dimension: Native task line: three execution forms of the same subject

Dimension: Native task line: three execution forms of the same subject (cont.)

tiers R1, R8, R125 (frame bytes 7, 21, 255); concurrencies 1, 4, 16, 64; each cell runs 100 warm-up plus 100 measured rounds, one child process per concurrency (4 in all)

data: Dimension: Native task line: three execution forms of the same subject

tiers where the framework under test is faster

tiers where the framework under test is faster (cont.)

data: tiers where the framework under test is faster

zoo-bench report

Openly published unfavourable data

Tiers where the framework under test is behind (the two sides differ by more than that tier's band). A report missing this part is not acceptable - a report that only shows where it wins gives the reader every reason to distrust it. Tiers whose difference is smaller than the band are not here but in the 'no winner' section: that is the verdict's outcome, not a place to hide things.

data: Openly published unfavourable data

Openly published unfavourable data

Openly published unfavourable data (cont.)

data: Openly published unfavourable data

zoo-bench report

tiers with no winner

Tiers where the two sides cannot be told apart: the difference is smaller than that tier's band. The band comes from the two sides' own across-round spread (stdev / median of the per-task medians) within the same run, so it varies with the sample and the reader can recompute it from the archived per-round samples. It also covers every measured tier and states a reason for each - this section is not a place to hide things: reporting noise-level differences as '1.00x faster' is.

criterion: band = the framework's own across-round relative spread plus the baseline's; a single side's across-round relative spread = stdev / median of that unit's per-round per-task medians

band this run: 1.0% to 33.3% (it varies with each tier's spread).

data: tiers with no winner

tiers with no winner

the rest of this section's details are in the full report

For one conclusion per slide, only the first 2 slides of this section are shown; the remaining 2 slides of per-unit numbers, percentiles and raw samples sit in the same section of the full report, verbatim checkable.

full report: the "tiers with no winner" section of report.md in the same directory

caliber limitations and sources of bias

caliber limitations and sources of bias (cont.)

• scope note (apscheduler): APScheduler's scheduled-job model, including job registration and trigger-decision cost; the library swallows exceptions raised inside jobs, so this tier re-raises them during drain. Caliber note: the library offers no job-completion notification, so the completion signal comes from this tier's Condition, and the cost of one notification per task counts toward its end-to-end
• scope note (asyncio_pool): the asyncio event loop dispatches synchronous bodies (equivalent to loop.run_in_executor), not native coroutine concurrency; the difference from a thread pool is the event loop's own cost

data: caliber limitations and sources of bias

zoo-bench report

bring your own baseline

the baselines of this harness are open: implement the four methods of zoo_bench.adapters.BaseAdapter (setup / submit / drain / teardown), register with zoo_bench.adapters.registry.register, or list your module path in extra_modules of matrix.yaml - no file in this repository needs to change. Your implementation goes through the same equivalence verification (submit N bodies, assert all ran and each exactly once);

failing it means the data is not trusted. We deliberately write this path into the report: a report written by the framework's own maintainer, measuring that framework, gives every reason to suspect a slowed baseline or a chosen caliber; letting anyone submit their own baseline is what makes this report challengeable and falsifiable.

data: bring your own baseline

self-check

self-check (cont.)

• 11, 70013, 70018, 70023, 70028, 70033, 70050, 70067, 70084, 70102, 70167, 70232, 70297], 'all_distinct': True, 'none_equals_parent': True}

data: self-check