96 lines
3.9 KiB
Markdown
96 lines
3.9 KiB
Markdown
# About benches
|
|
|
|
The `benches` directory contains benchmarks that test scryer-prolog performance.
|
|
|
|
Benchmarks are run via two harnesses:
|
|
|
|
* `criterion` - criterion performs statistical analysis of benchmark runs and is
|
|
great for benchmarking locally.
|
|
* `iai-callgrind` - this runs the benchmark with callgrind, which is able to
|
|
precisely track the number of instructions executed during the run. This is
|
|
especially helpful in a public CI runner context where neighboring VMs can
|
|
cause a very high wall time variance. While instructions executed is only
|
|
correlated with the desired metric (wall time), this is a good tradeoff for CI
|
|
where that metric is unreliable.
|
|
|
|
Run benchmarks with the following commands:
|
|
|
|
```
|
|
cargo bench --bench run_criterion
|
|
|
|
# run a particular criterion benchmark
|
|
cargo bench --bench run_criterion -- <benchmark_name>
|
|
|
|
# run in profiling mode which outputs flamegraphs. Set profile time in seconds:
|
|
cargo bench --bench run_criterion -- --profile-time <time>
|
|
|
|
# to run iai, you need valgrind installed and to install iai-callgrind-runner
|
|
# at the same version as is in Cargo.toml:
|
|
cargo install iai-callgrind-runner --version 0.7.3
|
|
|
|
cargo bench --bench run_iai
|
|
```
|
|
|
|
For consistency, both runners -- `run_iai.rs` and `run_criterion.rs` -- import
|
|
the same setup code from `setup.rs`.
|
|
|
|
## Setup
|
|
|
|
`setup.rs` contains the setup code to run benchmarks. `fn prolog_benches()` at
|
|
the top of the file is where the benchmarks are defined.
|
|
|
|
Benchmarks are organized around running queries against a prolog module file.
|
|
Before a benchmark starts, `benchmark.setup()` is called which reads the module
|
|
file and initializes a new `scryer_prolog::machine::Machine`.
|
|
|
|
Each benchmark measurement is done by running a query against the machine. In
|
|
the case of criterion each query is run many times, in the case of iai it's run
|
|
once.
|
|
|
|
## Adding benchmarks
|
|
|
|
This design is meant to support defining lots of benchmarks.
|
|
|
|
To add a new benchmark:
|
|
|
|
* Add a new file `benches/[module].pl` that contains setup prolog code. Import
|
|
libraries, define predicates, etc.
|
|
* Add a new section in `setup.rs::prolog_benchmarks()` that refers to to the
|
|
file and write a query to be benchmarked.
|
|
* If the query mutates the machine, then use `Strategy::Fresh` so the criterion
|
|
benchmark will recreate a new machine for each benchmark run, otherwise use
|
|
`Strategy::Reuse` which has lower overhead. (This is not used by the iai
|
|
benchmark because it only runs once anyway.)
|
|
|
|
Some tips:
|
|
|
|
* The goal of benchmarking is to know if a library or engine change improved
|
|
performance or not.
|
|
* Once a benchmark is defined and named, avoid changing it's definition. If a
|
|
benchmark needs to change to be more useful, give the new definition a new
|
|
name instead. This will prevent charts from showing wild changes in
|
|
performance just because the definition changed (see previous).
|
|
* Aim for queries to execute in less than 0.5s realtime. Longer runtimes make it
|
|
easier for humans to see big differences, but benchmarks either run 10x slower
|
|
(iai) or execute repeatedly to attain statistical significance (criterion) and
|
|
in both cases benchmarking queries that take longer than about 0.5s are
|
|
cumbersome to run.
|
|
* Consider that the library runtime actually parses the text output of the top
|
|
level. So don't use custom outputs or it will fail to parse. Also keep the
|
|
output small so it doesn't just benchmark the output parsing code.
|
|
* DO test the output of the benchmark run, we don't want to count broken
|
|
benchmarks.
|
|
|
|
## CI
|
|
|
|
Both benchmark harnesses are run in `.github/workflows/ci.yaml` in the `report`
|
|
job, and the results are published as build artifacts.
|
|
|
|
## Todo
|
|
|
|
- [ ] Currently, the execution time to load a module is not benchmarked. It
|
|
would be nice to have at least one benchmark for loading a module (probably a
|
|
big one).
|
|
- [ ] Write a new action that downloads the test and benchmark results
|
|
artifacts, plots them over time, and publishes a report to github pages.
|