Add benchmarks using library interface

benches/README.md

@@ -0,0 +1,95 @@
+# About benches
+
+The `benches` directory contains benchmarks that test scryer-prolog performance.
+
+Benchmarks are run via two harnesses:
+
+* `iai-callgrind` - this runs the benchmark with callgrind, which is able to
+  precisely track the number of instructions executed during the run. This is
+  especially helpful in a public CI runner context where neighboring VMs can
+  cause a very high wall time variance. This means that it doesn't track wall
+  time which is what we really care about, but it is a good tradeoff for CI
+  where tracking runtime is unreliable.
+* `criterion` - criterion performs statistical analysis of benchmark runs and is
+  great for benchmarking locally.
+
+Run them using the following commands:
+
+```
+cargo bench --bench run_criterion
+
+# to run iai, you need valgrind installed and to install iai-callgrind-runner
+# at the same version as is in Cargo.toml:
+cargo install iai-callgrind-runner --version 0.7.3
+
+cargo bench --bench run_iai
+```
+
+For consistency, both runners -- `run_iai.rs` and `run_criterion.rs` -- import
+the same setup code from `benches.rs`.
+
+## Setup
+
+`setup.rs` contains the setup code for the actual benchmarks, which are run
+using the `Benches` struct. `fn benches()` at the top of the file is where the
+benchmarks are defined.
+
+Benchmarks are organized around running queries against one prolog module file.
+Before any runs start, `Benches::new()` reads the module files and initializes a
+new `scryer_prolog::machine::Machine` for each file; multiple queries can be
+declared to be benchmarked in the context of that module/machine instance.
+
+Each benchmark measurement is done by running a query against the machine. In
+the case of criterion each query is run many times, in the case of iai it's run
+once.
+
+## Adding benchmarks
+
+This design is meant to suppoort defining lots of benchmarks.
+
+To add a new benchmark:
+
+* Add a new file `benches/[module].pl` that contains setup code. Import
+  libraries, define predicates, etc.
+* Add a new section in `setup.rs::benches()` that refers to it and add some
+  benchmarks.
+
+Some tips:
+
+* The goal of benchmarking is to know if a library or engine change improved
+  performance or not.
+* Once a benchmark is defined and named, don't change it's definition. If a
+  benchmark needs to change to be more useful, give the new definition a new
+  name. This will prevent charts from showing wild changes in performance just
+  because the definition changed (see previous).
+* Aim for queries to execute in about 0.1-0.5s realtime. Longer runtimes make it
+  easier for humans to see big differences, but benchmarks either run 10x slower
+  (iai) or execute repeatedly to attain statistical significance (criterion) and
+  in both cases queries that take 5+ seconds quickly become unweildly.
+* Consider that the library runtime actually parses the text output of the top
+  level. So keep the output small and don't use custom outputs or it will fail
+  to parse.
+* DO test the output of the benchmark run, we don't want to count broken
+  benchmarks.
+* Because a query may run against the same machine multiple times, don't
+  permanently mutate the state of the engine with the query since that will
+  taint subsequent runs. (Benchmarking assertz et al is desirable, but will
+  require some adjustments to how the machine is set up for runs.)
+
+## CI
+
+Both benchmark harnesses are run in `.github/workflows/ci.yaml` in the `report`
+job, and the results are published as build artifacts.
+
+A future action may consume the build artifacts and publish a report using the
+results.
+
+## Todo
+
+- [ ] Currently, the execution time to load a module is not benchmarked. It
+  would be nice to have at least one benchmark for loading a module (probably a
+  big one).
+- [ ] Adjust the benchmark execution strategy to allow queries to modify the
+  engine state (`assertz` etc).
+- [ ] Write a new action that consumes the test and benchmark results and plots
+  them over time and publishes a report (github pages?).

benches/edges.pl

@@ -0,0 +1,130 @@
+:- use_module(library(clpb)).
+:- use_module(library(assoc)).
+:- use_module(library(lists)).
+:- use_module(library(pairs)).
+
+/* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
+    Contiguous United States and DC as they appear in SGB:
+    http://www-cs-faculty.stanford.edu/~uno/sgb.html
+- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - */
+
+edge(al, fl).
+edge(al, ga).
+edge(al, ms).
+edge(al, tn).
+edge(ar, la).
+edge(ar, mo).
+edge(ar, ms).
+edge(ar, ok).
+edge(ar, tn).
+edge(ar, tx).
+edge(az, ca).
+edge(az, nm).
+edge(az, nv).
+edge(az, ut).
+edge(ca, nv).
+edge(ca, or).
+edge(co, ks).
+edge(co, ne).
+edge(co, nm).
+edge(co, ok).
+edge(co, ut).
+edge(co, wy).
+edge(ct, ma).
+edge(ct, ny).
+edge(ct, ri).
+edge(dc, md).
+edge(dc, va).
+edge(de, md).
+edge(de, nj).
+edge(de, pa).
+edge(fl, ga).
+edge(ga, nc).
+edge(ga, sc).
+edge(ga, tn).
+edge(ia, il).
+edge(ia, mn).
+edge(ia, mo).
+edge(ia, ne).
+edge(ia, sd).
+edge(ia, wi).
+edge(id, mt).
+edge(id, nv).
+edge(id, or).
+edge(id, ut).
+edge(id, wa).
+edge(id, wy).
+edge(il, in).
+edge(il, ky).
+edge(il, mo).
+edge(il, wi).
+edge(in, ky).
+edge(in, mi).
+edge(in, oh).
+edge(ks, mo).
+edge(ks, ne).
+edge(ks, ok).
+edge(ky, mo).
+edge(ky, oh).
+edge(ky, tn).
+edge(ky, va).
+edge(ky, wv).
+edge(la, ms).
+edge(la, tx).
+edge(ma, nh).
+edge(ma, ny).
+edge(ma, ri).
+edge(ma, vt).
+edge(md, pa).
+edge(md, va).
+edge(md, wv).
+edge(me, nh).
+edge(mi, oh).
+edge(mi, wi).
+edge(mn, nd).
+edge(mn, sd).
+edge(mn, wi).
+edge(mo, ne).
+edge(mo, ok).
+edge(mo, tn).
+edge(ms, tn).
+edge(mt, nd).
+edge(mt, sd).
+edge(mt, wy).
+edge(nc, sc).
+edge(nc, tn).
+edge(nc, va).
+edge(nd, sd).
+edge(ne, sd).
+edge(ne, wy).
+edge(nh, vt).
+edge(nj, ny).
+edge(nj, pa).
+edge(nm, ok).
+edge(nm, tx).
+edge(nv, or).
+edge(nv, ut).
+edge(ny, pa).
+edge(ny, vt).
+edge(oh, pa).
+edge(oh, wv).
+edge(ok, tx).
+edge(or, wa).
+edge(pa, wv).
+edge(sd, wy).
+edge(tn, va).
+edge(ut, wy).
+edge(va, wv).
+
+independent_set(G, *(NBs)) :-
+        findall(U-V, (edge(U, V),G@<U), Edges),
+        setof(U, V^(member(U-V, Edges);member(V-U, Edges)), Nodes),
+        pairs_keys_values(Pairs, Nodes, _),
+        list_to_assoc(Pairs, Assoc),
+        maplist(not_both(Assoc), Edges, NBs).
+
+not_both(Assoc, U-V, ~BU + ~BV) :-
+        get_assoc(U, Assoc, BU),
+        get_assoc(V, Assoc, BV).
+
+independent_set_count(G, Count) :- independent_set(G, Sat), sat_count(Sat, Count).

benches/numlist.pl

@@ -0,0 +1,2 @@
+:- use_module(library(between)).
+run_numlist(Upper, Head) :- numlist(1, Upper, L), L = [Head|_].

benches/run_criterion.rs

@@ -0,0 +1,14 @@
+use criterion::{criterion_group, criterion_main, Criterion};
+
+mod setup;
+
+fn bench_criterion(c: &mut Criterion) {
+    setup::benches().run_all_criterion(c);
+}
+
+criterion_group!(
+    name = bench_group;
+    config = Criterion::default().sample_size(10);
+    targets = bench_criterion
+);
+criterion_main!(bench_group);

benches/run_iai.rs

@@ -0,0 +1,21 @@
+use iai_callgrind::{library_benchmark, library_benchmark_group, main};
+
+mod setup;
+
+#[library_benchmark]
+#[bench::normal(setup::benches())]
+fn bench_edges(mut b: setup::Benches) {
+    b.run_once("count_edges_short");
+}
+
+#[library_benchmark]
+#[bench::normal(setup::benches())]
+fn bench_numlist(mut b: setup::Benches) {
+    b.run_once("numlist_short");
+}
+
+library_benchmark_group!(
+    name = bench_group;
+    benchmarks = bench_edges, bench_numlist
+);
+main!(library_benchmark_groups = bench_group);

benches/setup.rs

@@ -0,0 +1,116 @@
+use std::{collections::BTreeMap, fs, path::Path};
+
+use criterion::{black_box, Criterion};
+
+use maplit::btreemap;
+use scryer_prolog::machine::{
+    parsed_results::{QueryMatch, QueryResolution, Value},
+    Machine,
+};
+
+pub fn benches() -> Benches {
+    Benches::new(&[
+        (
+            "benches/edges.pl", // name of the prolog module file to load
+            &[
+                (
+                    "count_edges_short",                 // name of the benchmark
+                    "independent_set_count(ky, Count).", // query to benchmark in the context of the loaded module
+                    btreemap! { "Count".to_string() => Value::try_from("2869176".to_string()).unwrap() }, // List of expected bindings
+                ),
+                (
+                    "count_edges", // multiple benchmark queries can be defined per module
+                    "independent_set_count(aa, Count).", // consider making the query adjustable to tune the runtime
+                    btreemap! { "Count".to_string() => Value::try_from("211954906".to_string()).unwrap(), },
+                ),
+            ],
+        ),
+        (
+            "benches/numlist.pl",
+            &[(
+                "numlist_short",
+                "run_numlist(1000000, Head).",
+                btreemap! { "Head".to_string() => Value::try_from("1".to_string()).unwrap()},
+            )],
+        ),
+    ])
+}
+
+pub struct Benches {
+    machines: Vec<Machine>,
+    runs: BTreeMap<String, Run>,
+}
+
+pub struct Run {
+    machine_idx: usize,
+    name: &'static str,
+    query: &'static str,
+    bindings: BTreeMap<String, Value>,
+}
+
+// Required for using a mutex. It doesn't actually send anything across threads,
+// and this is just a benchmark, so it Should Be Fine(tm). ¯\_(ツ)_/¯
+unsafe impl Send for Benches {}
+
+impl Benches {
+    #[allow(clippy::type_complexity)]
+    pub fn new(
+        benches: &[(
+            &'static str,
+            &[(&'static str, &'static str, BTreeMap<String, Value>)],
+        )],
+    ) -> Self {
+        let mut machines = vec![];
+        let mut runs = BTreeMap::new();
+
+        for b in benches {
+            let content = fs::read_to_string(b.0).unwrap();
+            let name = Path::new(b.0).file_stem().unwrap().to_str().unwrap();
+            let mut machine = Machine::new_lib();
+            machine.load_module_string(name, content);
+            machines.push(machine);
+            let idx = machines.len() - 1;
+            runs.extend(b.1.iter().cloned().map(|r| {
+                (
+                    r.0.to_string(),
+                    Run {
+                        machine_idx: idx,
+                        name: r.0,
+                        query: r.1,
+                        bindings: r.2,
+                    },
+                )
+            }));
+        }
+
+        Benches { machines, runs }
+    }
+
+    #[allow(dead_code)]
+    pub fn run_all_criterion(&mut self, c: &mut Criterion) {
+        for (_, runner) in self.runs.iter() {
+            let machine = &mut self.machines[runner.machine_idx];
+            c.bench_function(runner.name, |b| {
+                b.iter(|| {
+                    Self::run(machine, runner);
+                })
+            });
+        }
+    }
+
+    #[allow(dead_code)]
+    pub fn run_once(&mut self, name: &str) {
+        let runner = &self.runs[name];
+        let machine = &mut self.machines[runner.machine_idx];
+        Self::run(machine, runner);
+    }
+
+    fn run(machine: &mut Machine, runner: &Run) {
+        assert_eq!(
+            black_box(machine.run_query(black_box(runner.query.to_string()))),
+            Ok(QueryResolution::Matches(vec![QueryMatch::from(
+                runner.bindings.clone()
+            )]))
+        );
+    }
+}