Benchmarking

Run this from your repository root on the instance after each change:

make benchmark

It saves your code, builds and tests that copy, and measures one 1024×1024 curves input by default. Use make benchmark WORKLOAD=directed throughout the directed portion to measure all three required inputs. The directed portion introduces the first experiment. Use this page when you need more detail about the results, saved versions, or other workloads.

Correctness Checks

make benchmark builds one testing executable, test_pipeline, alongside pgm2svg. Running test_pipeline without arguments checks graph connectivity, path tracing, filtering, simplification, and basic full-pipeline behavior. scripts/check.py runs those tests and checks application output and arguments.

The same testing executable accepts --dump INPUT.pgm OUTPUT to run the pipeline on an image and save its logical result fields in a deterministic binary file. The benchmark compares these dumps across versions, excluding timings and memory addresses, before measuring performance. Matching SVGs alone would not detect all changes to the pipeline result.

To run the checks on your working code without saving an experiment:

make bin/pgm2svg bin/test_pipeline
python3 scripts/check.py bin

Reading the Results

Example total kernel times:

Total kernel time
Version   Exp.  Mean (ms)  SD (ms)  Current speedup
current   003      80.000    1.000                -
starter   -       100.000    2.000           1.250x
previous  002      90.000    1.500           1.125x

starter is the supplied original code, so it has no experiment number. previous is the latest successful experiment; it may contain the same code as current if you reran without editing.

Speedup is the comparison version’s mean divided by current’s mean. Above 1 means current is faster; below 1 means it is slower.

SD is the standard deviation across measured executions. With RUNS=1, SD is unavailable and shown as -. A ratio is n/a when the current total is zero.

To compare against additional saved experiments, pass their numbers as a comma-separated list:

make benchmark COMPARE=002,005

These are added to the starter and previous comparisons. Each requested experiment must have succeeded; missing, failed, or interrupted experiments produce an error. Duplicate versions are shown once. Extra rows are labeled version-002, for example. All compared versions are checked for correctness and rerun on the currently selected inputs before speedups are reported.

Each executable discards one warmup execution before measuring. Total kernel time includes pipeline setup and cleanup, but excludes image loading and SVG output. If you combine stages, their original labels may no longer describe the work; use total kernel time for the overall comparison.

Repeat an experiment to see whether an improvement is consistent across invocations as well as within one invocation. If the change is smaller than the timing variation, collect more measurements before drawing a conclusion. Run one benchmark at a time, without a separate profiler alongside it.

Saved Experiments

Every invocation creates a new directory, such as experiments/001/. Use the experiment numbers to identify versions in your report. Nothing is overwritten.

Do not delete experiments/ or its saved experiments. They preserve the source and results needed for comparisons and your report. Git ignores this folder, so back it up separately before losing access to your instance.

Failed builds and correctness checks are saved but skipped when choosing comparison versions. Fix the problem and run the command again. To recover an older implementation, copy the files you need from its source/src/ back into your working tree’s src/. make clean removes the current build in bin/ and leaves the experiments alone.

Workload Presets

The default quick preset contains one 1024×1024 curves input for fast iteration. The directed preset contains all three required directed workloads:

make benchmark WORKLOAD=directed
make benchmark WORKLOAD=varied

Preset	Images
`directed`	1024×1024 and 2048×2048 curves; 2048×2048 texture.
`quick`	1024×1024 curves.
`texture`	2048×2048 noise with many short paths and junctions.
`vertical`	2048×2048 winding vertical boundaries.
`straight`	2048×2048 straight boundaries.
`varied`	Shapes, vertical curves, texture, and straight lines in rectangular and odd-sized images.
`borders`	5×5 shapes, 37×29 texture, and 513×257 curves; useful for checking small images and leftover vector lanes.
`large`	4096×2048 curves, 2048×3072 texture, and 3072×2048 shapes.

The definitions are in workloads.yaml. Add a preset or change its image entries using pattern, width, and height. Patterns are curves, vertical-curves, texture, straight, and shapes; dimensions must be between 5 and 8192. Small inputs are useful for correctness, but their timings can be dominated by timer and allocation overhead. Use WORKLOAD=directed for your directed results.

Other Options

You can use your own PGM image instead of a preset, increase the measured run count, or attach a note to help identify an experiment:

make benchmark INPUT=examples/valve.pgm
make benchmark RUNS=80
make benchmark NOTE="reuse row offsets"

INPUT overrides WORKLOAD. These options can be combined. Pipeline parameters stay at their defaults for this workflow; the Program Guide describes the executable’s arguments for separate parameter experiments.

RUNS controls both the normal timing measurements and the additional profiling executions when PROFILE=1. It defaults to 7 without profiling and 40 with profiling.

Profiling

When you reach the open-ended portion, use make benchmark PROFILE=1 to collect profiles after the timing measurements. The APerf guide explains how to collect, view, and interpret them. You can combine PROFILE=1 with a workload preset or custom input.

Files and Troubleshooting

An experiment directory contains:

source/: source directories, Makefile, and workload definitions for that attempt.
bin/: the saved compiled programs.
inputs/: the exact images used.
results.txt: the report; results/: all versions’ stage statistics (including min/max), correctness logs, and SVGs.
profiles/: APerf recordings and reports, when requested.
build.log: compiler output; check.log: pipeline and output checks.
experiment.json: automatically recorded settings, version numbers, and results.

If the build fails, read build.log. A correctness failure identifies the input and differing field. Keep an interrupted attempt’s directory and rerun the command. If profiling fails after the normal checks and timings succeed, those benchmark results remain available for comparisons; the profiling logs explain the failure.