runAllComparisons

Dynamics-engine comparison overview and execution guide.

Every scenarioCompare*.py script in basilisk/examples/dynamicsComparison exposes a run(showPlots, saveJson, ...) function returning a mapping from figure name to matplotlib figure. This script discovers all of them, runs each with its default arguments, and persists their artifacts:

  1. the JSON summary of the comparison metrics (saveJson=True), written to the results folder next to this script;

  2. every figure in the returned figureList, saved as a transparent .svg both to results and to basilisk/docs/source/_images/Scenarios (via simHelpers) so the docs build picks them up; and

  3. for the accuracy-comparison scenarios, an optional local BSM-vs-MJScene runtime .csv (saveTiming=True).

Required build features

The cross-engine comparisons require a Basilisk build configured with MuJoCo. Vizard and optical-navigation support are not required. Use a Release build for timing measurements; debug builds and machines under changing thermal or power-management conditions do not provide comparable performance data. The modules remain importable without MuJoCo so documentation discovery can still run, but cross-engine execution and the MuJoCo-specific sweeps then raise a clear ImportError or omit MuJoCo data.

Study types

The correctness scenarios progress from analytic rigid-body anchors through reaction wheels, flexible appendages, differential gravity, fuel slosh, and mass depletion. They report state agreement and conserved or analytic quantities. The two Pareto scenarios and the flexible-panel dimensionality study are benchmarks: they compare error with propagation-only wall time after warm-up. The six sweep*.py drivers are research controls for step size, velocity-dependent round-off, reaction-wheel bookkeeping mass, slosh displacement, torque-artifact scaling, and pendulum-bob inertia. Benchmark timings are descriptive measurements, not pass/fail guarantees.

Execution order and outputs

Run these commands from the repository root after building Basilisk:

python3 examples/dynamicsComparison/runAllComparisons.py
python3 examples/dynamicsComparison/sweepOrbitDt.py
python3 examples/dynamicsComparison/sweepTorqueArtifact.py
python3 examples/dynamicsComparison/sweepTorqueMechanismChecks.py
python3 examples/dynamicsComparison/sweepRwBookkeepingMass.py
python3 examples/dynamicsComparison/sweepSloshDisplacement.py
python3 examples/dynamicsComparison/sweepVariableMassPendulumInertia.py
python3 examples/dynamicsComparison/paperFigures.py --update-provenance

The documentation workflow instead invokes runAllComparisons.py --documentation-figures. That option uses explicitly reduced dimensionality and Pareto profiles to bound CI time. The affected scenario pages label those generated figures and state which conclusions require the complete default runs; the reduced figures are workflow demonstrations, not publication benchmark evidence.

The runner writes scenario JSON and SVG files to results/, copies documentation figures to docs/source/_images/Scenarios, and may write optional local runtime CSV tables. Each sweep writes its own JSON and SVG artifacts. paperFigures.py reruns every scenario and sweep used by a paper figure, table, or quantitative claim, writes print PDFs to results/paper, and replaces results/provenance.json with source, native-build, dependency, host, input, and output hashes. Run the provenance update only from a clean worktree whose Release build identifies the current commit. A later render validates those hashes before accepting the figures.

On a 2026 Apple M4 Pro Release build, the full scenario runner takes roughly five minutes, the documentation-only timing pass takes two to three minutes, and the standard sweeps finish in about one minute combined. The direct torque-mechanism control is intentionally longer at about ten minutes. Paper rendering takes less than a minute. These values are planning estimates only and vary with hardware, compiler, background load, and thermal state.

Interpretation limits

Agreement demonstrates equivalence only for the modeled force, mass-property, joint, gravity, and integration conventions documented by each scenario. The engines do not share every gravity or multibody approximation, so some extended cases intentionally retain a physical residual. Timing ratios apply to the listed model sizes, integrators, tolerances, and host; they must not be generalized into an engine-wide performance claim. Inspect the scenario-specific assumptions and the stored configuration metadata before reusing any result.

runAllComparisons.discoverScenarios()[source]

Return the module names of every comparison scenario in this folder.

Returns:

sorted module names (without the .py extension) of the scenarioCompare*.py scripts living next to this runner.

Return type:

list

runAllComparisons.requireMujoco()[source]

Require the build feature needed for complete comparison figures.

runAllComparisons.run(timingOnly=False, scenarios=None, scenarioRunKwargs=None, resultsDir=None, saveDocumentationFigures=True, saveJson=None, saveTiming=None)[source]

Run every comparison scenario with its defaults, saving JSON, figures, and tables.

Each scenario runs with showPlots=False and saveJson=True; the accuracy scenarios additionally run with saveTiming=True to regenerate their runtime table. Scenarios returning no figures (e.g. built without MuJoCo) are skipped after their JSON is written.

Parameters:

  • timingOnly (bool, optional) – if True, run only the scenarios that publish a local runtime table, and skip their JSON summaries and figures.

  • scenarios (sequence, optional) – explicit module names to run. Defaults to the timing scenarios when timingOnly is True and discovered scenarios otherwise.

  • scenarioRunKwargs (dict, optional) – per-module keyword overrides passed to run.

  • resultsDir (str, optional) – explicit artifact directory. Defaults to the comparison results folder.

  • saveDocumentationFigures (bool, optional) – if True, copy generated figures into the Sphinx scenario-image directory. Defaults to True.

  • saveJson (bool, optional) – override JSON generation for every scenario. Defaults to True except in timing-only mode.

  • saveTiming (bool, optional) – override runtime-table generation for every timing scenario. Defaults to True.

Returns:

module names completed in execution order.

Return type:

tuple

runAllComparisons.saveFigures(figureList, resultsDir=None, saveDocumentationFigures=True)[source]

Save every figure produced by a scenario as a transparent SVG.

Each figure is written twice: once into the results folder, and once into the documentation image folder via simHelpers.saveScenarioFigure (the same path the scenario unit test uses) so the docs build can pick it up.

Parameters:

  • figureList (dict) – mapping from figure name to matplotlib figure, as returned by a scenario’s run function.

  • resultsDir (str, optional) – explicit artifact directory. Defaults to the comparison results folder.

  • saveDocumentationFigures (bool, optional) – if True, also copy each figure into the Sphinx scenario-image directory. Defaults to True.