trade-study

Design and evaluation framework for scientific simulation studies. Score competing configurations — model formulations, solver choices, measurement strategies, or any design decision — against known ground truth via protocol-driven simulators, proper scoring rules, hierarchical phases, multi-objective Pareto optimization, and Bayesian model stacking.

trade-study is a design and evaluation framework for scientific simulation studies. Users define simulators — protocol-conformant objects that generate (truth, observations) pairs from a configuration — then score competing configurations (model formulations, solver choices, measurement strategies, or any design decision) against known ground truth, so that decisions validated on synthetic benchmarks transfer reliably to real observational data.

Evaluation proceeds through hierarchical phases — from broad experimental design (full factorial, Latin hypercube, or Bayesian-adaptive search) to focused refinement — scored with proper scoring rules (CRPS, WIS, Brier, coverage).

A four-tier observable hierarchy (embedded constraints, penalized objectives, diagnostic metrics, cost axes) structures multi-objective Pareto optimization with hypervolume and IGD+ front-quality metrics. Calibrated ensemble predictions are produced via Bayesian or score-based model stacking. Global sensitivity analysis (Morris and Sobol screening) identifies which factors matter most.

For expensive studies, successive-halving and Hyperband runners screen the design space at low fidelity before committing full budget to survivors, and GP/RF surrogate models — including a regime-conditional surrogate that interpolates recommendations across continuous descriptors like sample size or noise level — approximate scores without re-running the simulator.

Available in Python and Julia. Released on PyPI and archived at Zenodo. Documentation at jcm-sci.github.io/trade-study.