Decision-Making Under Uncertainty for Multi-stage Pipelines: Simulation Studies to Benchmark Screening Strategies
Multi-stage screening pipelines are ubiquitous throughout experimental and computational science. Much of the effort in developing screening pipelines focuses on improving generative methods or surrogate models in an attempt to make each screening step effective for a specific application. Little focus has been placed on characterizing generic screening pipeline performance with respect to the problem or problem parameters. Here, we develop models and algorithms to codify and simulate features and properties of the screening procedure in general. We outline and model common problem settings and potential opportunities to perform decision-making under uncertainty to optimize the execution of screening pipelines. We then illustrate the models and algorithms through several simulation studies. We finally show how such studies can provide a quantification of the screening pipeline performance with respect to problem parameters, specifically identifying the significance of stage-wise covariance structure. We show how such structure can lead to qualitatively different screening behaviors, and how screening can even perform worse than random in some cases.