Performance Measures for Selection of Metamodels to be Used in Simulation Optimization

Anthony C. Keys
Department of Management Information Systems, University of Wisconsin – Eau Claire, Eau Claire, WI 54702, email: keysac@uwec.edu

Loren Paul Rees
Department of Business Information Technology, Virginia Polytechnic Institute and State University, Blacksburg, VA 24061, email: loren@vt.edu

Allen G. Greenwood
Department of Industrial Engineering, Mississippi State University, Mississippi State, MS 39762, email: greenwood@ie.msstate.edu

ABSTRACT. This paper points out the need for performance measures in the context of simulation optimization and suggests six such measures. Two of the measures are indications of absolute performance, whereas the other four are useful in assessing the relative performance of various candidate metamodels. The measures assess performance on three fronts: accuracy of placing optima in the correct location, fit to the response, and fit to the character of the surface (expressed in terms of the number of optima). Examples are given providing evidence of the measures’ utility—one in a limited scenario deciding which of two competing metamodels to use as simulation optimization response surfaces vary, and the other in a scenario of a researcher developing a new, sequential optimization search procedure.

Subject Areas: Kernel Smoothing, Metamodels, Nonparametric Statistics, Performance Measures, Simulation, Simulation Optimization, and Thin-Plate Splines.

<top of page>