Evaluating predictive models with application in regulatory policy for invasive weeds

Exotic plant species, if imported, represent a possible threat to indigenous agricultural and ecological systems. Weed risk assessment is a methodology used to try and identify which exotic plant species, among those presented for import, are an actual threat, and which are not. The former group can then be denied entry, and the latter allowed. Data, usually on plant characteristics correlated with invasive weed potential and sometimes on the characteristics of exposed habitats, are used to develop a statistical prediction rule. Typically this is an algorithm that generates an indicator variable used as a basis for decision making in the risk assessment procedure. With a useful indicator, most weed species will correctly be identified as weeds (true positives) and most non-weed species will correctly be identified as non-weeds (true negatives). However, as with all predictors, errors may sometimes arise. Some non-weed species may wrongly be identified as weeds and denied entry (false positives), while some weed species may wrongly be identified as non-weeds and allowed entry (false negatives). Receiver operating characteristic methodology is based on a graphical plot of the true positive proportion against the false positive proportion. This methodology may be used to evaluate a statistical prediction rule. In conjunction with information on the costs of the different decisions that might be reached and the prevalence of weeds in the population of prospective plant imports, receiver operating characteristic methodology also provides a basis for finding the optimum operational threshold for a statistical prediction rule used in weed risk assessment. We describe the use of receiver operating characteristic methodology to evaluate the implications of the different types of regulatory policy for invasive weeds that may be adopted once weed risk assessment is in place.