The Evaluation of Competing Classifiers

This dissertation research makes contributions towards the objective evaluation of competing classifiers, i.e., classification systems (CSs) or pattern recognition algorithms. Automatic CSs have been under development for almost 40 years in a wide range of military and medical applications. During this period, scientists and engineers have developed extensive theory and algorithms for classification, but by comparison have focused little on the testing and evaluation of their systems. Classifier evaluation is very important in the fields of automatic target recognition (ATR) and pilot workload classification. In order for military operators to be confident in new CSs, they must have an objective way of testing and evaluating competing systems.The purpose of this dissertation research is to advance the knowledge of classifier evaluation. The basis of the research is a commonly used evaluation tool in ATR and medical applications, called the receiver operating characteristic (R.OC) curve. A proof of convergence with respect to increasing sample size for these ROC curves is provided. This ROC convergence theorem is important because it provides the basis for a framework for the comparison of ROC curves and hence, the comparison of classifiers.