Evaluation of Endogenous Systems

In system development we are faced with the necessity of evaluation. Evaluation measures our success relative to other: a) theories of development or domains; b) implementations of similar theoretical principles; or c) increments of a given system. For successful software engineering evaluation, progress is measured against a model, a task frequently accomplished through requirements analysis. This model is lacking in natural language processing (NLP) systems (not to be confused with neurolinguistic programming). NLP systems defy evaluation in part because they model an endogenous process—where the whole process is irreducible. Therefore, while specific feature-based evaluations appear reasonable, they fail to capture an overall measure of success. In this paper, we look at the part/whole aspects of evaluation in more detail with regard to one language system type—machine translation.