Definition: Tests designed to detect change from interventions, developmental, and/or situational influences.
Description. More recent approaches to test development with measures focusing on progress monitoring and outcome assessment (PMOA) have focused on methods of selecting change-sensitive items with adequate psychometric properties. It is important to note that items and measures that show change do not automatically correspond to the same items groupings identified by techniques such as factor analysis (Kopta et al., 1994; Weinstock & Meier, 2003). Kopta et al. (1994), for example, examined psychotherapy outpatients’ scores on the Symptom Checklist 90 (SCL-90-R), a widely used symptom checklist in psychotherapy. They found that scores on items could be categorized according to the three rates of change the items displayed in response to treatment: Acute items, quick response to treatment; chronic distress items, moderate response rate; and characterological items, slow response rate. Acute items that evidence rapid response to psychotherapy included content tapping anxiety, depression, and obsessive-compulsive symptoms. The items grouped in these three categories do not correspond to the single category found in factor analyses of the SCL-90-R conducted at a single time point.
Although relatively little theoretical work has focused on the construction of test items for assessing change, a growing body of literature does exist that examines tests designed for evaluating educational and psychological interventions. Criterion-referenced tests, for example, are composed of items based on specific criteria for which an intervention is targeted. In a mastery learning class, students might take a reading comprehension test that involves reviewing assigned readings and then attempting to answer correctly at least 90% of the items. Item selection in criterion-referenced tests (Swezey, 1981) includes (a) task analysis of the criteria, (b) determination of costs associated with measuring different aspects of criteria, (c) creation of an item pool, and (d) a pilot study in which items are administered to mastery and non-mastery subjects to determine which items can discriminate between the two groups. The item selection process focuses not on finding stable items but those that demonstrate the effects of an intervention to produce change to criterion.
Similarly, researchers employed longitudinal methods in areas such as developmental psychology attempt to measure intraindividual differences over time. While such researchers are interested in distinguishing between static (trait) variables and dynamic (state) variables, Collins (1991) observed:
Little in traditional measurement theory is of any help to those who desire an instrument that is sensitive to intraindividual differences. In fact, applying traditional methods to the development of a measure of a dynamic latent variable amounts to applying a set of largely irrelevant criteria. (p. 138-139).
In research on longitudinal measurements of physical activity and behavior, reviewed studies that found that physical exercise alleviated reactive depression in a variety of samples. Measurement issues in these studies center on finding reliable instruments that can demonstrate significant within-subject change in physical activity over time. Tryon (1991) proposed three criteria for the selection of items and tasks suitable for a test of an intervention:
- Items should evidence change resulting from the presence of an intervention; for example, items should demonstrate expected changes from pretest to post-test.
- Items should not change when respondents are exposed to control conditions; that is, item change should not occur over time independent of an intervention.
- Changes in scores from pre-intervention to post-intervention demand that such alterations not be attributable to measurement error. Thus, measures should not be affected by such non-intervention factors as social desirability, practice effects, or expectations for improvement.
Change-sensitive test procedures (see Chapter 3) are needed for the construction of tests intended to measure change in counseling and psychotherapy. Progress monitoring refers to the use of test scores for providing feedback during the ongoing process of counseling and psychotherapy, while outcome assessment provides information relevant to the overall amount of progress made by a client. With progress monitoring, data may be produced every session or on some other frequent schedule (such as every 3rd or 5th session) to track client change over time. The general rule of thumb is to produce data for progress monitoring as frequently as practically possible so that clinical feedback can be delivered in a timely manner. In contrast, outcome assessments are typically intended to provide a more final answer to the question, Did the client improve? They are often administered as part of a pre-post design that includes an assessment at intake and termination.
The noteworthy feature about all approaches that emphasizes PMOA is the close connection between assessment data and intervention that results in structured clinical feedback. In traditional therapy, assessment takes the form of an initial interview that provides information about the presenting problem(s) and important contexts such as family history. Based on this information the clinician may assign a diagnosis and develop a plan that focuses on an evidence-based treatment. But the majority of clients respond to interventions in diverse ways, often presenting the clinician with difficult decisions about whether to continue, alter, or discontinue particular treatment approaches. With PMOA measures and their resulting clinically-relevant data, the therapist receives explicit feedback about effectiveness as treatment progresses. When employed in the context of a change-based theory, such data can greatly inform the clinician about how to proceed more effectively.
These videos provide a 15m and 23m introduction to key elements of PMOA.
