THE ORAL EXAMINATION FOR LICENSURE OF PSYCHOLOGISTS

INTRODUCTION

The Office of Examination Resources of the California Department of Consumer Affairs evaluated the oral examination of the Board of Psychology. The analysis, part of the board’s ongoing review of the oral examination, is to determine whether the examination is fulfilling its intended purpose. To add or delete portions or even change the entire examination does not invalidate previous examination decisions. All regulatory agencies have the obligation to evaluate their examination programs even in the absence of challenges to the validity of the programs.

The board’s oral examination has been developed to high standards and has been serving as a means of assessing minimum competence. The content for the examination is based upon job-related behaviors and associated knowledge defined in a practice analysis conducted by the American Association of State and Provincial Psychology Boards. Structured questions and responses were developed, behaviorally anchored rating scales were developed to use as the criteria for performance levels, and standardized test administration procedures were implemented. The decision to pass or fail the candidate is based upon an aggregate of their scores in response to the structured questions. However, the oral examination is a “test” and must meet all the psychometric standards of the Standards for Educational and Psychological Testing.1

Despite efforts by the Board of Psychology to improve the psychometric quality of the oral examination, questions remain about the validity of the pass/fail decisions. The oral examination may not be measuring a candidate’s potential for safe practice because of problems in development, administration, and scoring. It is necessary to consider the oral examination in the light of its consequences of failing or passing candidates, not solely in relationship to the intent of the examination. The constructs being assessed are not testable in the context of a licensing examination.

There are several sources of error in an oral examination that adversely impact the standardization, and thus, the validity of the oral examination. One source of error has to do with content, and the others are procedures for examination development and administration. Granted, the content of the oral examination is linked to a practice analysis to control for error related to content. However, the content of the examination requires a second interpretation by examiners. The development of the vignette and the questions depends upon interpretation of the content of the practice analysis by psychologists in the development workshops. Regardless of the psychometric quality of the performance criteria, they have value only if they are applied consistently according to the level of candidates’ performance. All candidates should have the same examination experience, not only in terms of administration, but also in terms of scoring.

RELEVANT STANDARDS

The Standards for Educational and Psychological Testing are the standards by which the oral examination was evaluated. The most relevant standards to oral examination are:

1 American Educational Research Association, American Psychological Association & National Council on Measurement in Education. (1999). Standards for Educational and Psychological Testing. Washington, DC: American Educational Research Association.