Evaluation of the Voluntary National Tests: Phase 1 (1999)

Palo Alto, CA (May 11–12, 1998)

Participated in training provided to cognitive lab interviewers. Training consisted of two days of overview, tape observation, role playing, and discussion.

To acquaint evaluators with the interview methodology used in the cognitive labs. Also, attending both sessions provided evaluators with the ability to observe the evolution of the training materials.

East Lansing, MI (April 20–21, 1998)

AIR, New England (May 26, 1998)

M2 (2 taped)

To allow evaluators to observe the cognitive lab process and to determine whether the cognitive labs would provide information that could improve the quality of items. Observing interviews at several locations allowed evaluators to determine interviewer quality and whether the interviews were being conducted consistently across sites.

San Antonio, TX (June 8–9, 1998)

R9 (1 taped)

AIR, Palo Alto (June 10–12, 1998)

R7 (1 live interview, 1 taped)

East Lansing, MI (June 22–23, 1998)

R16 (2 live interviews)

R4 (4 taped)

R9 (4 taped)

M7 (1 live interview, 4 taped)

M10 (2 live interviews, 1 taped)

R20 (1 live, 1 taped)

M11 (1 live interview)

M19 (1 live interview)

M16 ( live interviews)

R22 (1 taped)

R18 (1 taped)

R22 (1 taped)

AIR, Washington, D.C. (July 1–2, 1998)

M2, M10 (9 tapes [some of which had already been seen] and 1-on-1 forms)

R4, R9 (9 tapes [some of which had already been seen] and 1-on-1 forms)

To allow evaluators to focus on the contribution of the cognitive labs to item quality for a small sample of items and to allow evaluators to review the 1-on-1 forms filled out by the AIR interviewers.

Math—San Antonio, TX (July 6–7, 1998)

Panel members reviewed 1,400–1,500 items.

To determine the impact of bias and sensitivity review on item quality.

Reading—Chicago, IL (July 6–7, 1998)

Panel members reviewed 350–400 passages.