No excuse for sucky items

John Behrens directs Cisco’s Networking Academy.  They partner with 10,000 schools in 160 countries and provide free online courses on computer networking.  In addition to being an important workforce development strategy, they wanted the Academy to be an important demonstration project for eLearning.
Prior to joining Cisco in 2000, John was a professor of education at ASU.  John directs curriculum, assessment, and technology associated with the Academies, so he has the opportunity to create a fully aligned instructional system with an integrated data architecture behind it.
The Academy serves high school and post secondary students interested in careers in networking and IT as well as students that just want broader job skills.   They serve 1 million students annually and deliver 40,000 exams most weekdays.
Initially, most of the Academy tests were multiple choice, but they have moved on to more sophisticated techniques including simulation-based assessment.
The work of Bob Mislevy, now at ETS, influenced John and Academy design.  Mislevy designed the Bayesian NAEP sampling strategy and pioneered evidence-centered design.
Behrens and his team talk about assessment activities not items.   He thinks a focus on ‘items’ is constraining U.S. thinking about assessment.  His adaption of evidence-centered assessment includes four elements: 1) activity selection, 2) presentation (e.g., question, video, observation), 3) work product, and 4) feature identification.
The key to quality is feature extraction.  Multiple choice questions trivialize the effort to extract evidence  by mushing together presentation and feature identification.
To paraphrase Behrens, when we were data poor, we sucked at feature identification, so we used sucky items.  But there’s no excuse today for limiting psychometrics to 1950s strategies.  Rather than trivializing multiple choice, Behrens suggests that we should be using data mining and pattern recognition strategies to extract evidence of learning.  Students should be engaged in rigorous activities including simulations and challenging performance tasks.
This country is stuck in a psychometric box based on a history of data poverty.  Behrens’ leadership allowed Cisco to rapidly evolve its assessment system to one that is rigorous, scalable, and available on demand.  If we take note, John’s contributions to workforce development will only be overshadowed by his contribution to online assessment and certification of learning.