What Does it Mean When a College Kid From Ecuador Beats the Best?

Yesterday the William and Flora Hewlett Foundation awarded $100,000 to the top five teams in an assessment scoring competition. The goal was to build software systems that could grade short answer responses on state standardized tests as accurately as trained experts—a more difficult challenge than the essay competition held in the first quarter of this year where the winning algorithms equaled experts.
Hewlett wants to see deeper learning in American classrooms and recognizes that assessment can influence instruction. The online assessment of Common Core State Standards will frame U.S. education the way NCLB and AYP framed the last. The short window to influence the quality of new state tests, which roll out in 2014-15, offers a big opportunity.
The Automated Student Assessment Prize (ASAP) was constructed to support the aims of the two state testing consortia, PARCC and SBAC—better tests of higher order skills at a lower price point. Meeting these objectives will require automated scoring of constructed response tasks. Anyone who has taken the time to investigate the consortia’s projected use of Race to the Top funding to influence the comprehensive overhaul of summative assessment among their 47 state partners can see that building a next generation of high stakes assessment can only work with a big boost from technology. So, there’s a lot at stake, and ASAP is now the new platform for fairly testing the capabilities of those solutions.
ASAP was hosted on Kaggle, a venture-backed platform that delivers a compelling data challenge to a pool of 55,000 machine-learning scientists from around the world. As evidenced by many Kaggle competitions, well-constructed prizes mobilize talent and accelerate innovation. ASAP appears to have succeeded on both accounts. Like the first competition, ASAP Phase Two attracted amazing and diverse talent. The three top competitors came to a CCSSO meeting in Indianapolis to pick up their checks. I’d like to introduce you to the winners.
Luis Tandalla, 1^st place. A Fulbright Scholar at the University of New Orleans, Luis is from Quito, Ecuador. His mother is an elementary school teacher, but it was a secondary school teacher that really turned him on to math. He entered about 20 math competitions and won many of them. He skyped his mom into the award ceremony, so that she could see her proud son pick up a $50,000 check.
With initial interest in aerospace, Louis is majoring in mechanical engineering. A newcomer to data science, Luis participated in Stanford’s Massively Open Online Course (MOOC) in machine learning last year from Andrew Ng, and it ignited a passion. He entered the first phase of ASAP and placed 13^th, equaling vendors who have been in the testing business for decades. After doing so well in data competitions, he’s applying to top computer science graduate programs. (How about we get the kid an H1b visa so he can work here?)
Jure Zbontar, 2^nd place. Jure grew up and lives in Ljubljana, Slovenia, where he is a teaching assistant at the local college. He’s pursuing a Ph.D. in computer science in the field of machine learning, but – given how well he did – we think he’ll have a lot of bright opportunities ahead of him. He was approached by several of the leading vendors in automated assessment at the Indianapolis conference, who are thirsty for bright young talent, especially an eastern-block grad student who can beat their team of Ph.D.s.
Jure began programming in high school. He has entered seven Kaggle competitions. He won a challenge to predict the Eurovision winner.

Xavier Conort, 3^rd place – A French-born actuary, Xavier consults to insurance companies in Singapore. He holds two masters’ degrees and is a Chartered Enterprise Risk Analyst. He has worked in Brazil and China and is married to a Malaysian.
Xavier is the top-rated competitor in the Kaggle community. He has entered at least a dozen competitions and placed in most.
Xavier notes that there are two types of competitors on Kaggle: computer scientists and statisticians. He modestly says, “I’m not a very good programmer.” It looks like he’s pretty good at pattern recognition.
“It’s how I keep learning,” said Conort. About the competitions he chooses Xavier said, “I like to work on interesting problems where I have a chance to make a contribution.”
He’s looking forward to getting back home to see his daughter who turned one last week. He’s flying LA-Paris-Tokyo-Singapore.
What it means. Jaison Morgan is my co-director of ASAP. We spent a couple years searching for a mega-prize that would transform global education. We didn’t find one. But we are convinced after a couple very successful targeted efforts that a sequence of small, targeted prizes can focus and accelerate innovation.
Second, it’s interesting to note that the winners all learned a lot from Andrew Ng’s Stanford MOOC last year. After the success of the first course, Andrew won backing from Kleiner Perkins to co-found Coursera. Learning is global. A lot of it is free.
Third, the stories above illustrate the fact that well constructed prizes mobilize global talent. Seriously, you could not make this stuff up—a kid from Ecuador beat some of the best testing companies in the world after six week of work. Young men (yes, it still is mostly young men) from Slovenia to Singapore, from Pittsburgh to Poland poured 100 hours a week into the competition hoping to see their name creep up the leaderboard. Crowdsourcing works.
The fourth lesson from ASAP is that most innovation is translational—something that worked in one field may work in another. The question is how to expose cross-discipline and cross-sector innovation? Prizes are a super-efficient means of promoting translational innovation. Some of the winners from ASAP Phase One are already helping some of the big testing companies improve their services.
Fifth, in Phase One, we let competitors keep their intellectual property. But in Phase Two we required that competitors open source their code (GPLv3 license) along with an instruction manual. Interestingly, there was exactly the same number of competitors in Phase Two. This suggests that the world, at least data science, is becoming more open.
Last, this is just the beginning of Big Data in education. Next week Digital Learning Now! will release a paper on comprehensive learner profiles. A warehouse of data on each student will unleash the power of predictive analytics that will empower teachers to personalize learning in new and powerful ways. Prizes will accelerate innovation in analytics.
It’s just about to get fun.
This blog first appeared on EdWeek.