A Conversation with Auditi Chakravarty of AERDF on Education Research and Development Measuring Edtech’s Impact

I was recently at ISTE 2024 where I got the chance to sit down with Auditi Chakravarty, CEO of the Advanced Education Research and Development Fund (AERDF). Auditi had just finished leading a session at ISTE, which focused on K-12 demand for evidence of the edtech lifecycle, including evaluation, implementation, and outcomes.

This conversation came on the heels of writing Unfulfilled Promise: The Forty-Year Shift From Print to Digital and Why It Failed to Transform Learning for the Hoover Institute. This paper addressed many of the points mentioned in a recent study that found that school districts accessed an average of 2,591 edtech tools and students and educators both averaged about 42 tools in the 2022 to 2023 school year. This number was then complicated by the fact that the number of unique digital solutions accessed by educators decreased by more than 14% over the year prior – highlighting that educators may be starting to feel “tech fatigue.” 

I sat down with Auditi live at ISTE to discuss what’s next for edtech. 

Tom Vander Ark: Auditi, I had an interesting experience this year. I wrote a 40-year retrospective of EdTech for the Hoover Institution, and it was a difficult paper to write because I’ve been involved in edtech for 40 years and the big conclusion was we haven’t done very well. We haven’t made the big intended impact that I hoped for. When I listed the reasons why, high on the list was the fact that we just don’t have much R&D in the education sector, or at least we didn’t until recently.

Unlike the health sector where there’s significant R&D linked to innovation and venture capital, we don’t really have that link. So there’s missing R&D and a limited link to private capital. 

AERDF is an important part of that solution for the education sector. I imagine that’s what drew you to AERDF. Tell us what is AERDF and what are you up to now? 

Auditi Chakravarty: Yes Tom, that is what drew me to AERDF. Similarly, having been a teacher, having built edtech tools and programs led research teams, and seeing how these are really siloed and separate spaces in education. that edtech space where product and program development happens, it’s like an entrepreneur gets that funding and thinks, “I have a great idea and I’m going to go build the thing.” 

Tom Vander Ark: This is a joke that I’ve told for 20 years because there was no venture capital until 20 years ago. Then in the beginning, in the first 10 years, like from 2010 to 2020, it was a tech dude whose sister was a teacher, who complained over Thanksgiving. He would say, “I can take care of that” and they would code an app. And so that was R&D in education 15 years ago. 

Auditi Chakravarty: Yeah, along with the idea of building based on what we already know from the research about how learning happens, what actually supports learners the best. Let’s start from that. Let’s bring researchers into this, around the table with us. So now I’ve got a teacher and a developer, but let’s also bring the researcher to the table. A lot of the edtech companies now have researchers on their teams.

They have research departments. They’re doing a lot of product usage research, market research, and mining the data they have. The kind of research that doesn’t happen in our sector, but does happen in other industries, is more like the researcher working at Stanford, or whatever university it may be, to really understand a product or problem or build a technology or tool. From there, the product developers and solution providers to build. Now, we have some of that with certain edtech products that came out of university labs, but once commercialized, it’s like the tether, the link to the research gets lost. Then they’re commercialized in the hands of a big commercial entity, and continuing to track and monitor for evidence, becomes an afterthought. 

I was on a panel yesterday at the Solutions Summit here where we were talking about evidence and why that’s so important. One of the things we talked about is how much it matters that districts and policies demand it. 

Tom Vander Ark: There hasn’t been strong aggregated demand. 

Auditi Chakravarty: That’s right. 

Tom Vander Ark: This is one of the downsides of a very decentralized system. That school systems haven’t aggregated demand in a way that helps to drive both research and innovation funding.

Auditi Chakravarty: That’s right, yeah, I think the demand side needs to move more. I’m curious to see if some of the impact of the ESSER cliff that everybody here is talking about, might actually be good in that respect, it might force districts and purchasers to say, Alright, I’m going to have to get smart about what I’m spending my money on, do the things that I purchase, actually work?

Do they produce results for my students, educators, and my context? To your point about 40 years of edtech and not seeing much change, it’s only in the last few years that we’ve started using this phrase, as researchers and developers talk about a lot now: “it’s not enough to understand that something works, it’s for whom it works.” That has been true for researchers for a long time. That’s part of the methodology and what researchers look for in a published study, However, people have not been as focused on that in the actual product research they’ve done. I think that’s starting to shift. In terms of what we do with our R&D programs, at AERDF.

Tom Vander Ark: What is AERDF? when did it start and what is it? What’s the charge? How much money do you have to spend? 

Auditi Chakravarty: We started AERDF officially in 2021, but our demonstration program launched in 2019.

Tom Vander Ark: Is it a public charge? What kind of an organization is it? 

Auditi Chakravarty: We are a philanthropically funded non-profit.

Tom Vander Ark: How much of it is federal funding?

Auditi Chakravarty: It’s all privately funded. We are fully funded by private philanthropy at this point. Our founding story dates back to Jim Shelton, who was part of the founding team. He was with Chan Zuckerberg Initiative (CZI) at the time and AERDF came from a dream and a vision he had when he was at the Department of Ed as assistant secretary looking to launch a DARPA for education. He was looking at the model of DARPA saying, we need this kind of R&D in education. That was a big deal. on his part, but it didn’t pass Congress. So philanthropy took up the charge and said, this is a space where we can show what’s possible.

We’re part of these efforts in coalition with other R&D organizations to see a DARPA Ed, which would be called NCADE. We’re also really interested in seeing increased public funding for education R&D because public funding for education R&D is minuscule in comparison to funding for defense, health, and pretty much any other sector supported by R&D. This has been philanthropically supported, but we can also help to move the needle in terms of public investment over time. 

At the core, what we’re demonstrating and doing now is with three programs — EF+ Math and two additional programs.

Tom Vander Ark: And EF+ Math is aimed at math instruction?

Auditi Chakravarty: Particularly for middle grades each of our programs addresses a hypothesis and an area of knowledge to unlock. EF+ Math was founded by Dr. Melina Uncapher, a neuroscientist specializing in executive function skills.

The question was, we know that it is possible to strengthen executive function skills in isolation, but much of that work happens in the lab. We’ve seen very limited application of strengthening these skills in the context of academic work by the time students are in middle school, you really want to see that it’s translating to their math learning performance or literacy performance. Moreover, much of the existing research had small sample sizes and didn’t prioritize our students, particularly regarding Black and Latino students and learners experiencing poverty, These students who’ve historically been on the margins, have not been centered in a lot of R&D. When you dig into effects on a lot of the studies and you see how they do for Black students or Latino students and you suddenly see, wow, they’re not getting the same benefits as the aggregate, the overall number of students is seeing. We believe that designing our R&D for these students will result in outsized benefits that will translate to all students. 

Tom Vander Ark: So is the EF+ Math research program focused on developing particular products or knowledge?

Auditi Chakravarty: Each AERDF program focuses on producing generalizable knowledge, and technical capabilities, as well as prototypes and products. EF+ math supports multiple prototypes each taking different approaches to strengthen executive function skills while focusing on equity to improve math learning They are learning about the levers that have the greatest impact in creating generalizable knowledge The prototype teams themselves, build various Materials and edtech tools including novel uses of AI for assessing executive function within academic contexts. we want others to learn from, share, and embed these technical capabilities into their tools For instance, one of our prototypes, CueThink, was recently acquired by Imagine Learning. It’s a product with substantial research behind it that’s generating generalizable knowledge, and now has the potential to scale because it’s embedded in a larger edtech company. That’s an example of how our programs work, but each program is different. 

Tom Vander Ark: What are your other programs? 

Auditi Chakravarty: Reading Reimagined, is one of our programs. It addresses the question of how we can help older elementary readers, from Grades 4 through 8, who are struggling to read. Rebecca Kockler, the Executive Director of that program, uncovered research showing that students who can’t decode by around 4th grade, if they don’t reach a decoding threshold, never become proficient readers. Post-Grade 3, we stop measuring and supporting decoding, we focus everything on comprehension. The interventions that exist for decoding, are not grade-appropriate for older elementary learners so the program looks at defining a decoding threshold. better measuring decoding skills of older elementary learners and supporting them with linguistically and age-appropriate instruction, to create an integrated tool.

Tom Vander Ark: Think of the Learner Variability Project from Digital Promise. It has really helped unlock a broader view of the cognitive challenges that could be part of reading difficulty. Is that fair? 

Auditi Chakravarty: That’s part of it. For example, considering a learner’s home language is crucial. A student who speaks Spanish at home…

Tom Vander Ark: Would then need different interventions compared to someone whose first language is Chinese. 

Auditi Chakravarty: There is also the aspect of students who may be somewhat literate in their home language. Translating that to literacy in English. The work of Digital Promise, among others, has been really helpful and has helped shape and form meeting learners where they are, and approaching them from a strengths and asset-based perspective rather than a deficit-based one. if a learner doesn’t speak English at home and is struggling to read, how do we see their home language as a strength, and build a reading intervention on that?

Assessment for Good led by Dr. Temple Lovelace aims to transform formative assessment of skills supporting learning. such as Belonging, engagement, and self-efficacy. It aims to do so in a culturally relevant way, engaging learners and providing educators, and caregivers with valuable information. the program seeks to unlock new knowledge about how we assess these skills differently and deliver assessment seamlessly into instruction. This means frequent, embedded assessments rather than pulling students aside for tests.

Tom Vander Ark: I’m really excited about experience-embedded assessment. So much of this can move into the background. AI will be super helpful here. 

Auditi Chakravarty: Yeah, AI is a component of each of the three programs R&D. It requires a deep commitment to equity in each program, particularly in Assessment for Good where they’re methodically assessing the ethical elements of the technology. Instead of avoiding AI due to concerns, they’re doing the R&D to test and improve it making it work for their needs.

Tom Vander Ark: I’m an old project-based learning guy. I’m a Ron Berger devotee. I love how he and the EL schools, and to an extent the New Tech Network schools, have done a nice job of using formative assessment in project-based learning. I’m trying to decide if I’m excited about immersive environments, game-based environments, and simulation environments.

Is that part of this? Are you bullish on AR and VR and sims and games to advance formative assessment? 

Auditi Chakravarty: Yeah, one of the capabilities and prototypes Assessment for Good is exploring is games and game-based assessment. It’s early to know if I’m bullish, but we need to advance these technologies and tools by integrating them into our work and seeing what works. Kids are already familiar with this tech, so we should capitalize on that while increasing the evidence base for what works or not. 

Tom Vander Ark: I don’t know much about games, but Fortnite and World of Warcraft are pretty sophisticated in assessing some dimensions of player interaction. They build on that in an interesting. way. There’s some sophisticated assessment happening there. Maybe not labeled as ‘assessment for good’ but it is assessing.

I wonder about Roblox. I see my grandkids play it. It seems like an exciting creative environment. but I’m not sure about formative assessment there any thoughts?

Auditi Chakravarty: Yeah, I know a bit about what they’re doing. While I’m not sure how much formative assessment they’re doing, they’re partnering with many providers to develop those capabilities. There are researchers like Eva Baker, have explored game-based assessments for long-standing, durable skills hard to measure otherwise. At the College Board, we tried to assess creativity for AP Computer Science principles. but couldn’t build an immersive game-based platform. Even then, about a decade ago, the vision was clear. We’re about to name our next group of fellows which is also exciting. That’s coming up in the next few weeks.

Tom Vander Ark: that’s what I wanted to ask you about. You must have this long list of stuff that you’d like to dive into. What are the driving questions you most want to investigate? 

Auditi Chakravarty: For this fellowship, we framed three opportunity areas. First, we know that students are increasingly disengaged from school. What if every learner had high-quality, relevant learning every day?

Second, we know that multilingual learning has many benefits, what if every learner became multilingual? 

Tom Vander Ark: Aiming for early years? Elementary or K-12?

Auditi Chakravarty: Our charge is pre-K through 12. So it’s open-ended. We want to support the vision of our program leaders who see problems and have hypotheses, whether it suits early or later stages of learning.

Tom Vander Ark: You think about how AI can integrate those first two into a fully immersive, personalized curriculum, and it could be multilingual. That would have been impossibly difficult before.

Auditi Chakravarty: Our third area is, what if assessments eliminated inequity? 

More R&D is needed to think about assessment as a lever for learning with student-centered methods. What’s interesting is how these areas can overlap. We’ve seen proposals that cut across them, showing that these challenges are interconnected. There’s a lot of interest in relevant learning every day. Not just schooling. I’m excited to see the ideas that come from this call for proposals. We’ve received hundreds of applications and are narrowing them down to eight fellows, there are many worthy of funding, and hopefully, we can support more in the future.

Tom Vander Ark: Let me underscore the importance of the quality of the question. Framing three programs and driving questions, creates focal points for research which is critical. In healthcare. university-sponsored research is often driven by funding. In education, we traditionally push PhD candidates into research areas that haven’t been studied but may not be meaningful. We haven’t had a way of, as you said before, aggregating demand. or identified important questions to work on. Instead, we’ve pushed researchers to dark corners with little relevance and without inviting rising talent to address substantial questions in a collaborative way.

We haven’t done either one of those well. It seems your work is doing just that, surfacing important questions and inviting collaboration.

Auditi Chakravarty: Thank you. That’s the core of our inclusive R& D model, educators researchers, and product developers work together. Shaping both the research and the tools. One lesson we’ve learned that we put into this call for proposals is, encouraging applicants to apply with a partner who balances their skills. This might be a school leader, with a research partner or vice versa. even if applicants don’t, apply with a partner, part of our process is to help them build a balanced team. when diverse perspectives come together we can unlock significant breakthroughs. That’s what we’re aiming for.

Tom Vander Ark: Is funding a barrier? Do you need more money to do this well?

Auditi Chakravarty: Absolutely. 

Tom Vander Ark: Some federal funding wouldn’t hurt. 

Auditi Chakravarty: Absolutely. We’re lucky to have strong support from our current funders and we’re working to bring others on board. We’re now at a point where we have great examples from each of our programs to show what this work looks like, and why it’s worth supporting. I’m optimistic about demonstrating our capability to, attract more federal funding to support and expand this work. There’s a great appetite for solving problems differently, evidenced by the hundreds of people applying to our programs. We can only support eight fellows at a time, but there’s clearly much more potential and interest.