Retrieval Practice, with Pooja Agarwal

Bonni [00:00:00] What is it called when we have our students practice getting information out of their heads instead of getting information into their heads. Yes. It’s called Retrieval Practice. On today’s episode of Teaching in Higher Ed number 94, I get the pleasure of talking with Dr. Pooja Agarwal about retrieval practice.

Production Credit [00:00:24] Produced by Innovate Learning, maximizing human potential.

Bonni [00:00:33] Welcome to this episode of Teaching in Higher Ed. This is the space where we explore the art and science of being more effective at facilitating and learning. We also share ways to increase our personal productivity approaches so we can have more peace in our lives and be even more present for our students.

Bonni [00:01:00] I’m thrilled today to welcome Pooja Agarwal to the show. She’s committed to bridging gaps between research teaching and policy. She’s passionate about evidence-based education and she’s done a ton of research on retrieval practice and that’s why I have her on the show today. She’s been working on that for more than ten years in collaboration with distinguished memory scholar Henry Roediger and she also has earned in her prior careers an elementary teacher certification and has extensive teaching experience at both the K12 and university level.

Bonni [00:01:39] And I’m just so excited to have her on the show and I’m very grateful to James Lang who connected us. Welcome to Teaching in Higher Ed.

Pooja [00:01:48] Thanks for having me.

Bonni [00:01:49] Well I wanted to start this conversation out find out just how did you first end up getting interested in retrieval. How did that sort of merged into this whole area of research for you?

Pooja [00:02:00] For me it almost started in high school. But fast forward a little bit in college I became really interested in education and I was starting to pursue a degree in elementary teaching and a certificate for teaching elementary in sort of K12.

Pooja [00:02:18] And as I was going through this teacher training program at Washington University I also as many college students was taking courses in psychology and I took one course in particular in cognitive psychology and it just it literally blew me away.

Pooja [00:02:36] I had no idea that there is an entire science of how we learn. And you know here on one side of campus I’m taking courses about how students learn and here on the other side of campus I’m taking courses about how students learn and they’re very different.

Pooja [00:02:53] And I found myself drawn to almost both approaches to teaching and learning. But the research was just fascinating. There’s something about using the scientific method and figuring out how things work and how there are commonalities of learning across humans that I really just wanted to know more about.

Pooja [00:03:14] And so I purposely completed their teacher training program and was certified. But I also pursued a research career in retrieval practice and memory.

Bonni [00:03:23] Let’s start out with just how does retrieval work. What is it? What is retrieval and then what actually makes it work?

Pooja [00:03:32] A good example of what retrieval is and how it works which is a bit more simplified than learning in higher education classrooms is what I imagine to be a common scenario when you’re at a cocktail party or you’re networking with colleagues and you meet someone you say, “Gee. Hi. You know I’m Pooja. It’s great to meet you Bonni.”

Pooja [00:03:57] And then Bonni and I have a conversation and someone else joins our conversation. Hi Ed it’s nice to meet you. And we all continue. And then I get pulled away and invariably I might come back to chat with Bonni some more.

Pooja [00:04:10] And I’m sure many of us have had the experience where we have forgotten someone’s name we’ve forgotten Bonni’s name. Oh no. What do we do. This is incredibly embarrassing or of course trying to learn students names in courses and invariably forgetting those.

Pooja [00:04:27] And so from from talking with a lot of my friends and family who are in these situations it seems fairly common that people have a tendency to say names over and over and over. So hi Bonni. Nice to meet you.

Pooja [00:04:43] And I’m trying to hold a conversation in my head. I’m literally going Bonni Bonni Bonni her name is Bonni. Don’t forget it’s bought Bonni Bonni and we’re just sort of incapable of doing all that while having genuine conversations and trying to meet people.

Pooja [00:05:00] And so this kind of rehearsal idea is what we think of as as trying to keep information in our heads. Again it’s a way to rehearse information kind of like a phone number a student’s name something you need to pick up at the grocery store. Again a very simplified example.

Bonni [00:05:18] I’m actually glad though that you used that simplified example because it’s so powerful because one of the things that happens so much not just with our students but it happens with us as faculty the number of times I’ve heard faculty say I’m just not good at learning names.

Bonni [00:05:34] And I don’t want to take us off too much on the track because this might start to wade into the area of mindset. But part of this is just our own confidence about the true power of retrieval. Have you had instances where you can talk to faculty and begin to really demonstrate to them in some compelling way that actually this stuff really works and it works with names and networks that it can work you not just for your students?

Pooja [00:06:01] One key aspect of this again the difference instead of rehearsing over and over keeping things in but really pulling things out of mind and retrieving them. I think one first took is that it again it works in very simplified manners.

Pooja [00:06:17] A lot of people the vast majority of people I would say complain that they can’t remember faces and names but in practicing this type of retrieval it works quite well. There’s lots of research to demonstrate that retrieval helps people with faces names something they encounter everyday.

Pooja [00:06:34] And then of course there’s a wealth of research which is growing more recently about how the use of retrieval whether in courses or otherwise helps in everything from higher ed courses to medical education to learning for older adults retaining information.

Pooja [00:06:54] And I think the aspect of retrieval improves very simple types of learning but retrieval is also applied to really complex learning can be a great way of pulling someone into thinking about using this strategy.

Bonni [00:07:08] I would love to hear some of the examples of different disciplines that are using retrieval practices in their classroom. I know one of the tools that you have set up for all of us in a whole wide variety of disciplines is retrievalpractice.org and so would you just share some of the examples that you’re seen ritual practice be used in the higher ed classroom and also how he might read more about that on your site.

Pooja [00:07:32] In terms of higher ed classrooms. There’s a set of research on retrieval that is specific to college students and actually I would say most of the research over a trivial practice is with college students and a lot of the previous research is with college students.But in laboratory settings where we bring college students into a laboratory they sit down they look at computers and we ask them you know here you’re going to learn a bunch of vocabulary words.

Pooja [00:08:03] We want you to learn Swahili for instance. And students love. To sit in these types of experiments and learn Swahili words and then depending on the research question of course we can ask them a few days later what they remembered or tweak different aspects of what they remember how we present it and how we test them.

Pooja [00:08:26] So there’s a lot of research about how retrieval improves learning for college students the laboratory and in terms of academic disciplines that ranges again from foreign vocabulary learning to science material English literature material and history material and then again a growing set of research, more recently, on retrieval practice is actually implementing retrieval strategies in higher education classrooms to see if it really benefits student learning. Kind of in the wild so to say does retrieval practice is something professors and faculty can implement.

Pooja [00:09:06] But as is retrieval practice also something that students benefit from and so a lot of that research again has been showing a nice breadth across academic disciplines in particular the sciences and social sciences. One study for example that was recently published by Andrew Butler and colleagues involves student learning in engineering courses at Rice University.

Pooja [00:09:35] And in that case again retrieval practice strategies with a few other memory based learning strategies were implemented in engineering classrooms and the researchers showed fairly substantial and consistent benefits in real courses so not just sitting at a computer but with student students daily lives in weather and quizzes and stressors. All of those things were still involved but retrieval practice strategies continued to improve learning.

Bonni [00:10:07] One of the things I know from myself in implementing so much more retrieval into my classrooms these days is that students don’t always like it. The whole I’m have is great now. I’ve had this epiphany of how well this stuff works. I’ve looked at the research. It looks really solid. Students may not like it because sometimes we’re breaking a norm for them.

Bonni [00:10:28] They’re used to I got to come in. I’d sit down for my 50 minutes. I took some notes maybe… uhh… and my learning happened outside the classroom. And so this really does change it? What have you found that’s important for students to know about retrieval and how have you found that to be helpful for them in their learning process?

Pooja [00:10:49] I would say that for both students and faculty it’s important to know how exactly that retrieval is challenging and it’s precisely that challenge that improves learning demonstrated by almost maybe more than 100 years worth of research.

Pooja [00:11:09] So for instance inside the classroom students typically may sit listen to a lecture and then go home. Review their notes that they took in class read the textbook reread the textbook reread it again.

Pooja [00:11:26] Then cram and all of those strategies that we know from research a majority of college students use are again focusing on this idea of rehearsal or just thinking about things over and over and over again. This idea of cramming.

Pooja [00:11:42] And actually there’s research to show that cramming works in the short term. But what I think is important for both students and faculty is we want learning to continue in the long term and so in that way it’s it’s important for both students and faculty to know that yes retrieval is challenging but the payoff will happen on a final assessment.

Pooja [00:12:06] Even on a midterm exam or a final exam so it’s a little bit of a tradeoff sort of I’m going to stick to my taking notes and rereading book chapters because it makes me feel like I really know that information but invariably students will forget very quickly. Whereas, if if faculty in particular can help turn this around so that students don’t necessarily have the opportunity to cram.

Pooja [00:12:31] But they’re retrieving information in every course meeting or once a week or they’re taking cumulative exams that will help a little bit with students to recognize that yes retrievals challenging. But yes it also improves my long term learning.

Bonni [00:12:49] What are some mistakes that we as teachers often make when we’re trying to implement the retrieval practices in our own teaching?

Pooja [00:12:57] I would say that there are two mistakes in particular. So one goes back to this idea that retrieval is challenging. And I can I know from experience that it’s not only challenging for students but it’s challenging for faculty.

Pooja [00:13:14] When I was teaching some courses in education and in psychology it’s kind of a pain to put together quizzes or retrieval options for students and to sit and grade them and to slog through that extra work.

Pooja [00:13:30] And I think that although retrieval may seem challenging and there’s this tendency to want to abandon it these two mistakes are either to abandon it because it seems difficult to implement and or to abandon it because students don’t seem to be improving their learning right away.

Bonni [00:13:51] Talk more about that when you talk about that second mistake. I didn’t mean to take you on a tangent but I just was thinking about – I share so often with my students that when you change something about what you’re doing – your other skills are going to get worse for a while.

Bonni [00:14:08] An example I often use are that sometimes bells those service bells that they have in hotels. Sometimes those bells are used to help people correct for ums and uhs and other types of filler words in their speech.

Bonni [00:14:21] So if I’m using that with students we’ll talk about back to the psychology classroom which I learned so much in my undergrad as well. Where you go, “OK we’re going to try this. It’s a form of punishment but it’s not so bad. Every time that you say we’re going to ring this bell.” I’ve done it with professionals too with faculty as well. It’s tough you got to have a lot of courage to put yourself out there like that. And when that starts to happen it’s incredibly powerful but it’s so powerful that all of the other aspects of your presenting are going to get worse.

Bonni [00:14:49] Lose your train of thought you’re going to go holy cow this is so incredibly hard. I normally can stand up in front of a group of people and share my thoughts and I can’t do that right now because of how your twisting my mind right now. I know  I’m not using the correct technical term but that is so true that we abandoned it because it looks like it’s not working but it’s actually working so powerfully that I think sometimes those other things get harder. That other stuff that we normally would have been good at. Have you found that in some of your research?

Pooja [00:15:22] Oh yes there’s interesting research to show that even if students engage college students engage in retrieval practice this is in laboratory research they will still ultimately choose to abandon it. And I think there’s current research being conducted about how of course we can change this so that students don’t abandon things quickly with retrieval.

Pooja [00:15:48] But one aspect is that retrievals not only challenging mentally because it’s different from what students have been doing with notes rereading textbooks taking notes but it’s uncomfortable. If you’ve ever had someone ask you a question and you don’t know the answer it’s uncomfortable.

Pooja [00:16:09] And that’s essentially the key feature of retrieval is to ask someone for information. Ideally they understand that information but even if they don’t answer correctly they still benefit from that retrieval experience.

Pooja [00:16:25] And so I think having conversations with students about this is why and using retrieval in my classroom this is the research to support it. Acknowledging it’s going to be uncomfortable but by the end of the course I promise that your learning will increase at least based on that research.

Bonni [00:16:43] It’s kind of back to your start of the conversation about retrieval – and retrieving names because we want it all to be easy. We want it to be authentic. We want to seem so natural but learning isn’t always natural it really can be a challenging thing but if it’s important to us when we can and we know what really works we know that rehearsal doesn’t work anywhere near as well as retrieval is going to.

Bonni [00:17:06] Then we can start to have the confidence behind getting through that more difficult part. Speaking of getting through the difficult parts what is something that you either got wrong about retrieval or mistakes that you made when you first started researching that really speak to you still today and you’re still learning from.

Pooja [00:17:29]  I’m sure there are many. I’ll pick two. One is that a lot of the research I personally have been conducting a retrieval practice has been in classroom settings and specifically in K12 classroom settings and one realization maybe not a mistake but a realization is that applied research is really difficult. And in K12 classrooms and even in the retrieval practice literature in college courses retrieval practice may not always work. And that’s difficult. As a researcher. That’s difficult as someone who strongly believes in research practice really being a powerful tool to improve learning. And but of course having to to take that with a grain of salt and then figure out is that really a feature of retrieval practice or is it because we’re in the wild?

Pooja [00:18:29] Are we subject to snow days or subject to family illness? There’s something I find exciting about conducting research in the wild but it’s also a realization that it may not always turn out the way we had hoped.

Bonni [00:18:46] I don’t know if I mentioned this to you before we started recording but we’ve got two small children here at home and I can only imagine some of the background behind what you’re saying. Oh yes they’re quite unpredictable aren’t they?

Pooja [00:18:59] Yes they are. And in that vein because of the applied research but also the laboratory research. The second realization comes back again to sort of when does it work and when might not work. And it’s exciting to see educators being starved almost or excited for more information.

Pooja [00:19:18] Does this work in science? Does this work in math? Does this work with students who have been diagnosed with A.D.H.D.? There was actually a recent paper published about retrieval improving learning for students with A.D.H.D and it’s getting to a point where this research is growing but again it’s been a mistake or a realization that we’re sometimes assuming it’s going to solve lots of things in education. We’re really still trying to work out the kinks.

Bonni [00:19:47] What have you looked at as far as or even just reflected on yourself as a educator in terms of the need for us to know some core sets of information before we can really experience deep learning because that’s some of the pushback that I’ll get where, “Oh this memorisation stuff you’re talking about…” And that’s often the word that they use is great for if I’m trying to learn human anatomy but if I’m trying to study philosophy isn’t going to help my students because they need to go deeper than that. What would be your reaction to someone who thinks that retrievals not really for them if they’re in one of those disciplines they perceive as needing to go deep?

Pooja [00:20:29] That’s an important question to ask and it’s one that I’ve seen a lot of pushback about as well. This relationship between retrieval practice and basic memory and whether retrieval practice is able to improve deep learning. And in my dissertation I was able to look at that research question both in K12 classrooms and in college laboratories studies not in a college classroom unfortunately.

Pooja [00:20:57] And there was this relationship between what I call fact learning and higher order learning where retrieval practice in both studies with K12 students and college students retrieval practice improved fact learning – great. There’s lots of research to demonstrate that that’s the case. What was exciting was that retrieval practice also improved higher order learning.

Pooja [00:21:21] What I would consider to be very complex materials more complex them much of the research done up until that point. What I was also interested in examining was the relationship between the two as you put it to a sort of need to know all of those facts before I can move on to deeper learning or can I just kind of go straight to the deeper stuff and what I found was that retrieval practice improved higher order learning when the retrieval itself was higher order.

Bonni [00:21:54] Talk to me a little bit about that. What might that look like in a classroom?

Pooja [00:21:59] With at least in my research that I conducted. And again it’s limited. And it was only with college students in a laboratory setting but to extend it to college students in a classroom setting what was important in my study was that students be challenged with higher order complex questions during retrieval meaning on a short quick quiz after they read the passage from a textbook.

Pooja [00:22:25] And that complex quiz helps students to answer higher order complex questions on the final exam. If on the other hand students only took fact based questions about the same topic. But the quiz questions were fairly simplistic. Then students performance on the higher order exam was lower.

Bonni [00:22:51] And can you describe a little bit about what the higher order exam looked like. How is that different than fact learning kind of exam?

Pooja [00:23:01] The materials I used – I don’t actually know if if higher education courses still use those – I took the reading passages and then I created my own quiz questions based on passages from a higher education series called Taking Sides.

Bonni [00:23:20] Now I heard about that. Yeah.

Pooja [00:23:23] I remember from my freshman year of college and I really liked it. And so I wanted to use that material and in Taking Sides. Each chapter presents two opinions about a controversial issue. So for instance should the United States provide aid to Africa. Or is welfare harming society. And in that case you had to have an author who writes about yes welfare is harming society.

Pooja [00:23:53] And one author would write no welfare is beneficial for society and I really like these complex passages because they allowed me at least in a laboratory setting to look at how to structure questions that could be both fact based and higher order.

Pooja [00:24:09] So the fact based questions for instance were more around. Why is it that the first author believes that welfare is detrimental to society. Is it ABC or D.

Pooja [00:24:23] Because taxpayers shouldn’t have to pay for it or because it doesn’t improve the situation for society etc.. So that would for me that was sort of a fact based question. What was stated in the passage.

Pooja [00:24:37] Now I didn’t use fact based questions like in what year was welfare instituted. Yeah. Yeah. So they were still fact based but not enough names and dates kind of way and the higher order questions would be based more on how would that yes author so to say the author that thinks welfare is detrimental. How would that “Yes” author react if he or she lost their job? Who would they pay except unemployment v etc. etc.? And so those were examples of fact questions versus higher order.

Bonni [00:25:17] Oh interesting. Well one of the things I love just about talking with you Pooja is that you’re so cautious about making sure that you’re clear about this happened in a lab and this was in the classroom to make sure that you’re so precise about what the research really says.

Bonni [00:25:33] But also recognizing that stuff is messy. So if people listen they have maybe experimented with retrieval practice and maybe they fall into one of your categories you said earlier about they abandon it because it’s challenging or didn’t seem like it was working. Or maybe they’ve never tried it before.

Bonni [00:25:51] What’s a way that we could dip our toe into retrieval practice and just get started with it? I would love it if there are some some resources on your website that you might want to point people to as well in this part.

Pooja [00:26:04] Yeah. Retrievalpractice.org which launched recently and so I’m still adding more content. There is a section of resources which includes books articles and websites about retrieval practice so for instance a few books that either give an overview about retrieval practice.

Pooja [00:26:25] One in particular I highly recommend that was discussed in episode ninety two was this recent book that just came out small teaching by James Lang. The very first chapter is about retrieval and how to use it specifically in higher education classrooms and this resources page also includes sort of more conversational articles that have been written about retrieval as well as just highlighting a few recent studies on retrieval practice in STEM classrooms using concept mapping even in medical education.

Bonni [00:27:03] And what would you tell someone just in terms of tomorrow. I’ve got I’ve got a class. How could I even start even smaller than that. I’m not sure I’m ready to read a book but just one thing I can do in my classroom tomorrow.

Pooja [00:27:18] From my experience that my preference that one thing would be to give a very simple writing exercise at the beginning of the course as students as students walk in. So they’re shuffling their papers and getting seated.

Pooja [00:27:35] I have typically presented one question at the front in the classroom on a PowerPoint slide which can be as simple as what did we talk about the last time we met.

Pooja [00:27:47] And students I would provide students only about two minutes to write down everything they could remember about what we talked about in the previous course meeting and then facilitating literally again a very short two minute discussion so that students who are already in the mindset of great that’s what we discussed.

Pooja [00:28:10] Oh I forgot about that. So I’m glad the other student mentioned it and then you can move on to your lecture. That doesn’t require preparation that doesn’t require writing multiple choice questions and that doesn’t require grading.

Bonni [00:28:23] Yeah I’m glad you pointed out the grading thing because that can be such a stumbling block for people and none of this has to mean any grading at all. There’s nothing that says we can’t have them if if appropriate depending on what kind of retrieval whether they exchange papers with each other turn to someone sitting next to you.

Bonni [00:28:38] You’ve got it where it’s coming in. Use it a tool like Poll Everywhere dot com so they can visually see how people answered. I mean it doesn’t involve anyone turning to anyone at all but instead you can visually see what their input has been either around more qualitative or quantitative oriented types of questions so that’s really helpful. Is there anything I haven’t asked you about retrieval practice that I should have before we go on to the recommendations segment?

Pooja [00:29:04] I would have to say the the one thing would be to really focus on the retrieval meaning to get information out from students hence instead of focusing on getting information in.

Bonni [00:29:19] Yeah. I love that you actually have that on the very first thing that I see when I go to your website when we think about learning. We typically focus on getting information into students heads. What if instead we focus on getting information out of students heads that so important and I tell you it’s really transformed my teaching.

Bonni [00:29:36] And I’m so grateful for the work that you are doing and other people alongside you as well. This is the point in the show where we each get to recommend something.

Bonni [00:29:46] And this week I want to recommend something related to what you talked about. I don’t always but I really was inspired by your work and seen how many books it is showing up in is exciting. Many of the books that I’ve read has mentioned your work as you talked about.

Bonni [00:30:01] So in terms of my own experience working with retrieval practice I can’t recommend enough asking students to questions as we’re going through some kind of a retrieval exercise. Ask what is that we’re doing right now.

Bonni [00:30:17] And my hope is that they will say retrieval or we’re taking it from our brains we’re getting it out of our heads whatever they’ve come to symbolize the act of retrieval.

Bonni [00:30:27] And then the second question that I like to ask is and why are we doing it. Because as we both mentioned earlier it can be so hard to do retrieval practice. It’s not an easy thing for any of us whether we’re trying to learn names or whether we’re trying to learn anatomy or whether we are trying to study the great works of Plato and Aristotle. Why are we doing it. Because it’s powerful because it really works and helps bring the learning into the classroom and really ultimately saves us time and helps us have that deeper learning.

Bonni [00:31:00] So again I’m trying to recommend changing the culture in our classrooms by asking these two questions and having it be more comfortable to struggle and fail and get back up again once we fail inside the classroom instead of doing that all alone once we’re outside the classroom. And what is it that you would like to recommend on today’s episode.

Pooja [00:31:20] I would love just briefly to recommend the Web site again retrievalpractice.org. There are great resources for more information. There are brief overviews about what retrieval practices and how it helps and there’s a very short guide downloadable for free about retrieval practice.

Bonni [00:31:42] And if people want to get in touch with you what’s the best way for them to do that?

Pooja [00:31:47] The best way is probably through a retrievalpractice.org.

Bonni [00:31:53] That’s how I got in touch with you.

Pooja [00:31:55] I recognized that my name is difficult to say and spell and so you can go to my personal website poojaagarwal.com which is currently under construction. You can also find me on Twitter @poojaagarwal.

Bonni [00:32:11] I love that you say that. I married in to the last name Stachowiak and my husband Dave recently sent out an email to all of his podcast listeners about his early business career at a business that failed and his one of the many reasons he cited was having the Web site involving the names Stachowiak and expexting anyone to be able to spell it. Oh oh I love it.

Pooja [00:32:36] Well I tried to marry out of my last name but my husband’s last name has seven letters and it’s just as confusing for it’s Rickert – it’s confusing for people as well.

Bonni [00:32:48] Well more power to people with confusing last names because at least when people are trying to retrieve them it will be easier because we had to struggle to get there right.

Pooja [00:32:56] Yes.

Bonni [00:32:57] Well I just appreciate so much to you coming on the show and investing your time in our audience and just look forward to continued conversation. Now that I’ve been connected with you I really appreciate your time,Pooja.

Pooja [00:33:08] Thank you so much. I really enjoyed chatting with you.

Bonni [00:33:12] It was such a pleasure talking with Pooja today and I do want to mention quickly that one of the other nice things about retrieval practice is that it saves us time and we spoke about that a little bit.

Bonni [00:33:25] But speaking of productivity which is one of the things we talk about here on teaching in higher ed it really can save our students and also us as teachers. A lot of time. And speaking of saving time if you don’t want to have to remember to go find the episode notes where you can click on the links that Pooja talked about. You can find those at teachinginhighered.com/94.

Bonni [00:33:47] But you also can subscribe to the weekly e-mail newsletter. And that’s just an email it will come in your inbox every week with the show notes and all those great links as well as an article on most weeks about teaching or productivity written by me and I just so much thank you for listening.

Bonni [00:34:03] I’m starting to think about summer time and what these episodes might look like perhaps different or perhaps the same in the summer and be watching for a survey from make up some hoping to get your feedback on that.

Bonni [00:34:15] But if you have any feedback from me on that question or about the show in general please feel free to always get in touch via Twitter @Bonni208. Or get in touch at teachinginhighered.com/feedback so much. Thanks so much for listening and I’ll see you soon.

Teaching in Higher Ed transcripts are created using a combination of an automated transcription service and human beings. This text likely will not represent the precise, word-for-word conversation that was had. The accuracy of the transcripts will vary. The authoritative record of the Teaching in Higher Ed podcasts is contained in the audio file.