Learning and Assessing with Multiple-Choice Questions

Bonni: [00:00:00] Today on episode number 155 of the Teaching in Higher Ed podcast, Jay Parkes and Dawn Zimmaro share about learning and assessing with multiple choice questions in college classrooms.

Production Credit: [00:00:15] Produced by Innovate Learning, maximizing human potential.

Bonni: [00:00:25] Welcome to this episode of Teaching in Higher Ed. This is the space where we explore the art and science of being more effective at facilitating learning. We also share ways to improve our personal productivity so we can have more peace in our lives and be even more present for our students.

Bonni: [00:00:53] Hello. I’m Bonni Stachowiak and I am excited today to be welcoming not one but two guests to the program both here to talk about Assessing with Multiple Choice Questions in College Classrooms, a book that they wrote together.

Bonni: [00:01:09] Let’s start off sharing a little bit about Jay Parkes background. He’s the Professor of Educational Psychology and has been the chair of the Department of Individual, Family and Community Education since 2010 at the University of New Mexico in Albuquerque. Jay teaches graduate courses in classroom assessment educational measurement introductory and intermediate statistics and research design. He’s conducted research in classroom assessment in public schools, at universitie,s and at a medical school.

Bonni: [00:01:43] And our second guest is Dawn Zimmaro and she leads the Development of Pedagogical and Assessment Solutions related to innovations in blended and online learning models. Dawn’s primary professional interests are measuring the effectiveness of new delivery models on student learning classroom assessment using learning data to inform instructional and learning practices and competency based assessment. Dawn has been working with the university faculty and administrators on higher education assessment issues since nineteen ninety eight. And Dawn is the Director of Learning Design and Assessment at the Open Learning Initiative at Stanford University. Jay and Dawn, welcome to Teaching in Higher Ed.

Jay: [00:02:36] Thank you. Great to be here.

Dawn: [00:02:38] Thanks for having us.

Bonni: [00:02:40] I have a confession to make to both of you and also to everyone listening that Jay when you first contacted me about coming on the show to talk about the book about multiple choice questions I know that this is important for us to study all aspects of our teaching and so many of us use multiple choice questions in some form of assessment or another. But I wasn’t at all prepared for how interested I would be in your book and I read it.

Bonni: [00:03:09] It was interesting from the first page all the way till the end. And I have to I guess I want to start out with a question for Dawn. Do you get this kind of reaction from other people where they’re just expecting gosh multiple choice really are we going to talk about this and this is this something we’re going to do. I mean there’s sort of seems like there might be this negative reputation about multiple choice questions. What can you tell me about that?

Dawn: [00:03:33] Yeah absolutely. I think that’s one of the first direct reactions we get not only from instructors but from students as well. Either Jay is been in front of the classroom as a teacher himself and I’ve been a faculty consultant for years and from both sides of the spectrum you know I have to use multiple choice questions in my classroom what can you do.

Dawn: [00:03:57] What advice can you give me about how to best use them in the classroom? And so there’s is already this negative perception of you know this is an assessment strategy I really don’t want to use but for a variety of different reasons I’m going to have to add this the reputation is often we often summarize it.

Dawn: [00:04:16] We use this quote in our book where we say tests don’t hurt students. People with test hurt students. You know it’s not the multiple choice question that is problematic it’s how the assessment can be used.

Dawn: [00:04:29] And one of the big issues is when they’re not written well when they’re written poorly you know students get frustrated. You know as test takers and you know getting these assessments and then the instructors hear complaints from students and kind of throw their hands up and say you know know I’m teaching a large class I have to use this assessment format.

Dawn: [00:04:50] And so it kind of becomes this vicious cycle. And what we’re trying to do and the information that we we try to do in our own practice is and share with our colleagues is that when done well they can be effective learning and assessment tools.

Dawn: [00:05:05] And that’s really the focus of the major part of what we wrote was how do you actually you know use this as learning tools for yourself learning tools for your students.

Dawn: [00:05:14] And then as a part of the process work to assess you know student mastery and student and different kinds of thinking skills and critical thinking skills with a format that traditionally has been viewed as you know simple recall of information and very limited in its applicability.

Dawn: [00:05:32] So what we’re trying to address is just getting at that like that negative reputation sort of that sense that you know this is not really effective or a worthwhile assessment it’s sort of a necessary evil. So we’re trying to break that stigma and you know bring a different story to ways in which we can do assessment in really effective ways to make us learning and assessing with students more enjoyable for the instructor and for the student.

Bonni: [00:05:58] You said something there I don’t want people to miss and that is that we can use this for assessment but we also can use it for learning. And we’ve had a number of episodes in the past that are linked to in the show notes about retrieval practice and of course multiple choice questions can be such a wonderful way for us to do retrieval practice right in the classroom.

Bonni: [00:06:20] And as you say in your book and also as all of the retrieval practice experts talk about how powerful it can be that students get feedback right away and just how powerful that can be in terms of their own learning and making those connections with those neurons and helping them recall the answers a little bit more before I get too carried away.

Bonni: [00:06:39] I’m just really interested in all of this and ask of you both so many questions I might stop myself for just a minute and ask Jay Why did you decide to focus on multiple choice questions?

Jay: [00:06:52] We like multiple choice questions focus on them because they’re everywhere. Folks use them all the time not just in their teaching not just in classes but there are a fundamental building block of a lot of licensure exams so students and graduates are going to encounter multiple choice questions.

Jay: [00:07:10] So many instructors we work with particularly in professional fields like medicine and law know that their students are going to encounter these questions on their license your exams and so they want to make sure they see them in their in their education.

Jay: [00:07:24] So there there are ubiquitous measurement tool regardless of how people feel about them. They’re also one of the most well-researched measurement tools available. They’ve been around for most of a century in education in the United States and there we understand them very well we know how they work.

Jay: [00:07:46] There’s a huge research base on them. One of the key things why a lot of people end up using them is they’re very economical They’re economical in terms of testing real estate that in say 25 minutes or 30 minutes of testing time you can ask a lot of multiple choice questions of students where with something like a even a short essay you can’t get nearly as many topics covered as you could.

Jay: [00:08:09] One of the things that I think happened over the last eight or ten years is with budget pressures in institutions of higher ed class sizes have crept back up. And so some instructors have felt forced to use them where before they were using other forms of assessment that more open ended kinds of things.

Jay: [00:08:29] One of the reasons we focused on multiple choice at this point too is the technological applications have really blossomed and help us improve the development of the questions delivery of them scoring them. You can add video clips now into multiple choice questions and so forth through technological applications.

Jay: [00:08:51] So it’s there really they’re everywhere. They’re widely used and we can now use them more fastly and flexibly than we ever have before so they’re a great tool in any instructor’s tool box.

Bonni: [00:09:06] I love how you phrase that in terms of it being a part of a tool box that has a lot of other tools in it. And one of the things I always get really sad about is when I hear someone that you talk about the increase in class sizes and you talk about this feeling pressure to use them because of they’re just the more transactional nature of things so I do get really sad when we give up as faculty and just say this is all we’re going to do. I don’t think it’s going to take us far enough.

Bonni: [00:09:32] But I do feel like they have gotten a bad rap and therefore perhaps we’re not focusing on the opportunities when they really are the right tool. And then if we’re going to use them let’s actually use them well and that’s what I really appreciated about your book and Jay I wonder if you would just spend a little bit of time telling us what how would we know if our multiple choice questions were any good or not how do we construct a good multiple choice question.

Jay: [00:10:00] Sure. I think the first thing we need do is reintroduce ourselves to the multiple choice question. It’s one of the things we all know but we probably haven’t thought of in detail that it’s almost like a social media friend.

Jay: [00:10:13] The multiple choice question has a couple of fundamental parts to it. The first is that part that gets presented to the examinee that sets up the task for them and in model choice parlance that’s called the stem is now called a question because sometimes it’s a statement.

Jay: [00:10:31] So it’s called a stem and then the stem is followed by a series of options. And there are two kinds of options. One kind of option is the right answer. Or the one that students are supposed to select if they know the material and that’s called the key and then the other ones are called distractors.

Jay: [00:10:51] And that basic structure is the multiple choice questions so good multiple choice question has a clear stem that tackles just one learning objective and then a series of options all of which relate in some way to the stem that could be reasonable answers. But one of which is probably the best or the right answer.

Jay: [00:11:14] And so part of what makes for a good multiple choice question is one that is clearly learned the learning objectives. It’s one that assesses just one thing. It’s awfully hard multiple choice really helps us there because it’s it’s hard in some respects to build multiple things into a multiple choice.

Jay: [00:11:34] And then another piece of a good multiple choice question I’d like to talk to faculty about that is just just ask a straight question. No tricks no traps no funny games. Just ask a question. The goal in any assessment in a classroom but particularly here with more of a choice is to find out which of the students have mastered the learning objective objectives and which have not.

Jay: [00:11:59] And so that’s why we’re asking the question there is in the research literature on humor and assessment there’s some disagreement as to whether humor is good humor is bad. I think the fundamental issue is it needs to integrate into your broader teaching.

Jay: [00:12:16] If you’re if you have your own sense of humor and you use it daily in class and you want to move that also into your assessments that’s fine. If you’re not doing that. Don’t try to be funny on the test. It confuses students concrete issues for folks for whom English isn’t their first language.

Jay: [00:12:36] And and they’re wound up in and tight anyway. They’re not in the mood to get a joke. So a good multiple choice question. In the end is it is well constructed line with learning objectives and tackles one thing and doesn’t try to get cute.

Bonni: [00:12:55] You talked about there being two types of options there’s that key that’s the correct answer. And then there are the distractors in a moment I’m going to ask Dawn a little bit more about those distracters.

Bonni: [00:13:05] But Jay let me just ask you to tell me what went wrong. I know what went wrong but that’s because I read your book but I’m sitting down I’m administering a test recently and I wound up having three different students come up to me. They are allowed to ask me questions during the test because I have English as a second language learners regularly. I say I can be your human dictionary. I don’t want you to use your phone to look a word up but as long as me giving you the answer to a vocabulary word.

Bonni: [00:13:34] If it was the introduction to business and I say what’s profit. Well that’s that’s a vocabulary that’s related to a learning objective so I’m not going to tell you what that means but if it’s something not related then I’m happy to do it so I had three students come up to me.

Bonni: [00:13:48] “Can you tell me what paramount means? What is paramount? What does paramount mean?” Jay what went wrong in that particular test construction for me for the intro to business where the word paramount was not a learning objective.

Jay: [00:14:02] Well what it sounds to me like part of my notion of just ask a straight question is don’t use language that is officious or pretentious or or trying to sound more learned use the language that is appropriate to the students and their level of education and where they are in the field.

Jay: [00:14:25] And you gave a very good example of in business. Profit is an important word. They should know that word. It’s reasonable for us to expect them to know that word and its content relevant. It’s relevant to the learning objectives in the course.

Jay: [00:14:42] And so that’s the we have to use that word a word like paramount depending on its setting and so forth is perhaps not as simple as it needs to be to get to that one idea that the question needs to be about.

Jay: [00:14:58] So linguistically, keeping our questions as simple as they can be is important because we want to stay out of the students way and let them show us what they know we want them getting the question wrong because they don’t know what profit is not because they don’t know what Paramount is.

Jay: [00:15:17] And so which again is it that’s not dumbing it down. Dumbing it down would be artificially lowering the language standards. But this is being really clear about what language is content relevant and what language is not and keeping the language that’s not out of students way. There’s another reasonably well known rule and mobile choice writing that says avoid window dressing and window dressing in that context is superfluous language.

Jay: [00:15:49] Those are the things that are in the question somehow but don’t actually aren’t actually necessary to know what students know and those kinds of things just confuse students more and more and that’s different from there’s kind of multiple choice question called a vignette based question where you you have a short scenario is the stem and then a question and one of the learning goal might be sifting relevant from irrelevant information.

Jay: [00:16:20] Well in that case there has to be irrelevant information in the in the stem so that students can do the sorting. And yet that’s only relevant if the learning goal is to find relevant from irrelevant information.

Jay: [00:16:36] So one of the keys is just to keep the language as simple as it needs to be. One of my favorite studies in this area was done with elementary kids in math and they were they were testing both English language learners and native English speakers in math and they deliberately lowered the reading level of the math questions so that they were well below the grade level of the students they were testing and they were hoping that would help the English language learners.

Jay: [00:17:05] And the surprising result was everybody’s math scores went up because they’ve got the language out of the way and let the math come through. So it’s a it’s a well-known research based strategies to keep the language out of the students way so dawn we get to you.

Bonni: [00:17:26] Now to talk about these distractors. Tell us one more time what a distractor is and then tell us why they’re so important to get right.

Dawn: [00:17:35] Sure Bonni. So as Jay mentioned one of the key pieces a multiple choice question are the options and the distractors refer to the incorrect options as a part of the multiple choice question. And I’ve actually had some questions from faculty members as to why we use the term distracter and you know essentially it’s a good measurement. We we do want students who don’t know the concept to be enticed.

Dawn: [00:18:04] Or as the name implies distracted by an incorrect option and select it because they if they haven’t really mastered the learning objective that’s contained in that question then they’re probably more susceptible to pick an option that is based on a common mistake or a common misconception that they hold.

Dawn: [00:18:22] And so the distractors aren’t just incorrect answer choices but should be based on those mistakes or misconceptions that students you know the questions they ask in class or in conversation with students that the errors that you see sort of as a part of the classroom discussion or homework assignments you know including those is a part of a multiple choice question is good measurement good assessment and it actually helps with diagnostics.

Dawn: [00:18:49] So students who pick a particular misconception if we’re able to give them individualized feedback or feedback to an entire class we can say you know Option B was the incorrect answer choice and here’s why. Here’s the misconception that this is addressing.

Dawn: [00:19:06] And similarly if we’re looking at performance of an entire class and a large portion of the class is picked picked a particular distracter maybe we didn’t teach it well enough or maybe there’s a misconception that I need to go back and read instruct on because if 30 40 percent of my class are all picking that same distractor there might be an issue with the way it was presented either in a textbook or other in search resources or the way we talked about it in class.

Dawn: [00:19:35] And so there’s some really helpful information and diagnostics that we can get from the way from the distracters and how we actually build them and I do want to point out that there are two common distractors the faculty members often right that are not good measurement and those are using none of the above.

Dawn: [00:19:56] And all of the above as answer choices and this is I would say I don’t know if Jay would agree with this is the number one question when I’m giving workshops or talking about effective assessment strategies and multiple choice questions almost invariably I get asked this question every time why should I not use those two options as distracters.

Dawn: [00:20:18] And in the case of none of the above if my learning objective you know to use your example Bonni is that students should understand the concept of profit and the correct answer is none of the above have I really assessed whether or not they understood the concept of profit.

Dawn: [00:20:34] They can eliminate negative examples of that definition but I really haven’t assessed whether or not they’ve mastered the definition of profit. In most cases though none of the above is used as a distractor incorrect answer choice. And students know that it’s sort of a throw away option. Rarely is it the correct answer. And so they they most often can easily eliminate it.

Dawn: [00:20:59] So as Jay talked about and we also talk about in terms of item writing the number of distracter there’s no hard and fast rules. Research has shown you know above four or five answer choices it’s really hard to write good enough at distractors that discriminate pretty well that students actually pick them.

Dawn: [00:21:20] And if some some research has also shown that your theory may be sufficient enough if again the fourth and fifth options really aren’t plausible. If you’re going to put none of the above is one of the answer choices then it’s probably best to only have three choices.

Dawn: [00:21:37] Similarly we we often see faculty members use all of the above as one of the choices they want to know if students can see if there’s a series of elements that came together in a list. The problem with that in a multiple choice format. There are other formats where that may be more appropriate but in a multiple choice question if the correct answer is all of the above let’s say there are five answer choices and it’s the last one.

Dawn: [00:22:03] Students only need to know that two of the four in the list are correct to know that all of the above is the correct answer. So we’ve really only assessed partial knowledge. We we we don’t know which one students actually get that just that they at least can identify two in the list or are correct.

Dawn: [00:22:22] Similarly if they can eliminate one of the options in all of the above is in the list then they can also automatically eliminate all of the above. So now students are sort of doing their test taking strategies and now instead of you know five answer choices they’ve eliminated one plus all the above. Now they could guess from three even without having much knowledge of the concept.

Dawn: [00:22:44] And so those distracters are really helpful they’re just they’re good measurement when the choices are written based on common misconceptions they can Bryde diagnostic information for both individual students as well as to you about groups of students and what choices they’re picking and they should all be plausible choices and not based on you know none of the above or all of the above strategies.

Bonni: [00:23:08] I had heard previously about whether the things that Jay mentioned previously don’t use humor in the distractors. I mean I think most of us probably are aware that that’s not the richest test taking or or even in this experience.

Bonni: [00:23:24] But one of the things that I think is important to ask you about is there are some times where it’s easy for me to formulate these distractors. An example might be there are four functions of a manager. There’s planning organizing leading and controlling and I can put a question up there.

Bonni: [00:23:45] And the goal the learning outcome for me would be that they could identify which form of management was being described. But there are other times where it’s much more crystal clear what the right answer is.

Bonni: [00:23:58] And I know I’m just kind of making it to the distractors I know I don’t do a good job always in those cases of coming up with these good distracters that are based on common misperceptions and what I think I’m hearing you say is that I’m way better off to slow myself down and really reflect on that and try to suss out what might be better distractors than just making up a word because I do that’s it is I don’t know if this is terrible but I’ll admit it on my show it’s ok I admitted failure before.

Bonni: [00:24:29] But sometimes I’ll just make something up that sounds plausible to see if they can distinguish between the actual answer and the thing I made up that wasn’t plausible but I think I’m hearing you say that that’s not a good practice.

Dawn: [00:24:41] Jay can also comment on this as well. But I would say that that is you are correct that is not a good practice and one check that you can do is an instructor if you’re having difficulty you know reading those distracters or those incorrect options. Can you explain why that answer choice was wrong.

Bonni: [00:24:59] Mm hmm.

Dawn: [00:24:59] So it’s. So even if you’re not providing feedback either you know in terms of written feedback or if you’re using technology to deliver feedback to students. But if you had to write to the student why that answer was wrong your explanation would be this is a nonsensical made up word.

Dawn: [00:25:18] And if that is the reason why you’ve selected that as distractor then that’s probably not your best strategy. There should be some rationale behind why that term or concept is related to the question related to the learning objective.

Dawn: [00:25:31] And you could say like you said the four practices of management you can say here’s why this is not an example of this particular practice. Right.

Dawn: [00:25:40] And so I think that’s a good like you said slow down you know don’t force there’s no hard and fast rule even across questions if at most your questions have for answer choices but you can only come up with three plausible ones. That’s better than trying to either use humor or make up a term that might sound reasonable. If you can explain it in a in a in a rational way that really addresses the learning objective then it’s probably not a good distractor.

Bonni: [00:26:07] This was really helpful to me I mean so much of the book was incredibly helpful to me but just because I want to not play games when it comes to testing like you said earlier Jay, I want to just have a high regard for my students and it can be stressful to take a test. I don’t like to take tests either and I’ve been at this for a long time and I don’t like to get things wrong. I mean it’s a common human thing to not want to fail or be wrong we have to fight against that.

Bonni: [00:26:33] And so I never want to play games or take it lightly that you know that that test is not something that can be stressful for them and how to make it as low stress as I can. Well still really measuring those learning outcomes.

Bonni: [00:26:47] There’s always going to be some stress and some of that’s actually good stress when we’re under stress I get stressed when I do this podcast but it’s a good kind of stress it brings energy right so it’s not that all of the stress is bad but I don’t want to be playing games for them. I want to really respect them and their own learning process.

Bonni: [00:27:01] And Dawn what you just said to me I really will take that away as far as can I explain why the answer choice was wrong. And if the answer was I made it up hahaha it’s kind of a little disparaging It’s doesn’t represent how I want to be as a teacher I can do better than that.

Jay: [00:27:18] That’s an excellent insight and I think the other way I’d recast it is not so much an error as it is a missed opportunity. What else can you learn about your students. And the answer may be as Dawn said nothing else. So why spend the ink on an option that isn’t going to tell you anything about what your students know or don’t know.

Jay: [00:27:40] So when I’m thinking about distractors is it’s not so much about doing it wrong it’s about capitalizing on this opportunity that we have in this question to learn something about what they do or don’t know about the learning objectives.

Bonni: [00:27:55] Oh I really liked that that that puts I think less. Less for me on the pass fail grade. And just like you said you thought there wasn’t much to do with toys. Not until I read your book and now I’ve really enjoyed how much you’ve challenged me to rethink some of these. Well before we go on to the recommendations segment I don’t want to miss out on two questions are all kind of move us along.

Bonni: [00:28:18] One is for Dawn. Tell us a little bit about how technology has allowed us to capitalize on these multiple choice questions.

Dawn: [00:28:27] So I think Jay alluded to in sort of his description of you know why why why are we focusing on multiple choice questions and I think one of the changes that we’ve seen in the last decade is the ways in which we can leverage technology to help support the entire assessment cycle instruction assessment and feedback.

Dawn: [00:28:50] And part of that loop in ways in which we can both create the questions themselves we can deliver questions more quickly and sort of in 24/7 access to students so we could give formative assessments you know low stakes assessments where students I think you mentioned earlier retrieval practice right.

Dawn: [00:29:11] Students get up multiple opportunities to engage with the material in and get feedback immediately on what they’re doing and they can do that outside the classroom before they come to class. They could do it actually in higher stakes situations with technology they can take assessments on smartphones and tablets.

Dawn: [00:29:30] So I think that technology has really expanded our ability to do some assessments and diagnostics in ways that we hadn’t been able to do in the past.

Dawn: [00:29:41] And one such strategy I think that we’ve seen as we can actually use technology to scale up your questions and create item pools so that. Which are you know a bank of questions that we use across different semesters and you can have students take assessments whereby one student is getting you know one question on a learning objective and a different student who may be sitting right next to them in a computer lab could be getting a very different question.

Dawn: [00:30:10] But again on the same learning objective and so our ability to create item pools in I own banks with technology and then deliver you know randomize questions and to deal with some security issues depending on how you’re doing your assessments so in a low stakes environment maybe you don’t care if students collaborate and work together.

Dawn: [00:30:31] But in other high stakes environments maybe they’re you’re teaching an online class and you want you know your assessments are kind of giving or giving them a course credit and so you can put some security measures in using technology. And also so that’s outside the classroom but also inside the classroom we can deliver formal assessments.

Dawn: [00:30:53] As I mentioned sort of with a smartphone or tablet or you know some faculty members of purchase clicker systems whereby you can say OK we’re going to check on management strategies and you know pose a multiple choice question you on the screen and students all submit their answer in a clicker question then you can see the distribution of students in your class in real time and say wow half of you really didn’t you know this is the wrong.

Dawn: [00:31:18] Everybody picked C and this was the wrong management strategy. Let’s talk about why. So that sort of real time feedback to you is the instructor and the students in the classroom that we hadn’t been able to do before I think is really advantageous way in which we’ve leveraged technology.

Bonni: [00:31:37] And Jay, for our last question and this part of the show how can we involve our students.

Jay: [00:31:43] Well it’s important to involve our students. After all the whole goal here is learning not assessing. We want students learning. And one of the things I love to tell people is that the Latin root of the word access is an idiot or a which means to sit beside.

Jay: [00:32:00] And that’s such a different mental picture of where we should be relative to our students we should be beside them as we assess them not in a confrontational position in front or facing them or whatnot.

Jay: [00:32:13] And I think that’s important because it it gives us a way to think about. There are folks who love to think of when they give a test they think like a police officer or a gatekeeper rather than thinking as a facilitator and an encourager and a coach and a teacher.

Jay: [00:32:31] And there’s a ton of strategies tactically that you can use to involve students. So if you think about the classic study guide for an exam my study guides tend to be much more involved at the beginning of the semester than they are at the end of the semester. Because I’m I’m scaffolding for supporting the students more at the beginning to understand my expectations.

Jay: [00:32:55] When I say some of my multiple choice questions are going to have applications go to involve you applying what we’ve learned to real world situations. What exactly does that look like. So I give them examples of multiple choice questions so they can see it and see what I mean by application.

Jay: [00:33:15] So there are just so many things you can do. There’s a whole set of techniques around collaborative testing where you can have students working together in various ways on tests that are nervous some people but there are ways to maneuver through the concerns and the questions about how to do that.

Jay: [00:33:32] So there are ways even during the test to involve students in their learning and assessment together at the same time. And then after the test they’re things like have them write about the questions they missed and do activities that come back as feedback.

Jay: [00:33:50] But could be broader activities about well where was this material on the test. In the book in the lectures in the course site to help them understand how the assessment and the learning tie together and are really just different complexions of the same thing which is are they mastering the material or do they know that they need to know to move on to the next course or to move into the professions.

Jay: [00:34:16] So one of my favorite ways to think about this is where can you pull professional ethics or professional skills into the assessment itself. One of my favorite examples that we write about in the book is a professor teaching an environmental science course and he has a no leg work policy. Why does he have no late word policy.

Jay: [00:34:44] Because if you’re an environmental engineer on the job and you submit a bid on a contract two days after it’s due. You don’t get the contract but he explains that to his students and he says I have a late policy because I’d rather have you lose some points in my class than lose that contract. Perhaps your job two years from now.

Jay: [00:35:03] So pulling what’s important in the profession in the way we do our assessments I think is an awesome way to make assessments more meaning for students and to involve them in this work.

Bonni: [00:35:14] That is such a more student oriented way of having high expectations and we have talked about that before on the show as far as just having a student centered teaching philosophy doesn’t mean that we lower our expectations. We can have superbly expectations and then have our students delight us and even exceeding what we ever thought they could do.

Bonni: [00:35:35] But this transparency is so vital in explaining this is why I don’t allow the work and it’s actually to serve you to build good habits. I just. That’s just fabulous. I love it. Thanks for sharing that example.

Bonni: [00:35:47] And this is the point in the show where we each get to give some recommendations and I will try to be concise on mine because I want to leave plenty of time for the two of you. I was so inspired by a Web site.

Bonni: [00:35:59] And of course that’s a free open online course coming this summer from a former podcast guest by the name of Mike Wesch. He has built an anthropology 101 course and I’m gonna post a link to it in the show notes. It is absolutely gorgeous as far as Site Design, tons of beautiful images tons of wonderful videos it just begs you to keep scrolling and scrolling and cooking and being inspired and encouraged and challenge.

Bonni: [00:36:28] He has the whole being framed around 10 lessons very rich media based 10 challenges. An example of some of the challenges would be talking to strangers try something new. The UnThing experiment. Humans of my stuff hero stories. Your manifesto.

Bonni: [00:36:49] And then for each one of those 10 challenges he has ten showcases where the students actually submit their own work and their demonstration of learning of these various challenges that are all based on his 10 lessons.

Bonni: [00:37:03] It’s just a wonderful wonderful site. He’s a brilliant teacher and I think any of us that would go have a look at anth101.com or again linked to in the show notes will just be inspired and left wanting to do more with our teaching. And Dawn, I’m going to pass it over to you for your recommendation.

Dawn: [00:37:23] Thanks Bonni. I have two things that I would recommend or one pertains to some of the work that I do in higher education. It’s a weblog by Michael Feldstein and Phil Hill called e-Literate and it addresses topics around teaching and learning in higher education specifically those impacted by technology.

Dawn: [00:37:45] And I found the post by the CO publishers and some of the featured articles there to be really informative as we as educators and folks to work with faculty and faculty development as we try to sift through the landscape of what’s happening in higher education and educational technology. It’s often feels like we’re inundated by choices choices about ways in which we can help support our learning and we can help students social connections and we can engage in research around educational technology in and kind of filtering through ways in which to most effectively use those educational technologies.

Dawn: [00:38:28] And it’s sort of distilling some of the research and applications. I found this web site to really be accessible to the educator and faculty members who are trying to figure out you know where can I go for some advice about how you know how this applies to what I’m trying to do in my practice and most a more recent article that I found particularly helpful is can there be a microscope of the mind that really looks set machine learning and cognitive science and distills it into a digestible format of saying how how do we know how students learn. What does the research say about that and then how does that affect what I do in it in the teaching teaching. And what I do in the classroom. So I think that particular resource and it said mfeldstein.com is where you can find that weblog.

Dawn: [00:39:22] And then the other resource is more about sort of personal and professional growth that I’ve used in my past in various leadership positions. And it’s called The First 90 Days by Michael Watkins.

Dawn: [00:39:35] It’s geared towards executive level leaders but it’s also a placable I think anyone coming into a new management position about how do you define your role in thinking about those first three months on the on the job. How is it that you want to set yourself up for success.

Dawn: [00:39:51] And also how do you work with those around you to support your you know personal and professional growth as a manager and as a leader in the organization. And really those those first you know first 90 days on the job are where you establish your relationships where you sort of set precedent for how you’re going to engage as a leader in the organization. I found some helpful tips about how I could use that time that transition time when taking on a new position to really be successful in what I’m trying to do.

Bonni: [00:40:25] It’s such a wonderful resource. I’ve recommended it so many times it’s it’s fabulous and I’m so glad you brought it up on the show because we’ve never had it come up before so thank you. And Jay let me pass that over to you.

Jay: [00:40:37] Thank you Bonni. I want to recommend the podcast Manager Tools and it’s sister podcast Career Tools and their manager-tools.com. They focus on very practical advice. It’s not management theory. It’s very corporate focused it’s meant for folks in a business context but so much of what they discuss transfers so well into higher education. I use a lot of their tools with my advising with my own students and I use it with colleagues and I use it in my own work around the university day in and day out.

Jay: [00:41:12] They’ve got very concrete practical advice on things like how to run an effective meeting or how and when to write thank you notes. You’d be surprised. There’s a fabulous podcast about how and when to write thank you notes. They have covered a lot of other routine activities and they’re just always fabulous and they’re there. Method is to talk about. Here are the three things you do about this on Tuesday morning. It’s not theoretical it’s not high minded stuff it’s real practical. They have a 45 minute podcast on how to shake hands. It’s just fabulous stuff.

Jay: [00:41:52] So I ask students if a student says to me would you write me a recommendation. I say you go listen to the career tools podcast on how to manage your references first and then if you’re willing to do that I will write you a recommendation. So I’ve been listening to them for years and I can point to dozens of their tools that I use here on here at the university myself. I just can’t think highly enough of them and they’ve been at this for 12 years now.

Bonni: [00:42:20] Well Dawn and Jay thank you so much for joining me on today’s Teaching in Higher Ed and thanks for introducing me to your wonderful book. It was an absolute delight to read. It was exactly what you say multiple choice questions should be in the sense of there’s no there’s no I can’t think of how to explain it it’s not written in such a way to make it not engaging to any of us it’s written just very practically.

Bonni: [00:42:43] And really helpful and encouraging and I just enjoyed reading it from cover to cover it wasn’t work for a second. It was just a delight but also in terms of really really challenged me so just thanks so much for joining us today and participating in the community.

Jay: [00:42:57] Well I’m so glad to hear you liked the book and found it useful and I really appreciate you having us on.

Dawn: [00:43:02] Yes thank you so much.

Bonni: [00:43:04] It was just fabulous having both Dawn and Jay on the program and as I said previously it sure was a delight to me to get to know so much more about using multiple choice questions in my teaching through their work and I’m just so grateful they came on the show.

Bonni: [00:43:21] And I’m also grateful that you decided to listen today and to be a part of the Teaching in Higher Ed community. If you’d like to have access to what the guests recommend each week on the show and what I do as well.

Bonni: [00:43:33] And also to get it article about teaching or productivity each week that you don’t have to remember to go look at. You can subscribe to the weekly update and you can do that at teachinginhighered.com/subscribe.

Bonni: [00:43:47] When you subscribe you automatically receive a guide that has 19 tools that I have found to be really helpful in technology that helps me either facilitate learning or be a little bit more productive. Back to the enter of the show so I can be more present for my students and other people that I am close to and are important to me. Thanks for listening and we’ve got some great episodes coming up. And I look forward to seeing you then. Take care.

Teaching in Higher Ed transcripts are created using a combination of an automated transcription service and human beings. This text likely will not represent the precise, word-for-word conversation that was had. The accuracy of the transcripts will vary. The authoritative record of the Teaching in Higher Ed podcasts is contained in the audio file.