Bonni: [00:00:00] I know I’ve had some complaints about them myself and a lot of good information to help me improve my teaching. On today’s show Betsy Barre joins me to share about their research on course evaluations.

Production Credit: [00:00:15] Produced by Innovate Learning, maximizing human potential.

Bonni: [00:00:26] Welcome to this episode of Teaching in Higher Ed. This is the space where we explore the art and science of being more effective at facilitating learning. We also share ways to increase our personal productivity approaches so we can have more peace in our lives and be even more present for our students.

Bonni: [00:00:53] Today I welcome to the show Betsy Barre. Elizabeth Barre. And I’m excited to share a little bit about her background and to have this conversation with her about course evaluations. She received her Ph.D. in religious ethics from Florida State University. And after that she began her post-doctoral fellowship program that was designed to introduce recent graduates to the challenges and rewards of teaching undergraduates in the context of a residential liberal arts college.

Bonni: [00:01:28] Since then she has had extensive experience teaching in higher ed. And today she works as the Assistant Director of Rice’s Center for Teaching Excellence so we’ve had a couple of people from that department be on the show in the past I’m so grateful for their contributions and for them introducing me to Betsy. Again she is trained as a comparative ethicist and her research lies at the intersection of moral philosophy and the history of religion with a specific focus on Muslim, Christian and secular political ethics.

Bonni: [00:02:05] And I would encourage you to visit the show notes at teachinginhighered.com/89 and we’ll have a link to her biography where you can learn more about her teaching which she continues to do at least one undergraduate course a year. And of course is helping to develop the teaching skills of so many of the faculty at Rice. Betsy, welcome to Teaching in Higher Ed.

Betsy: [00:02:27] Thanks for having me.

Bonni: [00:02:29] Well I was so excited to be connected with you with some of some of your colleagues and I’m excited about the topic we have to talk about. But I’m going to have a go on one quick tangent before we do that and I’m curious what is one thing that’s not in your bio that’s important for us to get to know about you so that we can have a full appreciation of you as a person.

Betsy: [00:02:50] Well sure. I think there are probably many things but I’m glad that you narrowed it down to one specific thing for me to mention. And I guess one personal thing that is particularly relevant to the conversation today is that my husband is also an academic and he is here at Rice and he’s in the political science department but he’s a methodology.

Betsy: [00:03:07] So he is an expert on Quantitative Social Science and Research Design. So he has helped me whether he’s been happy about it or not. He’s helped me a lot with sort of navigating the research on student evaluations over the past year or two and thinking about the way that the research is designed and quality studies etc. So I’ve been lucky in that regard as a humanist to have a social scientists by my side helping me think through this material.

Bonni: [00:03:31] And does that show up in dinner conversations. Like my husband and I when we talk about teaching.

Betsy: [00:03:37] Yeah. Again I don’t know. I mean I think there was a point I think you know probably four or five weeks into my research on this topic where he was just like you just need to take a civics class when we can’t sort of work through this through a dinner table conversation. But for sure yes we definitely would talk about it often or even today I was sending him some questions about some of the things I’ve been reading that have recently been published.

Betsy: [00:03:57] So it’s great to have him as a backup you know to get a little help when I need it. And I actually did sit in for a little bit on one of his classes but it’s nice to be able to have that around. Outside of the working hours to get some help with this material.

Bonni: [00:04:10] Speaking of this material how did you first get interested in studying more about course evaluations.

Betsy: [00:04:17] Yeah that’s a great question. You know it’s funny lately at the CTE I’ve been spending a lot of time reading about you know valuations and talking about certain evaluations and also grading. And I was reflecting that and how it’s a strange thing that these two issues in teaching became my kind of areas of interest.

Betsy: [00:04:34] As someone who is trained in the humanities as an ethicist but then it occurred to me that technically both of these things are about evaluation which is kind of what ethicists are into. So I said well maybe I’m just somebody who’s interested in evaluation and that’s why I was interested and also being an ethicist.

Betsy: [00:04:49] And I thought about it more that in some ways both student evaluations and grading also have some ethical implications some direct normative implications to them about you know how do we justly think about employment practices that are institutions and how do we think about fairness with our students and so I think that’s part of the reason that I’ve always been interested in both both of these questions within teaching. Among other things.

Betsy: [00:05:11] And so when I started my position here at Rice they were the faculty were already discussing looking over our student evaluation system and they were thinking about revising the system and I had just started my position at the CTE and I initially suggested that well I know there’s a lot of literature out there I had heard from friends but I didn’t know much about it. One of my first tasks as a city staff member would be to do a literature review for this committee.

Betsy: [00:05:38] And so I decided to start reading the literature and it was I had no idea why I had gotten myself into. It was an extremely expansive literature and then through that process got more and more interested presented to different community constituencies that rise and had many conversations back and back and forth wrote a couple of blog posts and conversations about it online and and unexpectedly I have found myself to be a sort of primary source of conversations about about student evaluations over the past year in online communities that are think about teaching and learning so I didn’t expect to do that.

Betsy: [00:06:10] But it’s been fun and I’m glad and we’re still moving forward at Rice and thinking through our evaluation system and possibly revising it. So we look forward to what will come. But that’s how I got into it.

Bonni: [00:06:22] Before I was in academia I used to train people on computers. And after each class we would pass out the evaluations for those courses and people used to call them smile sheets. That’s oftentimes in the training business but what they are essentially there if you just take what data you’re getting you’re really taking whether or not they liked or disliked the instructor.

Bonni: [00:06:47] And it was one of the things as I was first introduced to a professor of instructional design he used to be out at Utah State although he has since retired but just the idea that actually what I was teaching my students to do was to follow my directions.

Bonni: [00:07:01] You click on this inside of Microsoft Excel and then click on this and then after they left the class they had a great feeling and felt so inspired but actually couldn’t solve problems with Excel like I had hoped that they would be able to really eye opening for me.

Bonni: [00:07:15] But what if I compare those traditional smile sheets to what’s going on in the vast majority of our evaluations system inside of academia. Is it fair to say we are at least one level up from there.

Betsy: [00:07:30] So it’s unclear. I mean and one of the first things that I think is important to talk about we talk about the research on doing evaluations is that we talk about it in the research literature or even on blogs etc. We often assume that the idea of a student valuation is universal.

Betsy: [00:07:47] That Student evaluation is the same across different campuses and different institutions and that’s one of the things that’s most striking when you get into the literature is that most schools have variety of different forms different process these different goals and so it’s really actually very hard to make any sort of definitive statement generalized statement about whether student valuations are one step above smiley. This smile form or not some maybe some may not be depending on how your institution implements student evaluations.

Betsy: [00:08:15] You know when I first started teaching I’ve been teaching for about 10 years. This came into this kind of suspicious about student evaluations in part because generally for whatever reason I’ve always been pretty good at getting students to like me. And so and I had relatively good evaluations but I was always skeptical. I always thought that really what I’m getting in the student evaluations had nothing to do with whether they’ve learned or not. Right.

Betsy: [00:08:37] It’s just about they’re happy to be in the class and kind of enjoyed you know what was happening they they cared about me I cared about them and I was always really curious about to what extent the evaluations that I was getting back in a semester had anything at all to do with student learning and that’s kind of what drove my research initially to see what the research had to say about that.

Bonni: [00:09:00] One of the things that I have thought a lot about often is that it seems like there would be an effect on when you distributed evaluations. There’s I wrote a blog called The Depths. Some years ago just thinking about the traditional it doesn’t always happen but just like teams have for me norming, storming, performing and adjourning that there’s a common pattern that’ll happen within teams there’s kind of a common pattern that I see happen within semester long courses in this case 15 week classes.

Bonni: [00:09:34] And so I wonder is any of the research that you’ve looked at does it show that it has an effect. If I were to give it Drame week 12 versus week 15. That’s because by the end if you do it closer to the end I imagine students have a pretty good idea of what their grade will be in the class and perhaps that abomination is more reflective of how pleased or not pleased they are with their potential grade.

Betsy: [00:09:56] Part of a question that is sort of interesting that the question you raised is what are we actually trying to evaluate. Are we trying to evaluate the quality of the course. Are we trying to evaluate the quality of the teacher. Are we trying to evaluate the quality of the teacher in that instance of teaching the course.

Betsy: [00:10:13] And so one assumption you’d make is as if I’m trying to. If we were interested in that course right in the quality of the teacher in that course you need to wait till the end of the semester so the student has had enough information to sort of reflect upon the quality of the semester.

Betsy: [00:10:28] But there’s some studies or some interesting studies and they’re controversial in fact usually brought up to critiques doing evaluations where they’ve shown that at least in terms of certain features of teaching effectiveness students can make judgments within about a minute but a half a minute that are similar to the judgments they would make at the end of the semester.

Betsy: [00:10:48] Anything that happens throughout the semester seems to be that they had some sort of really good intuition or just based on things we don’t want to actually worry about right the early on within the first 30 seconds. They made their judgments and they said that’s why people say you know the first day in class is really important because they make their judgments and they stick with them throughout the semester.

Betsy: [00:11:06] Actually interesting that the way that that question your question it answers depends upon what you’re interested in evaluating and if you’re interested in evaluating the overall quality throughout the semester you wanted to be later in the semester.

Betsy: [00:11:20] Most of the psychological research will say the most important thing is just that you don’t leave too big of a window for the students to answer the question. So those of us that are now moving to online evaluation forms if you give students like a four week window to return their evaluations the students who respond in the first week of that four week window will have very different a different scenario and context and the students who respond at the end of the fourth week and that’s considered to be particularly problematic. But yeah the grading question is an interesting one that I also heard in there too I think I want to wait on that and so that could take us a little bit far afield.

Bonni: [00:11:55] What are some of the most common complaints that we hear faculty talk about. And then we can we can go specifically and look at the grade.

Betsy: [00:12:02] So one of the biggest complaints that faculty have about students vibrations is that it’s not a reflection of teaching effectiveness. So we’ll start there that it’s actually not a reflection of teaching effectiveness. Now part of the problem there as well what do we mean by teaching effectiveness and so some will say well it should at very least whatever we need teaching if I did this should be a marker of learning of whether the students have learned or not.

Betsy: [00:12:25] And it’s not clear that student evaluations are a reflection of learning faculty think that they might be a reflection of other things whether that be whether the students like you whether that be that they’re getting good grades the courses easy etc.

Betsy: [00:12:39] So that’s one primary concern that faculty have is that it’s it doesn’t seem to be a reflection of student learning but there’s also a more subtle point that some faculty are also concerned at just because a student doesn’t learn does not necessarily mean that you’re an effective instructor and so there’s also a larger philosophical debate about how we define teaching effectiveness.

Betsy: [00:13:01] Is it tied to whether the students are learning completely. Or is it tied to the actual types of methods that I’m using in class and my knowledge of the subject material. How much time I put into it. My organization my preparation etc. because there could be many other reasons why students aren’t learning because they’re not studying they’re not coming to class etc. And so where does this teacher’s responsibility come into play a role there.

Bonni: [00:13:25] I’m curious about the two examples that you brought up the whether they like you or not. Does the research show that we actually are able to learn from people that we like that we that that but that is a factor in our ability to learn in the class or does that not show up.

Betsy: [00:13:42] I’m not so sure there’s been a lot of research to look at the correlations between whether students like you and the correlations between whether you’re doing well and doing evaluations but one theory of why there might be a correlation between high student evaluations and learning is that well if students are happy in a class they’re more likely to be motivated to come to the class to participate.

Betsy: [00:14:03] They’re going to be more engaged they’re going to be more interested and so they will learn more. So it’s certainly the case that there’s not a problem with suggesting that students liking a class is something we should strive to achieve. But that is not the same thing as learning.

Betsy: [00:14:17] Right. It’s important for us to remember that as well that just because it’s like a class doesn’t mean necessarily learning. So it works one way and not not the other way so we can all say that it has the potential to improve learning but just because you like it doesn’t mean that you’re learning as much as we would want you to.

Bonni: [00:14:32] And then what about with the case of the course being easy. Does that show up to have higher or lower course evaluations when the course is perceived as having been easy.

Betsy: [00:14:41] That was one of the most surprising things in the research literature is that it actually turns out that the harder the courses the higher the evaluations you get. And this is where there’s a there’s a difference between grades. Right.

Betsy: [00:14:53] And also you know how well they perform versus the challenge that they have in the class and how much they feel like the workload is. Every school has a different form. But for me I actually have separate questions for workload versus grade your expected grade.

Betsy: [00:15:08] Right. Well challenge and so if you have a workload question it turns out that the more work you give your students the higher your evaluations will be. There are some forms that actually split out whether it was all the work is valuable work or busy work.

Betsy: [00:15:21] And if you split it up along those lines some students think the work is valuable if they think all the homework they’re doing and the work that in your class is something that’s helping them learn. You can go all the way up to twenty extra hours a week outside of class.

Betsy: [00:15:34] And students will still be giving you higher evaluations eventually. There’s a drop off point right. As you might imagine if we’re expecting 30 to 40 hours a week for students to work on their class but that was really shocking to me that actually works in the opposite direction that the less work you give the lower your evaluations will be. And then there’s an interesting interaction with with grades as well.

Bonni: [00:15:54] Well why don’t you talk about that a little bit then how grades come into play with course evaluations.

Betsy: [00:15:58] We do know that the research does show that the grade you get are correlated with student evaluations on average across a section of a course. And so one obvious thought about that correlation is that well that’s a sign that there is this relationship that we call the leniency hypothesis in the literature where faculty are giving you a good grades so that they’ll get good evaluations and you’re giving students are giving good evaluations so that they’ll get a good grade.

Betsy: [00:16:24] So that’s one interpretation of the relationship. What a lot of the psychologists who have done work on measurement have said have a couple other interpretations as well and one could be that actually you want grades and student evaluations to be correlated. If you believe student evaluations are a measure of learning and that’s because if students are learning they presumably will get higher grades.

Betsy: [00:16:45] And so if it were the case that student evaluations reflect learning and they aren’t changing with grades. So if you have a class where a bunch of students are getting A’s because they all learn you want the evaluations to be higher there. So some psychologists have said the fact that they’re correlated and at a small level by the way incidentally they’re correlated the correlation is relatively small.

Betsy: [00:17:04] The fact that there’s a modest correlation is actually something that proves their validity rather than that there’s a bias and then a third hypothesis which I think is really interesting is that they actually the correlation between grades and soon evaluations are actually tied to a third factor.

Betsy: [00:17:20] Prior characteristics so we know for instance that if students come into your class with a prior interest in your course material if they are particularly motivated students that you will get higher evaluations that that is a bias in student evaluations.

Betsy: [00:17:34] And we also know that that will affect grades and so some people suggest that it’s that third thing the prior Karasik characteristics of your students that are leading to the correlation between grades and student evaluations. What most people say is that it’s really probably all three of the things working together that explain the correlation.

Betsy: [00:17:51] But most likely leniency is the one that can be ruled out. So just because there’s a correlation if you if you’ve done studies that your institution has seen that there’s a correlation between grades and innovation. That doesn’t necessarily mean that the leniency hypothesis is correct because there could be other interpretations of that correlation as well.

Bonni: [00:18:10] In terms of the different disciplines and university one theory that gets thrown out a lot at my institution is that those disciplines that are more quantitative in nature will always receive lower evaluations than those that are more qualitative or what would traditionally be thought of as the more interesting topics or disciplines to study.

Bonni: [00:18:31] How does that show up in the research that you did.

Betsy: [00:18:34] Well actually interestingly in two ways. So the first thing to say is that the research does recognize that there are certain variables that do bias the results particularly on the the overall quality. Question that’s often on many of these forms and so there’s two ways in which I think those who teach quantitive courses can be disadvantaged.

Betsy: [00:18:54] And one is that sometimes both quantitative courses at least for example say in a case of my husband who’s teaching in political science their statistics course is required of all majors. And so if you’re teaching a required course that students do not have a choice to take when you said interesting that made me think of this is that if the students aren’t taking it of their own volition not something that they’re interested in.

Betsy: [00:19:15] We know that that your evaluation numbers will drop and actually in some systems and some validated measures it can drop my entire point if the students are not choosing to take the class out of their own interests. And that’s a big deal on a five point scale an entire point is a big drop and then we also see the literature shows that there are differences among the disciplines in terms of the average evaluation that faculty get.

Betsy: [00:19:40] Now how you interpret that is it’s basically impossible for us to know what’s causing that. I mean you could make the joke and say well just you know humanities faculty are just better right.I mean that’s what

Bonni: [00:19:47] I was about to make that joke.

Betsy: [00:19:52] But obviously that’s silly. But the problem is that there’s no way we can actually test for that because there’s no way you can do these controlled random assignment across a philosophy class compared to a chemistry class because they’re different exams and different assessments.

Betsy: [00:20:04] So it’s very difficult to figure out why there’s disciplinary differences so most validated forms will control for discipline or or institutions will control for discipline but we’re not really sure why there’s that difference.

Betsy: [00:20:18] But we do definitely know that quantitative and qualitative STEM disciplines are lower. I have some stem colleagues who are hypothesis that STEM students are just more rigorous in terms of their evaluating. So it’s not that they are harder classes they just are more stingy with their high evaluations they’re just the culture of it’s of a certain type of student which I thought was interesting. So there’s a couple of ways in which those biases show up for sure.

Bonni: [00:20:44] That is such an interesting example. Way back when. This is this is 20 years ago my dad came to where I worked and took a computer class and that was one of the benefits that we had of getting to work. There was a family member who come for free and I went in to see him after the class was over and he had his evaluations in there and he had given the instructor all nines and that industry.

Bonni: [00:21:09] I mean you pretty much would just get straight tens if you were good at what you did. I mean yeah I would have evaluations for an entire month that were nine point nine eight and I was not unique in that industry. We really it was all about pleasing the customers giving them great education. And I was saying you can’t put that in the box.

Bonni: [00:21:29] He said because he has not very much that engineering type of person he said well they were great they were a wonderful but there’s always room for improvement.

Betsy: [00:21:37] And a nine is good on a ten point nine is good.

Betsy: [00:21:40] So I mean it is kind of interesting and actually some of the faculty joked to that because it’s also the case that grade grade distributions are different in them as well. So it’s kind of like if they’re getting graded where they use the full distribution if the STEM students are they’re also more likely to grade their faculty using the full distribution whereas in the humanities that means there’s more compression. So who knows.

Betsy: [00:22:00] I mean there’s not a lot of logic to explain that difference but it’s certainly there is a difference that seems to show up for sure. The one thing it’s worth noting too is there’s a lot of research about the different questions that show up on student evaluation form.

Betsy: [00:22:15] So most of the stuff I’m discussing right now are all about that overall question. Overall this instructor was effective in this class the detailed questions about did they show up to class did they have knowledge of the subject matter. No. Were they enthusiastic. Those ones are much more messy and fuzzy in terms of these questions about bias.

Betsy: [00:22:34] So most of the research literature really focuses on that question of overall effectiveness which is worth keeping in mind at your own institution.

Bonni: [00:22:41] Has that shaped at all your thinking because I know you said you’re just thinking about where is headed that rice but has it shaped it to be thinking gosh maybe it’s not even worth asking those kinds of questions if they don’t necessarily tell us that much information or is that at all.

Betsy: [00:22:56] Yeah that’s a really good question and I think that a lot depends on why you’re using these instruments on your campus. What are you trying to actually measure with these instruments and so you know on the one hand I sort of mentioned earlier you could be primarily focused on interpreting teaching effectiveness as whether the students have learned. Right.

Betsy: [00:23:17] And if that’s your primary focus whether the students have learned of course the best measure of whether they’ve learned is direct measurement of learning through some sort of standardized assessment. But we don’t do that in higher ed for interesting reasons.

Betsy: [00:23:29] So if that’s not the measure we’re going to use then we want to know whether students have learned one of the best things they do is ask them whether they’ve learned.

Betsy: [00:23:36] And so part of the movement in evaluations now is to focus on questions about learning rather than questions about what the faculty members are doing because at the end of the day if you faculty can do very different things that as long as the students are learning that’s what we care about.

Betsy: [00:23:52] And so at Rice we’ve been talking a lot about thinking about these questions so removing some of those method questions and focusing more on what are the goals of your class. And have the students asking the students themselves have you achieved the goals that the professors set out in the class.

Betsy: [00:24:09] So the faculty have some control over sort of setting the learning outcomes and I know that Stanford has moved to a system like this as well where the primary focus of the student evaluation is on learning outcomes and their student evaluation of whether they’ve achieved those outcomes.

Betsy: [00:24:23] But of course students may not always know whether they’ve achieved the learning outcomes that other folks have argued that well what’s more important is actually the method in the classroom. And we should be evaluating teachers on whether they are actually using methods that have been proven in evidence based research etc.

Betsy: [00:24:43] So in that case then you would want certain methods in the student evaluations but then there’s a whole host of debate about which method go in to the to the new evaluations and that’s a longer debate that we sort of decided to set aside at rest.

Betsy: [00:24:56] At least we’re thinking about it at the moment and instead focusing on learning outcomes.

Bonni: [00:25:01] I imagine that would be so tough because one professor might call it problem based learning but their version of Problem Based Learning doesn’t look anything like their next door neighbors version of problem based not even if we’re using the same name.

Betsy: [00:25:12] And it’s not even things like what’s really funny.

Betsy: [00:25:15] There’s oftentimes a lot of institutions questions like did they return assignments on time right or did they show up to class prepared. And a lot of the research shows it’s not clear that’s actually really necessarily correlated with learning about how the faculty do on those questions.

Betsy: [00:25:32] And so while we as an institution may care about those things in terms of evaluating those we’ve hired. So again it just depends on what you want to know about when you’re making decisions about hiring and firing. If that isn’t even is the primary purpose of these forums which I would argue it shouldn’t be you know then you would ask different questions.

Betsy: [00:25:51] So for us we’re really focusing on these forums as a way to improve our teaching at Rice. And since we wanted to improve our teaching we are interested in asking questions that will be most helpful for that purpose.

Bonni: [00:26:02] How are you seeing all of this being covered by the press.

Betsy: [00:26:06] Yeah that’s a great question and that really is what motivated my first public discussion of these issues. I had read a lot about evaluations and the Chronicle of Higher Education and Inside Higher Ed and I was very skeptical about student evaluations and what they might tell us.

Betsy: [00:26:23] And so when I went to the literature what I was most shocked by was not just that there were different conclusions in the literature which there were but there is also a lot of debate but we’re just more shocked by the fact that there was so much out there that had been written that was never being discussed in the popular press.

Betsy: [00:26:41] And in fact often what you would find is that the popular academic press would discuss one recent study that came out and everybody will be posting about it and blogging about a newly almost always they were critical of student evaluations and there would be no reference to the larger literature literature.

Betsy: [00:26:58] So I just kind of shocked by that. And that was something that I wanted to express to my colleagues who I know don’t have time to read the literature that sometimes things that we read about in reports on the research literature are not always giving us the full story and turns out that this is an extremely complex and sophisticated research literature. The most researched question in higher education. Thousands of peer reviewed studies that have been reading about it for 100 years.

Betsy: [00:27:23] So it’s not something that we haven’t looked closely at but rarely do we as faculty get any access to that literature and when we do it’s often oversimplified and in one direction which was distressing to me that is really truly remarkable to me.

Bonni: [00:27:38] And now I just can’t help but ask. So it’s been thousands of studies a hundred years.

Bonni: [00:27:44] Is there anything that you that you have just gone in and said How come no one ever asked this question about evaluations. It just bugs you to this day.

Betsy: [00:27:53] Oh that’s a really interesting question. I mean there are many things that were really even just the example of the grading relationships the different way. You know I know that a lot of schools will will.

Betsy: [00:28:04] We’re very faculty are very concerned about grading and so they’re interested in whether there’s a correlation between grading and evaluation. But no one ever bothers to think about other possible causal relationships that might lead to that correlation.

Betsy: [00:28:17] And so many things like that that the literature will highlight but I also think just as a non statistician it was very helpful for me as a social scientist not even to suspicion as a non-social scientist to understand what it means for first into evaluation instrument to be valid.

Betsy: [00:28:35] And it doesn’t mean that it’s a perfect measurement and understand that correlations of point five are really strong in the social sciences but that doesn’t mean that if I get a score of a four point four and my colleague gets a score of a four point two that I’m necessarily a better teacher in them or that measure the measure can still be valid.

Betsy: [00:28:54] And it could still be wrong. In one instance right in terms of ranking faculty and that was really powerful and important for me to think about as well in terms of how these instruments get used by administrators.

Bonni: [00:29:05] What is some of the emerging research that’s coming having to do with gender.

Betsy: [00:29:10] Yeah there was so in January which is why this is good timing in January. There was a recent piece that was published in many of the again popular press sort of latched onto it and began discussing it. That actually looked at two prior studies on gender as a possible bias in student evaluations and read and analyzed with more sophisticated statistical measures that data and one team it’s really interesting about gender even before that study came out is that there have been attempts to look at systematic gender bias in student evaluations and there have been experiments where they Breen’s bring students into the lab.

Betsy: [00:29:50] And they manipulate it that way there have been observational studies and the results have been all over the map even in as early as 1993. There had been many analyses done on gender as a bias which was surprising to me. And sometimes in the studies you’ll find that that actually men get higher scores than other studies.

Betsy: [00:30:07] Women get higher scores and even when you account for mystical significance. That was the case and so early on at least in 1993 there was a sense that you know there wasn’t any clear evidence about gender being a bias and soon evaluations. But one of the things they noted is that all of those studies were just correlational study.

Betsy: [00:30:27] So if it were the case hypothetically again like humanities professors that women were better professors we just were right if that were the case that we were better teachers then we should not be getting the same scores as men we should be getting higher scores.

Betsy: [00:30:42] And so a bias can still exist even if we’re getting the same score as men. So there have been calls for studies that look at differential validity. So we would want to do the little the studies to see if like the teachers who are actually better by comparing test scores etc. would get lower scores because simply because they’re a woman.

Betsy: [00:31:02] And the reason this recent study that came out by Phil Stark at Berkeley who’s looking at some studies that some data from France and then some data from online courses in the U.S. are really exciting in terms of their research design.

Betsy: [00:31:14] But there are still some questions I have about the research design as well but it’s it’s moving us forward to think about to do more research and really at the end of the day I think having looked at all this literature what I think about it in evaluation research is that we just need more of it that there’s a lot of it that’s been done in the past that seemed to suggest that you know they’re reasonably valid instruments and there are certain biases that we know about but that there’s still room for doing more research to show that there are biases we haven’t accounted for.

Betsy: [00:31:47] There is room for making them better so that they are are more accurate. I’m always excited when there are new studies that come out like the one that just came out from Berkeley about gender and I do think there have also been other studies. One thing important to say to there was think sociologist Ben Schmidt – maybe he’s a historian? – who created a database of all the words that were used in Rate my Professor evaluations and you can search them by gender and discipline.

Betsy: [00:32:14] And so there’s it’s really just amazing you know graphic that will show that women are a bit – if you would have been brilliant for example – men in all disciplines are more brilliant than women. And so it’s really fun – fun and depressing – to play around with that to see the way in which language gets coded and gendered ways.

Betsy: [00:32:32] And that’s one of the thing that’s been it’s really important to emphasize is that all of the research I’ve been talking about has been about the quantitative numbers not about the qualitative very little research has been done on the qualitative feedback that students give us which is often what we as faculty sort of focus on for various reasons.

Betsy: [00:32:51] Because it’s often so bold and you know strong strongly worded but we don’t really know much about the qualitative responses they get and my hunch has been that that’s where a lot of the gender differences come come up and are quoted in the qualitative responses we get from students and so that’s also worth thinking about in terms of how our institutions use the qualitative versus the quantitative responses.

Bonni: [00:33:13] I mentioned to you. I don’t know if it was an email or on the phone but it doesn’t matter. I mentioned that I had just heard about Ben Schmitt’s research that he had done on the rate my professor but the conference I had seen it that I was sitting toward the back of the room and couldn’t see that well and now I’m sitting here playing with even minor things.

Betsy: [00:33:30] It’s amazing. It’s really depressing. Part of it is you have to come up with the word yourself and so it’s hard to get a sort of systematic analysis but if you type in something like “organized” – women are more organized almost across the board.

Betsy: [00:33:42] But if you type in “disorganised” women are also more disorganised across the board and there are a lot of words like that and what that suggests and the same thing with there are certain words where the positive and negative men are stronger at both.

Betsy: [00:33:57] And part of what that suggests to me I mean on Evan any evidence for this that maybe women are evaluated along different metrics than men. So it’s not that women are negative and men are positive but that when students think about what makes a good female teacher they think about different criteria than when they think about what makes a good male teacher.

Betsy: [00:34:17] And so when they start giving their qualitative feedback they’re going to sort of pull from different dimensions. I mean who knows if that’s the case but it was interesting that both the positive and negative along certain dimensions would cluster for women or men depending on what the language was.

Betsy: [00:34:31] It’s is great really great service that he did. I mean one interesting thing is they’ve actually done studies of quantitative numbers and rate my professor and they’ve actually found there’s no gender difference in the quantitative numbers. You don’t rate my professor which is not as validated as some of our other reading.

Betsy: [00:34:49] If the quantitative numbers of Rate my Professor there’s no gender difference but the qualitative there is I think that suggests probably what we see on our campuses is that the numbers may not be biased but the qualitative responses may be very much gendered.

Bonni: [00:35:03] As you’re talking I just want to make people aware that everything that we’re talking about will be in the show notes at teachinginhighered.com/89.

Bonni: [00:35:12] And I know you are working on a annotated bibliography that if it gets done by the time the show goes we’ll put it up there but if it doesn’t I told you not to put pressure on yourself and whenever it’s done.

Bonni: [00:35:23] But I just link to the BenSchmidt.or/Poffgender and it’s so worth taking a look at it’s staggering.

Bonni: [00:35:30] Well before I move to their recommendation segment of the show I want us to get practical for a minute because as we’re listening and hearing about all of the research that you’ve done on evaluations how do we bring it home and actually as institutions and as individual faculty members actually use the data.

Betsy: [00:35:48] Yeah I think it’s really important for me to emphasize because we kind of got into the weeds of the research that even if student evaluations even if there’s a scientific question of whether these measurements are valid or can be Mallot valid if you get a good enough form even if the measurement is valid it’s not a perfect measurement and it’s definitely not direct measurement of student learning.

Betsy: [00:36:08] And even if it were there are other things that we may want to take into consideration when thinking about whether we keep teachers on board. And so one of the things that I think almost anyone who writes about food evaluations in higher ed research will say is that we need multiple measures multiple pieces of information if you’re using it for promotion and tenure decisions on campus.

Betsy: [00:36:29] So I strongly am an advocate for not just student evaluations but also portfolios observations you know perhaps even if it possible some way of getting some substandard assessment into the mix as well of whether your students are learning.

Betsy: [00:36:43] So that’s if you use it for the summative purposes which are many of our campuses. That’s the reality that these will be part of our summative evaluation. But if that’s true I think we all need to advocate for other pieces of information to come in because even even the strongest advocates of the validity for the validity of student evaluations will still say they are not perfect measures and they need to be used with care and they also need to be used alongside other measures of teaching effectiveness. But then I think for my sake when I’m most interested in.

Betsy: [00:37:14] I think faculty it right from Rice is most interested is the formative use of those two innovations to help us improve our teaching and to help us get a sense of whether the things we’re doing in the classroom are working or not. And then again they’re not perfect mentors for that either.

Betsy: [00:37:28] But I think that more than anyone else in our classrooms you know we can have people come and observe us. We can have people look over our syllabus but our students are there every day with us.

Betsy: [00:37:40] And if we want to know whether students are learning there’s no better way to know than to ask them that they’re learning right really we can do it in a variety of ways.

Betsy: [00:37:47] But getting as much feedback from our students about what’s working and what’s not working and whether they feel like they’re learning is I think crucial first step for us improving our teaching in a formative way.

Bonni: [00:37:59] I mentioned to you that I hadn’t figured out what my recommendation was going to be before we started the show but I think I may have just figured it out at the very last thing you said.

Bonni: [00:38:07] My recommendation is going to be that we also think carefully about how we present the evaluations. I know that. Yes yes absolutely especially because for so many of the students I think that at least in my institution they’ve expressed that they feel like they don’t really matter.

Bonni: [00:38:23] And I’ve always tried to take time aside that first of all I want them to know that I really care about what they think and I explain that you’re going to fill out these evaluations and I won’t get them for a while but that I always read them and I blog about that. The changes I want to make based on the feedback.

Bonni: [00:38:39] It’s something I’m trying to do a lot more of is to blog about the course evaluations I receive and the improvements I’m trying to make so I try to be transparent in that way. But then also as an institution that it might feel like if you have been trying to express that this person may have a better career option to you that there is a gap between when that feedback happens and when a decision of potentially changing that person out might happen and they don’t always understand about things like tenure about even what an adjunct professor is in.

Bonni: [00:39:11] I mean all of these things that are new to them I don’t want to spend too much time because it does come at a time in the semester when they’re pretty overwhelmed. But I do like to set that aside and also make sure I never make any jokes about the evacuations are Enever.

Bonni: [00:39:24] I mean that sarcasm has no place in that particular conversation and then I’m doing all the professional things up leaving the room and explaining to them that I won’t see the evaluations till they’re done and I’ve done it a million times and I’ll do it a million more times and that would be my advice is just to really think carefully about how those are administered.

Betsy: [00:39:41] Yeah. And so are you ready for my recommendation. I’m ready. I’m excited.

Betsy: [00:39:45] So I think you know unfortunately unless you’re on a committee or you’re heavy in sort of you’re involved in back in the Senate at your institution you may not have a huge family may have a huge role to play in shaping what your evaluation form looks like and what the process is and so one of the things that if you happen to have a forum that you don’t does you don’t think works for you or it’s just not very helpful in terms of getting the information that you want.

Betsy: [00:40:11] I would strongly recommend that actually the research literature supports this designing your own instruments and distributing it yourself especially at the mid semester point when you still have time to make changes in your course. And so this allows you to do two things.

Betsy: [00:40:25] One is to get feedback and respond and students actually the research shows that your end of semester evaluations will go up even just giving the mid semester evaluation at the mid semester point you don’t have to change anything just giving it shows that you are that you care.

Betsy: [00:40:40] But of course if you respond and change things then students are even more you know or are happier. And also I think probably are likely to learn more if you if you respond to their needs in that way.

Betsy: [00:40:51] But it also allows you to design a form that shows your communicate to your students that you take their feedback seriously and to ask the questions you really want to know. Specifically about your course about your discipline about the moves that you’ve made pedagogically and I’ve done this for a number of years and it’s been extremely helpful.

Betsy: [00:41:08] And it’s something that we try to encourage at our center for all of our faculty to do and if you have a teaching center on your campus they’re likely to offer mid semester course reviews where they’ll come in and watch you or talk with your students.

Betsy: [00:41:19] So I would definitely take advantage of that as students tend to immediately recognize that this is more serious in some ways than the end of the semester evaluation or they think it is right they think it’s more serious. And so you often get more substantive feedback at the mid semester point.

Bonni: [00:41:35] I also want to just encourage people I mentioned to show notes earlier but that I’m going to have a link to some of the blogs that you’ve written about evaluations. And then also you have a wonderful screencast that you think it was designed specifically for Rice but certainly would be applicable to all of us that goes through your research as well. And anything else that you e-mail me after the fact that you think.

Betsy: [00:41:56] I think it would be helpful yes specially because some of the things I was talking about today it’s nice to have some visuals with so having that screen because if you’re curious will will be helpful.

Bonni: [00:42:04] It’s really good. I’d recommend it to just added a second recommendation. Check out the notes there at teachinginhighered.com/89 and Betsy I just wanted thank you so much for investing your time in the Teaching in Higher Ed community and just for sharing with us today about evaluations.

Betsy: [00:42:21] Absolutely. Thank you for having me on.

Bonni: [00:42:24] I’m so glad that Betsy was able to join me to have such an interesting conversation about course amalgamations and if you would like to receive the show notes with all of the links of the things that we talked about during the show you can have them show up to your inbox every single week by going to teachinginhighered.com/subscribe and when you do that you’ll also receive a copy of the Educational Technology guide. With 19 essential tools for helping you use technology to facilitate learning and also in your personal productivity.

Bonni: [00:42:58] After I’m done interviewing each of the guests I asked them who would they recommend be on the show or what topics would they recommend that we talk about. And I’d like to as always extend that invitation out to you is there someone you think should be on the show that we haven’t had on before or we have had on before and you want to hear more from them.

Bonni: [00:43:16] Please feel free to make suggestions and give any other feedback about the podcast at teachinginhighered.com/feedback. And one exciting thing like we mentioned in a recent episode up until today when I record this two days from when this podcast will publish we’re on the home page of iTunes podcasts page.

Bonni: [00:43:37] So if you would go up there to iTunes or whatever service it is you listen to and write a review or just give it a rating it’ll really help other people discover the show and really build our community so we can just continue to grow our own teaching and productivity.

Bonni: [00:43:51] Thanks so much for listening and I’ll see you next time.

Teaching in Higher Ed transcripts are created using a combination of an automated transcription service and human beings. This text likely will not represent the precise, word-for-word conversation that was had. The accuracy of the transcripts will vary. The authoritative record of the Teaching in Higher Ed podcasts is contained in the audio file.