EPISODE 628: The Fair Feedback Project with Remi Kalir

Bonni Stachowiak [00:00:00]:

Today, on episode number 628 of the Teaching in Higher Ed podcast, the Fair Feedback Project, with Remi Kalir. 

Bonni Stachowiak [00:00:12]:

Production Credit: Produced by Innovate Learning, Maximizing Human Potential.

Bonni Stachowiak [00:00:20]:

Welcome to this episode of Teaching in Higher Ed. I’m Bonni Stachowiak and this is the space where we explore the art and science of being more effective at facilitating learning. We also share ways to improve our productivity approaches so we can have more peace in our lives and be even more present for our students. Doctor Remi Kalir serves as Associate Director of Faculty Development and Applied Research with Duke University’s Center for Teaching and Learning. He earned his Ph.D. in Curriculum and Instruction from the University of Wisconsin. Before joining Duke, Remi was program leader and a tenured Associate Professor of Learning, Design and Technology at the University of Colorado Denver’s School of Education and Human Development. He began teaching 20 years ago as a public school teacher at Middle School 22 in the South Bronx.

Bonni Stachowiak [00:01:27]:

On today’s episode, Remi joins me to talk about the Fair Feedback Project, a set of evidence-based, practical strategies for addressing bias in student evaluations of teaching. We talk about why these biases matter, what a small, tangible solution actually looks like in the hands of us Educators, and how he built the project himself using AI as a creative collaborator. Along the way, as you will hear, Remi reflects candidly on his own complexity inside this AI moment and what it really takes for feedback to be fair. Remi Kalir, welcome back to Teaching in Higher Ed.

Remi Kalir [00:02:11]:

Bonni, it is such a pleasure to be in conversation with you. Thank you for the invitation.

Bonni Stachowiak [00:02:17]:

I am almost embarrassed to tell you how often you come up in, whether it’s on-air conversations, not even just on this podcast, but also on my husband’s podcast, Coaching for Leaders. In fact, most recently, I don’t even think this episode has come out yet as of you and I talking today, but I was mentioning about annotation, and how we have on our bathtub the hot and the cold, and he goes, ” Oh no, I took that off a few months ago”, so it ruined it. But your work, you just, I associate you so much with curiosity, and you inviting me and so many countless others into things we should care about, things that are relevant for us, and things to get curious about. And I’d love to start our conversation about the Fair Feedback Project before you even tell us more about what it is. Just with you talking about your experience as a professor with Feedback, what’s an example of a time you recall where you just became that much more aware of perhaps a need for some consideration around fairness?

Remi Kalir [00:03:24]:

Oh, Bonni, wonderful, and thank you for such sweet comments. I so appreciate that. And let me also just return the pleasure by saying that the number of times that I continue to refer folks back to the gold standard of higher-end podcasting, we’re right here. So thank you for you and for Dave’s, you know, legacy of, and I should say, if you don’t mind me already digressing for just a moment here, speaking of feedback and work as a professor, in anticipation of today’s conversation, I noticed that you interviewed a dear colleague, Betsy Barre, who I’ve gotten to know very well over the last few years about this very topic over a decade ago, Bonni. And so again, the legacy of this podcast and the community that you’ve built with the thinkers and these scholars and the leaders who have shaped this field and multiple fields is just astounding.

Remi Kalir [00:04:15]:

So, Bonni, my goodness, it is just again, such a pleasure.

Bonni Stachowiak [00:04:18]:

I remember that conversation so vividly with Betsy and her not wanting to only be known for course evaluations. And so I’ve so enjoyed all the conversations since. And I feel like, I have been remiss in just saying that we talk about you all the time, but not explain why. I associate you, of course, a lot with annotation. And while today’s conversation isn’t about annotation, it becomes one of those things that is just so associated with who you are and, and so everywhere I go, whenever I see it, and I do see it almost on a daily basis, you’re just who comes to mind. And it’s so great. But yes, Betsy, such a fascinating person to talk to. And I’m so glad that you could point back to that because, yes, this is a conversation that exists for pretty much if you’re ever going to be in a position where feedback is solicited, or perhaps not even solicited, but still given.

Remi Kalir [00:05:13]:

Well, absolutely. I mean, it was a decade ago when I was a professor, and at that point in time I was a junior faculty member and I was still in the early stages of my career, teaching undergraduate and graduate students and for multiple classes every semester, administering student evaluations of teaching and thinking about how I would be judged based upon not only the quality of what I was professing, but how students were perhaps learning or not in my courses. And here’s the thing, and it’s important because we’re getting into a conversation about fairness and bias that is, of course, the core focus of this new project I’ve launched, that I’m a white man in academia with many, many unearned privileges and social and institutional power that then inevitably informed how I was perceived by my students. And of course, we know, and I began my career as a professor at the University of Colorado in Denver, and I had other colleagues who were starting, either they were maybe a year or two ahead of me, or they were in my cohort, or they were perhaps coming in after I’d started, who did not look like me or did not sound like me, or did not walk through the world carrying the same unearned privileges that I still carry to this day. And so that inevitably informed then how students evaluated us in our class. We cannot disentangle in some clean way the fact that, as a white man, I know that some of my evaluations were likely rated more highly, or that my course effectiveness or my instructor effectiveness was then a reflection of some aspect of an unearned status.

Remi Kalir [00:06:59]:

And certainly, and again, speaking of Betsy and her appearance on this podcast, that decade ago, in that episode, which I again enjoyed listening to in anticipation of this conversation, Betsy’s talking about the persistence of some of these challenges. The challenge, let’s say, around gender bias in student evaluations of teaching, the fact that there are then compounding biases, let’s say, that are racial, that are ethnic, that are directed towards folks who again, speak multiple languages, but perhaps maybe whose first language is not English. We know that age and disability status, these aspects of the identity, and then in some cases, intersectional aspects of identity, which can then compound, let’s say, for faculty who are women and who are women of color, that all of this then can be reflected in the ways in which students, if they are filling out an evaluation of a course, rate their instructor and rate the course. And so I was experiencing all of this and seeing all of this firsthand when I was a professor, and also keenly aware of many of the broader discourses that, again, I know, often appear in these types of conversations around the fact that not only is there bias in these instruments and how these instruments are then used, that they don’t accurately reflect the quality of teaching, nor do they reflect what students actually learn.

Bonni Stachowiak [00:08:23]:

One of the items that didn’t come up in your examples of the ways that bias can enter actually was introduced to me by a colleague, Dr. Ludmila Praslova, and it’s lookism. So, even down to someone’s physical attractiveness and how that can play into it, there are two things among many things that I want to point out about this project. I truly think it’s brilliant, and I’m so excited not only to have you share more about it, but also to invite people to use it and share just how easy it is to use, how simple. And then also hear a little bit more about how we might be able to take our ideas that we have for challenges that we see and do something about it. But the two things I want to identify it’s just something small. I hesitated to get up there because I thought, I’ve talked to you so many times and I’ve read your books, and I’ve read so many blog posts and been so inspired that I thought, “Oh, gosh, I better really put that on my to-do list so I can, you know, is this gonna be an hour or two?” And I was really excited. It truly was something I wanted to carve the time out. It just came at a very busy series of weeks for me.

Bonni Stachowiak [00:09:27]:

And then when I got up there, I went, well, this is really very small. So I wanna encourage people, as you’re listening, there’s a lot, as Remi’s described his experiences with this, and there’s a lot of literature and a lot to consider. But in terms of you going and trying this out, it’s gonna be as simple as clicking the link that’s gonna be in the show notes and just going and getting a very small number of minutes, and then some real practical tools and resources. So those are the two things I want to frame. And I’m going to have Remi talk about it and describe, if we click on that link, what we’re going to experience. But we’re talking a few minutes here, and then we’re talking what you’re going to get, in my experience, will not cost you time; it will actually save you time, because it’s going to give you some tools that you can instantly integrate into your classes. And if you’re teaching right now, I mean right now, and even if you’re prepping the class and getting ready, you can even have it queued up and ready to go.

Bonni Stachowiak [00:10:23]:

And just know that that part of your class is already taken care of. If you use a learning management system, for example, that would let you- What is that? Predate, assign, maybe when an announcement might get sent, or put it into some type of reminder, so…

Remi Kalir [00:10:39]:

That’s it, that’s it, yes. I mean, and thank you. This is in some respects a very, very small project. You’re actually the second person in two days now who’s been providing me with, like, well, I guess, like, dare I say, feedback about some things I’ve been working on, and actually, I’m finding that when we provide very small, discrete solutions to everyday problems, people, of course, really appreciate that. You know, one of the greatest compliments that somebody ever gave me was you’ve created essentially a really good spoon. You know, that’s an analogy. But it was, you know, it was a very simple tool that can be easily used to help somebody out. And that’s what this is.

Remi Kalir [00:11:17]:

You’re right. This is a very small project in some ways. Now again, it’s got some great stories. There’s a huge literature behind it. There’s a lot going on here. But yes, let me step back and describe the project again at a kind of larger level and then break down the two tracks, one that again exists, and Bonni, you’ve described that you can just use it today. One that I’m developing, and developing still through some very kind of consciousness design work, but essentially the Fair Feedback project. And I’ll, I’m going to do a little bit of reading here because I want to be very careful about language because that really matters here.

Remi Kalir [00:11:49]:

But these are evidence-based strategies to address bias in student evaluations of teaching. That’s it. Evidence-based strategies to address bias. This project was something that I created based upon, actually, a few kind of origin stories and inspirations. I’ve mentioned a few of them when I was a professor. Even recently, there was a wonderful interview in the Chronicle of Higher Ed. It was by, and I want to again get everything right here with by Beth McMurtrie. She did a post back in February with Michael McCreary.

Remi Kalir [00:12:20]:

We’ll make sure that we also link this in the show notes. So Michael McCreary put out in early February of 2026 what’s been titled A Practical Guide to Modern Teaching Evaluation. Now again, this is the broader conversation, the broader conversation of we need to better understand how universities are approaching the evaluation of teaching, again, high-quality teaching. Whether again, that’s from the perspective of students, from colleagues, or from peers, or from professors themselves who are doing again, kind of like self-reflections on their own teaching and all kinds of broader reform efforts around that. But when I saw Michael’s resource, and I again then read Beth’s kind of interview and post about it in the Chronicle, I got thinking, like, what am I going to do about this right now? And what am I going to do about this right now that can then be in the hands of instructors this semester? To get back to your point, Bonni, that is small, that is tangible, that is immediately applicable, that they can use when evaluation season hits in April, or maybe even heading into May. And so that immediate, in the hands, two minutes, if that track in this project is called the instructor track. And as you mentioned, Bonni, you can go there, and you, again, less than even two minutes, you basically put in some very high-level descriptive information about what you’re teaching. Now I should mention here that none of this input data is intended to be personal in any way.

Remi Kalir [00:13:54]:

It doesn’t ask for your name, it doesn’t ask for your institution, doesn’t ask for any of the things that may be identity markers of bias, including like lookism, as you mentioned, Bonni, a moment ago, none of that’s asked here. What is asked is what kind of a course are you teaching? Like, in what discipline are you teaching? A STEM course? A course in the humanities? Maybe like a course in professional education? Is this an upper-level course versus an introductory level course? Is this a course that is a smaller seminar-style course, let’s say just like a dozen students, or is this a really large high-enrollment course with like over a hundred students? And then you can add in maybe a little bit of additional information, like, do I teach online, or is this face-to-face? Is this a required course or is it an elective? These are just enough questions that one, they’re not invasive. Two, they’re not requiring you to turn over any kind of personal information, but they’re all research-based, because all of these coarse characteristics then directly tie back to where bias might appear. So, for example, we know there can be bias actually in small courses where one or two evaluations carry, in this case, more statistical weight than in a larger course where, again, you may have means and medians that are showing across, let’s say over 100 students that are completing an evaluation. We also know that bias shows up when courses are required versus when they’re elective. We also know that bias can show up for online courses versus courses that are face-to-face, let’s say on campus. When you then add in all that information, this system, which has been designed with AI, but is actually not using AI in the generation of responses. And I can, I can unpack that a lot, but let me just cue that up a little bit here.

Remi Kalir [00:15:48]:

It’s using a deterministic set of responses to then provide to you, Bonni, what you just mentioned. So it can provide things like I can queue up an email statement or an announcement in my LMS. Let’s say I want to share a quick set of comments before my students take an evaluation. I want to share that briefly in class, or maybe I want to have a full-on class discussion with my students about implicit and unconscious bias. Or maybe I want to have students actually complete a very short, what’s called a self-affirmation prompt. If you actually have students write about affirming values as a kind of open, free right before they complete an evaluation of teaching, it actually has been shown to mitigate bias. And there’s all these different little intervention strategies along with language that can be shared with colleagues and language that can be shared in, let’s say like a portfolio or a dossier that this system will generate for you.

Remi Kalir [00:16:50]:

And it generates all of that, that is, again, evidence-based. It’s linked to specific studies where this type of, this type of anti bias mitigation was shown to be effective. And all of that is generated then tailored to those course characteristics that you shared. All of that, though, and I know I’m rambling now, folks, because I’m again, I find this to be so exciting. But all of that is generated in a way that again doesn’t invade any kind of personal privacy, doesn’t collect any data. It’s done in about two minutes, and then you can immediately copy and export that and go and use it. And that to me is a very tangible, in your hand way for individuals, in this case, to try and create some anti-bias mitigation, should that be useful in their course context.

Bonni Stachowiak [00:17:39]:

I love that you pointed back to the Michael McCreary piece. I was blown away when I saw it, too. We just spent an entire academic year completely revamping. I mean, revamping isn’t even the right word. This probably isn’t polite, but we burned it to the ground. We started from zero because we had sort of fallen victim prior to that to just trying to add more and more and more. And, you really, at some points, have to examine an entire instrument and is it accomplishing what you want it to? The really nice thing about this piece is no matter where you are in the process, there’s still more to be done.

Bonni Stachowiak [00:18:12]:

I don’t think we’re ever done with this work, but it, to me, it was so representative of where are, where might you be. I don’t know that it was necessarily entirely chronologically based, but for me, I interpreted it that way, and I saw what might be the next footholds for us and what’s necessary. But what’s so brilliant about what you’ve done here? Because when I looked at his piece, his was much more macro in my mind but very rooted in the research. But then this is just kind of like, oh, is this where you are? Oh, okay. And it’s just so easy to use. And I was fascinated. I would like to dig in a little bit because I know we’re going to be talking about too, your use of artificial intelligence.

Bonni Stachowiak [00:18:54]:

And I really want people to hear the difference between AI doing this every single time one of us were to go up there and do what I did, versus deterministic. There’s some key elements here that would be important if any of us were ever going to try to address any problems we might face or try to build in some systems or tools, understanding the difference between what Remi said there. So could you talk a little bit more about why design it in a deterministic way and the kinds of differences that that makes, both for you, as the person who is behind this project, because this is you as an independent project, what does it all mean? Would you make that choice? And why might we want to think about this if we were ever going to try to embark in a similar endeavor?

Remi Kalir [00:19:43]:

Thank you, Bonni. Yes. And so let me thank you for reminding me to remind all of us of that. That, yes, I’m here really speaking about this project in my capacity as Remi. As again, a former professor turned member of a CTL, although again, my day job is at Duke University. This is not a Duke University project. Again, I was inspired because of my prior experiences as a professor, because of the fact that I care a lot about our field and my colleagues who are teaching in all kinds of circumstances, and the fact that again, there are many people who are experiencing the effects of these patterns, of these structural patterns of bias, who, again, don’t look like me. So what can I, again, what can I do? How can I show up as an individual in this? And so, yes, let me again be extremely clear. And I’ve written about this, not only in the project documentation, but I’ve been very public about this through some of my LinkedIn posts about this project.

Remi Kalir [00:20:37]:

I designed this using Claude code, and I’ve been now about two months into pretty serious use of this agentic coding platform as I think about how it changes my stance and my practice as a designer, and what I can create. And how we can use existing literatures, we can make use of what we know is research-based truth and turn that into practical products. Products, not that are monetized, again, this is a completely free and open project, but products that can then be put in the hands of people to again, then make a difference. In this case, they’re teaching and learning. And so yes, this was designed between me, Remi, and Claude AI. And yet there became a point in that design process where I said, as I was collaborating with this AI system, I don’t want people talking to Claude in this project. And let me say, as a bit of a digression here, we are increasingly seeing projects, and I was actually just messing around with one yesterday, that is not a project that I’m affiliated with. That’s essentially a kind of course design slash course knowledge tracking platform, very clearly designed using Claude code.

Remi Kalir [00:21:59]:

Again, we’re seeing a lot of these projects pop up. That project, though, does have a chatbot built into it. And I was able to go into that chatbot and start to ask it all kinds of, kind of off-the-wall, goofy questions, and get it to kind of reveal that it, yes, is indeed Claude and that it’s Claude that’s getting piped in to this project. And then I got it to like act in some very kind of odd ways. And I don’t want to waste listeners’ time about that. But there are some, even though again, I’m using Claude to design something, there are still some very, I think, real vulnerabilities and risks to deploying an LLM or even a well-scoped chatbot into a system because you don’t necessarily know how anyone might then end up talking with that thing. And it will ultimately then reflect the project.

Remi Kalir [00:22:51]:

And so in this case, I did not want that. I did not want people coming to the Fair Feedback project and then having long-winded, tangential, potentially problematic conversations with Claude as a chatbot. That also would mean that data were being sent back to anthropic servers and that things just get messy. I did not want that to be in any way affiliated with my name as an individual. And so yes, Claude helped me to actually co-author some of the writing and the statements that appear. I’ve been transparent about that. Claude was essentially a research assistant. As we combed through the literature, and we figured out ways of saying, okay, if we know from the literature that this kind of implicit bias framing is actually more effective than gender framing.

Remi Kalir [00:23:42]:

And that’s actually again what the research shows. If you talk to students about unconscious bias as opposed to explicitly saying, well, women are often receiving faculty who are women, women of color are explicitly referenced in trying to mitigate bias, that can actually backfire. And then, actually, you can see in experimental conditions that there can be actually lower ratings when that kind of gender focused framing is provided as opposed to a framing that is about unconscious bias or implicit bias. And so then I need to tell Claude, when you create then the strategies and the statements for this project, we need to privilege what the research shows as effective framing in that way and not in an alternative way. And so I was working with Claude in that respect, and of course, there is then an entire technical design aspect of this, where we are then creating code and troubleshooting code, and I’m actually mucking around, and some of the HTML that’s behind this, and actually creating this living web resource. But the deterministic piece, and I’ve, I know, Bonni, I’ve taken a very long time, I’ve answered your question here. But the deterministic piece and is very important, which is to say that there’s only a set number of response types, a set number of research that is referenced, that all of that statement development is grounded back in the sources that we cite, and that you can’t use the Fair Feedback project to create something essentially out of nothing or something that is not already pre programmed into this system.

Remi Kalir [00:25:21]:

In that respect, the loop is very tightly constructed, and you’re not going to be able to go to this project and create something that is not again, research-based, that is not grounded in the literature, and that is not you talking to an LLM. You are not talking to Claude when you show up to use the Fair Feedback project. Even though I talked a lot with Claude to create the Fair Feedback project.

Bonni Stachowiak [00:25:47]:

You mentioned using Claude code, and then you also talked about, you were getting in there a little bit with the HTML. Where does the HTML live? I know you have a domain name, but where, where is that actual HTML living?

Remi Kalir [00:25:59]:

Yeah, thanks, Bonni. I mean, this is the thing, right? Like again, it’s just, it’s just me, Remi, shelling out $20 a month to Anthropic. I should have mentioned, by the way. Let me, let me just, if I can quit just like, but this is, you know, I laugh only because I. The alternative would be to cry, actually, because we are in such a weirdly entangled AI moment. Let me just go on the record and say for anybody that’s listening to this, because you mentioned my books earlier, like I am currently a member of the class action lawsuit that is suing Anthropic, because I know that Anthropic ingested my first book annotation with MIT Press, I very willingly joined that class action lawsuit.

Remi Kalir [00:26:46]:

I can’t imagine, after sharing part of the author damages with both my publisher and my co-author, that I will be enriched with the settlement from this class action lawsuit. There might be a LEGO set for my 6-year-old that comes out of that. But in any case, this is the complexity of living in this AI moment. I am, on the one hand, part of a class action lawsuit, suing a company that I am also now a paid subscriber of because it helps me do these kinds of projects. And I just want to be very, very frank about, you can call it my complicity, you can call it my complexity, whatever you might call it, but I am very much entangled in this AI moment, trying to understand how I am navigating all of this. And I think that we’re all trying to make our own sense of all of that. So let me again just be very, very upfront about where I stand in this project.

Remi Kalir [00:27:43]:

Because again, it’s just me trying to figure some things out. But having said that, yes, when Claude Code and I are creating then, the code and the files for this project, ultimately I’m able to get them onto, you know, GitHub. The code is open, it’s there. If somebody wanted to fork this as the term is and spin up their own version of that, they could. I should say that I’ve gotten some incredible feedback from folks, not only individuals, but also the beginnings of some institutional interest in essentially saying how might we create a kind of institution-specific version of this project. Which you could, then again, because it’s using, again, literature and language in this kind of deterministic way, you could, Bonni, you mentioned, again, your own institutions, let’s say, teaching evaluation reform efforts. An institution out there, I’m not suggesting, Bonni, you’re a student.

Remi Kalir [00:28:37]:

But at any institution out there could say, “hey, we’ve reformed how we evaluate teaching. We have new kinds of instruments. We actually are trying to incorporate some anti-bias language into, not only the instrumentation, but how, let’s say, professors, instructors show up and introduce this to their students, or again have an email announcement or an LMS announcement”. You can take this project right now. This is an open invitation to any institution out there to take the Fair Feedback project and create your own institution-specific fork variants of this project. Go for it. It’s there to be used. That code is on GitHub; it’s out there, and I would be thrilled to see this iterated in that kind of a way.

Bonni Stachowiak [00:29:30]:

One thing that Remi and I haven’t spoken about yet in today’s conversation that I want to make sure I share is the way we can mid class feedback. And I would like to share that in the context of thanking Poll Everywhere for the partnership between them and teaching in higher ed. I get the great privilege of sharing every couple of episodes or so, a tip, and today’s tip is about getting effective mid-class feedback. You can take a lot of the same principles around reducing bias in evaluations and introduce that when you ask for feedback midway through the class, or perhaps even more frequently than that. Like was so well modeled for a very long time by Stephen Brookfield through the way he suggests that we ask for feedback. This can be just a small thing. Sometimes it feels like a heavy lift.

Bonni Stachowiak [00:30:25]:

But especially with Poll Everywhere, we can just align with the feedback we already know students will be asked to give at the end of the semester, and be able to bring some of those things up sooner in the process, and be able to hear about what you might be able to shift. Or perhaps it’s really not something you need to do. Perhaps it’s something that the class as a whole needs to commit to doing. I try not to make it when I do mid-class feedback, not all about what do I have to improve on? Not that there aren’t always things I need to improve on. But I look at what should you, as the learner, what should I, as a facilitator of these learning experiences? And also, what should we as a learning community do differently in order to enhance our experience and deepen that learning? And again, you can use many of the principles that are talked about here, but Poll Everywhere just makes it so simple to be able to gather that information. You can have it where you share it publicly, or perhaps you just gather the data and then reflect on it, and then come back at that following class and share it out.

Bonni Stachowiak [00:31:29]:

It’s totally up to you. It is essential, though, that you share back with the class what it is that you have gleaned from their feedback and any concrete changes that you commit to making in order to enhance that learning or make the experiences better. Better for people. That’s a really important step in the process. Thanks once again to Poll Everywhere for your partnership. I’m having so much fun with these messages. If you have any ideas, by the way, on how to use polling, please email me at feedback@teachinginhighered. I would love to hear from you, and it’s fun to share ideas that you have about using polling.

Bonni Stachowiak [00:32:10]:

This is the time in the show where we each get to share our recommendations, and I want to share and encourage people to go look at a post by someone named Doug Belshaw, and he wrote a tech disclaimer, and he begins, and I’m going to read his words, but I’m quickly going to jump into him quoting someone else. I’m reading from Doug’s post on his tech disclaimer. A year ago, I published this on Thought Shrapnel, which is his blog, but little tiny blogs with just ideas that he has that are still seeds. So he posted that on Thought Shrapnel, which quoted Elena Rossini. So now we’re quoting Elena: I just think that people who write about technology should have a disclaimer about the tech stack they use, in order to see if they’re walking the talk. And if people who speak the truth to power feel they need to be on VC-backed, centralized for-profit social networks, sure, no problem. But I believe that anyone speaking up against the broligarchy should be active on the Fediverse too, a galaxy of independent free open source networks that is not funded by billionaires or crypto bros.

Bonni Stachowiak [00:33:23]:

And we’re back to Doug now, quoting him here: Independent journalists regularly have an ethics page or disclose in the pieces they write any financial interest that they might have in the topics they cover. What I’d like you to go do is look at his tech disclaimer, and I will say he also has built with another collaborator a whole way for organizations to look at the entire institution’s tech stack. And to what degree is it overly US-centric? I should mention that Doug is not from the United States and has, sees some risks of being overly invested in the United States technology sector and some real ethical concerns there as well. But for him, he’s looking at his tech stack, and there are a couple of reasons why I want you to look. One, I want you to look because I’d like you to look at what it looks like to live out your values, but so resonant with what Remi said earlier about complicity and complexity that this is not easy to do. And one of the things he writes in his conclusion is that he recognizes that he has some privilege being independent, that he gets to choose. And he recognizes that people who work for employers aren’t going to get to pick their tech stack.

Bonni Stachowiak [00:34:46]:

And that is just more complicated than that. And I’m going to invite people, if you’d like to, to also have a look at a post that I wrote on my tech stack. I will say, I cringe a little bit because I feel a little bit, speaking of tech bros, I feel like a tech girl here because I think yes, I definitely get sometimes caught up in the consumerism of it. I do really enjoy new gadgets. I’m fascinated by what they can do. And I’m sure that my thought is not where it should be in terms of that, and I just want to be very frank about that. But I also want to celebrate the small steps.

Bonni Stachowiak [00:35:18]:

And I think especially because Remi’s here talking about that. I’m looking at it, and I’m going, “Oh, I have colleagues who would be gently teasing me about, yeah, I’m a little, a little wrapped up in an ecosystem that is a fruit company, if you will”. And we tease each other back and forth and such. But I will say that sometimes it’s you can just get overwhelmed, and you can go, well, there’s nothing I can do, and my work subscribes to this, so there’s nothing I can do about the kings of Microsoft and all the things. But I look, and I go, you know what? Dave and I decided we didn’t want to have people who visited our respective websites for our podcast get constant pop-ups and steal their private information for them. So we did decide some years ago to move off of Google Analytics and to move over to a different platform that doesn’t require that. Do I really need to know the private information of people and inconvenience them with pop-ups and things? So even when I look at my tech stack, I go, okay, you know, there, there certainly is much that could be critiqued, but I feel good about taking small forward movements toward my values and taking opportunities to reflect. And I will be candid with you, too. It is also fun to think about the things that I have to experiment with and the things that we are able to use these tools.

Bonni Stachowiak [00:36:38]:

I’m going to finish with this. This is from… This is from that same post, and now he’s quoting William J. Mitchell: I think it’s probably a good idea for us to all reflect on how tools shape our lives. Tools are made to accomplish our purposes, and in this, they represent desires and intentions. We make our tools, and our tools make us.

Bonni Stachowiak [00:37:06]:

By taking up particular tools, we accede to desires, and we manifest intentions. And that- I’m not quoting anyone. Actually, I’m going to quote a movie now. This is all wrapped up in The good, the bad, and the ugly, so I’m happy to share, and I look forward to hearing from anyone else who knows of other tech stack things. I do enjoy seeing what technology people are using and how they’re reflecting on it. So I get to pass over to Remi for whatever he’d like to recommend.

Remi Kalir [00:37:36]:

I love it, oh Bonni, that’s so good. It’s hard to follow. It’s hard to follow that. Thank you for reminding us to live, live our values. You’ve got me thinking now, then I’m going to throw another recommendation into the mix here. Let me start with what I had queued up, and then I’m going to throw a few more things into the mix. But, but let me start with what I started because I mentioned earlier.

Remi Kalir [00:37:53]:

And we’ll have it in the show notes again, Michael McCreary’s piece again, A Practical Guide to Modern Teaching Evaluation. It’ll be there again, a big shout-out to Michael for doing this, and for really tracking a lot of that kind of, as Bonni mentioned, this kind of macro-level institutional-level work to understand the kinds of reforms that are happening in this space. If it, again, wasn’t for that broader conversation, certainly this conversation that you and I are having today, and the project that I’ve been able to work on wouldn’t be possible. So I just want to really give credit where credit is due. Michael mentioned, and I’ve been able to have a chance to read a bit about, a book I want to recommend to folks. It’s just out, it’s Transforming College Teaching Evaluation: A Framework for Advancing Instructional Excellence. I think this is really going to become the defining text of this moment in this, again, space. It’s by Anne Austin and colleagues.

Remi Kalir [00:38:47]:

It’s reporting on cross-institutional work, it’s reporting on longitudinal work. It’s, again, looking at that broader question of how do we make sense of, in this case, the evaluation of teaching and doing so in a way that is transformative and that is institution-wide. And so again, cannot recommend, it’s from Harvard Education Press, can’t recommend it enough. So we’ll include a recommendation to that that book in the show notes as well. Okay, Bonni, you got me thinking about living our values, though. And I know, I hope it’s not too improper on a podcast to recommend somebody else’s podcast.

Bonni Stachowiak [00:39:21]:

No, it’s wonderful. It means they’re really good then.

Remi Kalir [00:39:24]:

Having said that, having said that, I do want to recommend a podcast that is much younger. It’s a kind of new Kid on the block, dare we say, Bonni. And it’s explicitly about AI and the data moment that we live in. And it’s called The Data Fix by Dr. Mel Hogan. I’ve known a few folks, I’ve been tangentially connected to a few folks who’ve shown up on the podcast over the years. It is very critical politically, socially, epistemologically; it’s bringing in scholars, social critics, folks who are doing tremendously valuable work in this moment. Whether it’s around the environmental impacts of AI, looking at things like data centers and water consumption.

Remi Kalir [00:40:10]:

It’s taking a very hard look at how some of the discourse around so called educational productivity and what it means to teach and learn, and this kind of, you know, using AI in that context, not to perhaps build tools like what I’ve discussed, but to say, well, we’ll just have personalized AI tutor bots for every, you know, kid on the block. And in any case, I, I, appreciate her podcast for a variety of reasons. One, it’s your point earlier, Bonni, of really trying to live perhaps more critically about, it resonates with me. I’ll just say that it resonates with me. It reminds me that people see this moment in very different ways, that people are circulating discourses about again, the kind of AI turn, the data turn in this moment, in a variety of ways. I also just find that it’s a very creative podcast. It’s organized kind of thematically; it comes out maybe once a month.

Remi Kalir [00:41:03]:

And as a newer podcast on the scene, I find it to be a kind of refreshing take, and so, can add that into the mix as well. Bonni, you got me thinking about that, so I’ll add that in as well.

Bonni Stachowiak [00:41:13]:

So much to look at here. I get to revisit the post and check out the book that he mentioned and that you’ve recommended here, too. And a new podcast. Thank you so much. And until we meet again. Right, because we both have decided it was too long this time. So, so looking forward to having you back the next time. I know that there’s going to be a lot more to discuss.

Remi Kalir [00:41:34]:

Bonni, thank you again. I am just so inspired by the legacy of your work and by the conversations that you curate. You know, the ways in which folks have built community around the gifts that you’ve shared is singular, and I know you know that. But we cannot take for granted that the kind of labor that you and Dave, you know, that this is atypical in the best of ways and has defined how people show up in their profession in ways that is really immeasurable. So, Bonni, thank you so, so much.

Bonni Stachowiak [00:42:09]:

Oh, those words mean a lot, and can’t wait till we get to have the next conversation. It’s always so good to talk to Remi Kalir, and always so good that you take the time out of your week to listen. Thank you so much. Today’s episode was produced by me, Bonni Stachowiak. It was edited by the ever-talented Andrew Kroeger. I have somewhat recently redesigned the Weekly Teaching in Higher Ed email. It’s now called Field Journal. I hope you’ll check it out.

Bonni Stachowiak [00:42:41]:

Head over to teachinginhighered.com/subscribe. It’s hard to describe until you see it, but you get a what I’ve been reading, listening, noticing. I’m getting lots of positive feedback, and I hope you’ll join in on the conversation through that weekly Field Journal. Thanks for listening, and I’ll see you next time on Teaching in Higher Ed.