Bonni Stachowiak [00:00:00]:

Today on episode number 555 of the Teaching in Higher Ed podcast, a big picture look at AI detection tools with Christopher Ostro. Production Credit: Produced by Innovate Learning, Maximizing Human Potential.

Christopher Ostro [00:00:14]:

Produced by Innovate Learning, Maximizing Human Potential.

Bonni Stachowiak [00:00:22]:

Welcome to this episode of Teaching in Higher Ed. I’m Bonni Stachowiak, and this is the space where we explore the art and science of being more effective at facilitating learning. We also share ways to improve our productivity approaches, so we can have more peace in our lives and be even more present for our students. As regular listeners know, I typically read the bios or paraphrase the bios of guests, but I’m grateful that Chris is here, and I love bios that are in the first person. Chris has agreed to share a little bit about himself as we get started. Chris, welcome to Teaching in Higher Ed.

Christopher Ostro [00:01:07]:

Hey there, Bonni. Thanks for having me. And, yeah, I mean, I didn’t know I’d be signing up for additional work, but it’ll rot. So a little about me. So, hey, I’m Chris, and I love teaching. I consider it just like a core part of my identity. Right? I’ve taught at the University of Colorado in some way since 2011. First as a grad student in the classics program, then as an adjunct in the program for writing and rhetoric, and I was an assistant teaching professor in the division of continuing education teaching those same, writing and rhetoric courses.

Christopher Ostro [00:01:34]:

I’ve also tutored a variety of topics since 2008, just, you know, for a side hustle. My passion for courses on actually initially started as a necessity. You know, adjuncts live and die off the reviews, and to me, it just made sense to do whatever I could to make my classes as positive as possible. Right? If they’re super polished, that’s great for my career. In addition, a bigger variety of classes in my quiver seemed like it would help guarantee some course loads. Right? If I can teach online or in person or hybrid, if I can do evenings, 4 weeks, 6 weeks, 8 weeks, 15 weeks, just seemed like a good way to guarantee that I’d always get a class. So little by little, I began doing more course design work with the learning design group at CU, and I love that I get to work with them. I help them run a month long introduction to online teaching, but I also give talks to them pretty regularly, including the one that got me invited here.

Christopher Ostro [00:02:17]:

And I do a ton of 1 on 1 mentorship with them, often with faculty designing their first classes or faculty who are really passionate but maybe a little unfocused. And, honestly, it’s one of the most joyful parts of my job. In spring 2023, I noticed AI suddenly show up in the classroom and causing tons tons of issues. I think a lot of faculty remember this moment. I think all of us remember that first paper we got. We’re like, this feels weird. And I started trying to update my courses just to respond to this new tech and how it was impacting me. But because of my unique role as both faculty and staff, I was able to observe how this was impacting kind of everything and every facet.

Christopher Ostro [00:02:49]:

Every office I was working with was having a crisis about this, and so it allowed me to kinda get involved in a variety of ways. So I’ve been involved in vetting products, discussing policy at the university, leading trainings, but also, again, doing a lot of that one on one mentorship and trying to advocate for faculty points of view on this as well. You know, if I don’t know if you’ve seen the movie Blade, but I’m sort of like that day walker except, like, administration walker. I don’t know. I’m half faculty, half staff in my contract, and it gives me some unique perspectives, I think.

Bonni Stachowiak [00:03:13]:

I am so glad for us to get to have this conversation, Chris, and thank you for that wonderful introduction to yourself so we can get to know you a bit. I feel like at least I am self aware enough online to know which of the memes are really overdone, and I should avoid them. But there is one I still I I’ve not been able to find a less overused meme than this, and it’s Jason Bateman, and he’s kind of something happens and he’s kinda like, like, he’s he has this expression, and and and Chris is nodding his head. So, I’m trying to I don’t even know the context that it’s from, but that is how I feel. And I will link to this in the show notes, so if anyone has no idea what I’m talking about, I’ll link to this Jason Bateman, the expression on his face, but it’s kind of like, yeah, but but no. No. Not really. And that is how I feel with friends, dear friends, and colleagues who are understandably frustrated with students using artificial intelligence and wanting there to be a panacea for it.

Bonni Stachowiak [00:04:19]:

And the panacea that they want is give me the detection tools. We had it for plagiarism. Let’s have it for AI, and that’s why I was so excited to learn a bit from you today. I almost titled today’s episode a legit interesting lit review, but I actually find a lot of literature reviews interesting, but you’re gonna be providing for us an overview of what do we know about AI detection tools. So let’s start off, Chris. Just tell us a bit, what are some of the common misconceptions that we can have about AI detection tools?

Christopher Ostro [00:04:53]:

Yeah. I mean, I’m gonna say up top, I’m pretty biased in this. I think there are certain things they are clearly demonstrably good at and things they are demonstrably not good at, but I really do think that if you talk to people, you’re gonna get extreme views all over the map. I I have had an at that presentation I gave, there were a number of people there who firmly believe that AI detection can’t detect anything reliably, that it’s essentially a coin flip. And I think that is a pretty big misconception. But also, there were people in that room who thought, oh, yeah. This is actually a magic bullet. It’s perfect and couldn’t make a single error.

Christopher Ostro [00:05:23]:

And that’s also not definitely not true. Right? I think there are very clear things that it can and can’t do. And it also increasingly seems like it depends on the tool. Something like GPT 0 has very different capabilities than Writer AI than Turnitin. And, also, they’re developing these things in different ways. Right? Turnitin is you know, I don’t I’m not not sponsored by anyone, but I will say, like, the approach they’re taking is different than everyone else. GPT 0 is trying to essentially make an algorithm that detects AI writing, whereas Turnitin with their database of a few 1,000,000,000 student papers that they have private that AI can’t train off of, they’re essentially trying to make a human writing detector, and that’s really interesting to me.

Bonni Stachowiak [00:06:02]:

And what are some of the research that you have gleaned from around the use of these tools?

Christopher Ostro [00:06:09]:

So one of the things I really want faculty if you only hear one thing from me that you internalize, one of the big ones I want you to get away from this is it is basically impossible for you to be reliable at detecting AI writing. Right? I think all of us, myself included, I teach writing. I like to think of myself as a writer. I I dabble with words on occasion. Right? And even I, I rarely feel a 100% confident. And even then, it’s only if it’s, like, unedited output or something like that. And so one of the first things the research shows this. You know, one of the ones I would cite right off the bat was Cassel and Kessler.

Christopher Ostro [00:06:41]:

They asked a bunch of trained linguists who also worked in the field of linguistics as editors. And across the board, not a single one of them was able to get all 5 right. They literally just gave each of them 5 writing samples and said AI or human. None of them got it a 100% right. And on the whole, the entire group was barely better than guessing. And I think about that a lot because I think all most faculty that I work with, myself included, have some level of confidence that we are smarter than the average pair, that we can look at these things. And I’ve seen enough papers that I can reliably pull this out, but I actually think that instinct really needs to be thought here. Right? I think that it is don’t get me wrong.

Christopher Ostro [00:07:16]:

I am not saying that you have never looked at a piece of writing and been like, oh, that’s definitely AI and been right. Of of course, that’s happened. But there also definitely have been times where you have looked at it and been wrong, and you’re just not kind of factoring that into the personal data bank, so to speak. And when we’re talking about student writing, that can really matter. Right? If I look at an article online and think it’s a and it’s human written, whatever. Who cares? Right? But if I do that in a classroom, I could really mess up someone’s scholarship. I could mess up someone’s livelihood, or someone’s career in the very beginnings. So that’s one of the things I I really like to point you off the bat.

Bonni Stachowiak [00:07:46]:

I think another another thing that can be helpful as we start to explore this is to not talk about it in in dichotomous terms. And just like with plagiarism, there is a whole spectrum of things where I’m literally paying someone to write an entire paper for me, and that’s off of a prompt, perhaps, you know, accompanied by a rubric or something like that Yep. All the way down to using a tutor or a writing center coach, something like that, and and which, of course, would not be not be, missing any academic integrity. I wanna be clear on that. And that’s where I have been confused myself a lot and continue to this day to still wrestle with this idea even for myself of wanting to be transparent about when I do or don’t use it. And but then I’ll think, like, well, I’m I have this really cool app on my watch where I can just tap a they call it a complication, but, basically, tap a button on my screen, if you will, and start talking. And then it’s it’s a app called WhisperMemo, and what I like about that is it puts it into paragraph form, which to me is a lot more readable later. And and so what at what point did AI start and stop helping me when those were my original ideas? And what what would the scholarship say about that to the degree Sure.

Bonni Stachowiak [00:09:03]:

To what degree should I be sharing that? Of course, it depends a lot on context, etcetera, etcetera. But it that’s a really, really confusing part. And I was gonna mention one thing. You’re we’re still talking about misconceptions and and what the research tells us. These trained linguists were looking at a writing sample. And most of the time when I hear people thinking, oh, the student definitely used artificial intelligence, it’s in contrast to the other writing that they’ve done in a class, and that being a role. Wait. What happened here? This sounds like an entirely different person, or was this in fact even a person?

Christopher Ostro [00:09:36]:

Yeah. A 100%. And I I try not to think of these things in that kind of dichotomous way, but this is just what that research did. Right? That’s the only reason I was framing it that way. The fact is I don’t think most students are using it in a dichotomous way. My experience has been that most students, even if they are well, let me back up a step. I think there are tons of students I interact with who are really just curious and trying to use these tools to dig deeper and understand them better and under help with their learning. And they’re also but we would be remiss if we acted like there weren’t just ne’er do wells that exist as well.

Christopher Ostro [00:10:03]:

And I one of the things that frustrates me as a teacher is I feel like people are pretty consistently over or underestimating the amount of cheaters in basically every conversation I

Bonni Stachowiak [00:10:11]:

have.

Christopher Ostro [00:10:12]:

Right? There are always a few students. It’s not many, but, like, they will exist. And I think acting like every user of AI is actually trying to achieve some sort of digital Nirvana is misleading. Right? But, also, on the other hand, assume that every student is just some snidely whiplash mustache twirling villainous cheater is also not the case. Right? Most students don’t know where the line is because most universities or most faculty don’t know where the line is, And I think that creates a lot of the issues up front. But, yeah, in terms of, like, other misconceptions, you were asking about the Whisper app. Yeah. Right? One thing I find interesting is the way these softwares work is I I’m the most familiar with Turnitin, so they’re who I’m gonna talk about the most.

Christopher Ostro [00:10:50]:

But, like, that is not in any way an endorsement or anything like that. But they do they basically scan each part of the paper in context multiple times. So Turnitin does, like, all they estimate they try to get to 300 ish word chunks, and they kind of take that panel and go through the whole paper that way. So every sentence is scanned in context at least a few times. It’s with the stuff before it, the stuff after it, a few times where it’s in the middle. And then it’s run through these algorithms that give it a score. Usually, a 0 means that they’re 100% sure it is human written, and a one would mean they’re 100% sure it’s AI written. Nothing ever gets a 0 or 1, basically, though.

Christopher Ostro [00:11:23]:

Everything is a decimal. The only time something like Turnitin highlights something as AI written is when all 6 of its scans of that sentence average it to 99.97% likelihood, I believe. So pretty high. And one of the things, you know, a major thinker in this field, James Faulkner, is someone I talk to often. And one of the ways he put it in one of his writings that really struck me was that this is more accurate than PCR testing for COVID. Right? So if you are if you are someone who has ever used a PCR test to inform a decision you’re making, this is more reliable. And that being said, though, when it misses, those misses really matter. Right? And I think that’s where a lot of the anxiety over this comes.

Christopher Ostro [00:12:02]:

Right? Because faculty are most interested in something that can help us save time in the classroom. AI detection can do that. And 99.7% accuracy for that need is enough. But for the administrative need of, like, that might be the reason a student loses their visa. If they’re an international student, if they get accused of academic dishonesty, that’s probably not high enough. And I think that disconnect is where a lot of the tension currently comes from. I honestly think that people outside of the classroom don’t realize how much work has increased inside of the classroom because of AI. But vice versa, I think people inside the classroom don’t realize all of the complexities that come with suddenly just enabling something like that and all the processes that would need to be updated.

Bonni Stachowiak [00:12:40]:

Tell us about some other research that you’ve uncovered around the use of AI detection tools.

Christopher Ostro [00:12:46]:

Sure. So there’s a slew of really great papers. A bunch of these, I’m sure, will be linked to the show notes. Right? But Perkins et al is sort of the big cited one where Perkins and his team, they went through using a bunch of different AI detection tools and also a bunch of adversarial techniques for circumventing them. Right? First of all, just asking Chatche P to write a paper saying that doesn’t scan like AI or something is actually pretty effective. So they did things like that. They they tried different ways of jumbling the text and all that just to see how reliable or not these things are. And what they generally found is that most things were some amount of accurate with completely unedited text, and then there’s a big drop off a cliff.

Christopher Ostro [00:13:27]:

And this is kind of, I would argue, what you see in most of the research. I do wanna add this one caveat, though, that a lot of these studies so the the big three that people cite the most are Perkins et al, Sadasivan et al, and Weber Wolf et al. And these 3 all generally I mean, there’s a lot of nuance to them, but they generally find the same thing. That totally unedited AI detection is reliable, some amount of reliable, arguable reliability, and then everything after that is increasingly less reliable. The more manipulation of AI output has happened, the less likely it is to be taught. To me, though, as a faculty and instructor, I hear that in, like, as a good thing. I want my students putting work in. I want my students.

Christopher Ostro [00:14:05]:

If they’re using ChatChate Wiki, I love that in some ways. I think I want them getting practice on these things that are gonna be part of their future careers, their future lives, and I love that my classroom is a stage for that. But it is also illuminating to me that it’s pretty reliable catching the a 100% because that is actually the thing I most wanna push back against. And so those three studies really focused on the efficacy and the accuracy of those things. And we can come back to that in a second because I also think there’s some methodological errors and and counting errors that they make in a few of those studies, most specifically with Turnitin. Mhmm. Turnitin reports its scores in a totally unique way. It’s the only one that gives you a percentage of the paper that it thinks is AI written, whereas every other one gives you a score that is how likely it thinks the thing is AI written.

Christopher Ostro [00:14:48]:

And those are just different scores that often make it hard to compare to other tools.

Bonni Stachowiak [00:14:52]:

So Turnitin how does Turnitin track and then how do the others track you?

Christopher Ostro [00:14:55]:

Yeah. So if I have a piece of, you know, a a piece of student writing that I think might be AI written, if I put it through something like gpt0 or writer dot ai or any of those other ones, it’ll give me a score of what likelihood it thinks that that is AI written. Mhmm. It’ll say, you know, after looking at it, running it through our our tests, we think this is 42% chance that this is AI written.

Bonni Stachowiak [00:15:15]:

And and in that case, treating it like a dichotomous thing? Mhmm. Okay. Good. Good. Good. I’m not

Christopher Ostro [00:15:20]:

Most of those most of those do have advanced versions. So I have a paid version of GPT 0. With that, you then can go into the paper, and it’ll review the sections it thinks are AI written, and it’ll tell you certainty on each of those individual sections. Mhmm. But Turnitin does a different thing where what Turnitin does is they give you a score, and that represents the proportion of the paper it thinks is AI written. So if it gives a paper a 41%, it’s not saying we’re 41% sure. It is saying we are above 99.97% sure that this 40% of the paper is AI written, which is different. But that often leads like, if you go through the data and the charts, especially in Weber Wolf and especially in Perkins et al, you can see places where I think just some data doesn’t line up because of that disconnect.

Christopher Ostro [00:16:01]:

It’s an easy mistake to make, but it it is in the data. Some other research that I really wanted to highlight quickly is Liang et al. I think this is one of the ones a lot of people have heard about, and there’s a lot of of rightful anxiety about this. Right? Liang et al. Was a team that found 7 different detectors were more likely to falsely accuse pieces of writing written by ELL students as being AI written. And that is a really troubling, like, piece of research if true. But part of what I think is important context to keep in mind here is when you first of all, the things they were looking at were really weird comparison. The things they were comparing were 91 TOEFL papers and then a bunch they I don’t remember if they gave a number, just a handful of 8th grade papers from American students.

Christopher Ostro [00:16:41]:

And those are the things they were comparing about accuracy. I think that’s an just kind of a weird dataset. And there have been subsequent studies that try to confirm this, and, actually, most of the studies tend to go the other direction. Right? The biggest one would be Zhang et al, and this team has actually published multiple pieces on this. They both published a conference talk proceeding that led to an article as well as another article that was their separate data, and both those articles come to the same conclusion. They actually would argue there’s a very, very, very, very minor bias. They it is not statistically significant in their research, they clarify, but there is a small bias against native speakers. That will make more sense if you really think about how these tools are made.

Christopher Ostro [00:17:18]:

Right? These tools are generally trained on pieces of literature, media art written by native English speakers. Right? They’re trained on journals. They’re trained on newspaper articles, Wikipedia, things like that. They’re gonna sound pretty idiomatically native English speaker, and it makes so, Jang et al, those two studies generally support that. They actually find that there is no bias against ELL. And where this leaves me is wanting just to see more research on this. Because if Liang’s hypothesis is true, that’s a huge red flag. That’s a blaring red light.

Christopher Ostro [00:17:47]:

We should not be using a detection at all. But it hasn’t really been corroborated by any other studies that have used bigger datasets or more comparable datasets. Right? I mean, for what it’s worth, Zheng et al one problem with Zheng et al is that because they were only looking at the ELL question, they actually use no adversarial prompting, no anything. But what they used was GRE essays, and they used GRE essays from a variety of students. And there, they found no bias. And I would argue that GRE essays are probably closer to student writing at the college level than something like an 8th grade paper or a TOEFL paper.

Bonni Stachowiak [00:18:20]:

Yeah. And some of our listeners may not know what TOEFL papers are either because of their discipline or where they’re located in the world. So just give us one more recap around what your your concern oh, no. It’s fine.

Christopher Ostro [00:18:31]:

The TOEFL is the test of English as a foreign language, and it’s the standardized test that students take, international students take when they are coming to America if they’re from a country that’s primary language isn’t English. And it’s usually just a test that you are required to get it. You know, most different universities might put the threshold of different places, but you have to get above a certain score to successfully transfer there as an international student. So, again, some of those could be some of those might be a pretty interesting dataset, but it I don’t know how comparable it is to just normal, quote, unquote, traditional essays at the college level.

Bonni Stachowiak [00:18:59]:

Yeah. And the I’m definitely gonna wanna ask you, and now what? Because we’re we’re we’re not gonna wanna leave people without getting some guidance from you, but I think an important aspect of the research that hasn’t come out of our conversation so far is what are students saying about their perceptions of AI detection tools?

Christopher Ostro [00:19:18]:

Yeah. We’ll come back to the so what after. But with students, I actually love it. I I I think it’s really important. One piece of advice I would give any listener here, talk to your students about this. One of the best things I did in my class was I have a policy about AI disclosure. I give them a form that they can fill out with their paper. It’s not long.

Christopher Ostro [00:19:32]:

It’s just, here here are the things I do with AI. That’s it. And if they tell me what they use with AI, the deal I make them is that I will never honor code them. I’m not I mean, at least not for that paper. Right? If they do something totally unrelated, they’re this is not like a blanket pardon. Right? But if they are using AI in this piece and I think they were a little too reliant on it or using too much of that output. If they were upfront with me about the use, I’m gonna assume that they’re trying to play, you know, be on my team, be open and transparent. It’s never gonna be an honor code thing.

Christopher Ostro [00:19:57]:

It might be an awkward, like, hey. Can you actually go rewrite this part of your paper? Whatever. But I would encourage faculty listening to this to do something like that with your students because you’ll be really surprised the uses that your students have for these things. I guarantee there are students in your class using these tools in ways you have not expected and things that you have never considered. This has always been the case. Right? Like, the younger generations are always using technology in ways we would not consider. But what my students are saying, it’s it’s all across the board. Most of my students are pretty upfront about using AI in one way or another.

Christopher Ostro [00:20:25]:

I I’d say about 3 quarters to 4 fifths of my students have given me an AI disclosure at some point.

Bonni Stachowiak [00:20:30]:

And just tell us a little bit about what that would look like. What so so tell us what you’re gonna see a typical AI disclosure form, what it might look like for you, and what kind of an assignment are they doing it in in in relation to?

Christopher Ostro [00:20:44]:

So I I give them the opportunity to use it for anything. They could do it on discussion posts if they wanted to. Most often, I’m seeing it in essays and in presentations, the the bigger assessments in my writing courses. Also, though, I’m seeing it in scripting. Right? I have students who will make videos for their classes, and they’ll regularly be like, here’s the script. I used AI in these ways. And so what I usually say is, if you’re giving this AI disclosure form, just tell me what you’re doing. So usually, it is literally a paragraph that will say, on my honor as a University of Colorado student, I have used AI or AI tools in the following ways in this paper.

Christopher Ostro [00:21:14]:

Then they give me a list. And then after that, I do ask them to give me a few sample prompts. They give me that. And then I have one last spot where it says, if I use Chat gpt, here are some links to the chats I used. And they will link me chats.

Bonni Stachowiak [00:21:26]:

And this so this this is a online form. And is it a checkbox of the ways that I might have used it, or is this an open ended text?

Christopher Ostro [00:21:33]:

It’s open ended. And, honestly, I’ve been I should make it a web form at this point. I’ve been pretty lazy, and I just have it as a Word doc that I can copy paste to the end of the paper. Oh, god. It’s being real old tool lazy there.

Bonni Stachowiak [00:21:42]:

But it no. It didn’t sound lazy

Christopher Ostro [00:21:43]:

at all.

Bonni Stachowiak [00:21:43]:

It actually, I would have a tendency to make something more complicated than it needed to be, and then there’s another context I have to go to. And having it right there in the paper sounds like a really smart way to do it. Anyway, sorry.

Christopher Ostro [00:21:53]:

It’s just at the end. It’s nice because it keeps it all together. So when I get to the end of the paper, I’ll read it and be like, wow. Parts that seemed a little AI, and then I’ll be able like, look at that and be like, oh, okay.

Bonni Stachowiak [00:22:01]:

The other

Christopher Ostro [00:22:01]:

thing that’s about that is I tell students, if they’re doing that, I really encourage them to give me a link to their Google Doc or to their Office online, you know, Office 365, whatever the online one is, partially because both those keep revision histories. And I think you can usually see the fingerprints of AI use pretty well. I mean, there are definitely tools that are made to mess with version history, but, also, in my experience to this point at least, those tools are still pretty obvious. Right? But if my students tell me I used AI to help me with my introduction and my conclusion because I was drawing a blank, and then I went in and I edited it to make the introduction fit my paper more, I can probably go into their Google revision history, see a place where an entire introduction pops into existence, and then see them editing it.

Bonni Stachowiak [00:22:40]:

Yeah.

Christopher Ostro [00:22:40]:

So if if their AI disclosure matches the things that I’m able to look at in the revision history, we are kosher. We are good to go. So what I’m ultimately trying to do here really is help my students develop a new writing process, right, a writing process that I don’t have. And that’s, I think, where a lot of faculty feel a lot of anxiety. It’s hard to be asked to teach a thing you don’t know yet. Right? But these tools, it would be very difficult for me to look at these tools and not realize that they are going to change the way that people just process ideas and organize thoughts and organize those thoughts into a paper. Right? Of course, they are. I have students who, know, maybe because of the pandemic or what, but have issues where they’re way more confident and comfortable talking to a chatbot before they wanna talk to a person.

Christopher Ostro [00:23:22]:

And if they wanna use Chatcheapie to help polish their ideas before they raise their hand, if that helps them feel more comfortable and safe to do that, I love that that tool exists. Right? But I kinda need to update I need to come up with policies that help them come up with a more updated writing process. And that’s hard. Right? And and I wonder, honestly, how it must have been for the faculty know, in the nineties when the Internet suddenly came out. Yeah. Right? This was a whole new thing. The writing process pre Internet looked totally different, and these tools totally changed it.

Bonni Stachowiak [00:23:51]:

You’ve touched on a couple of times the some of the consequences that we we really wanna have center of our minds and our hearts and any potential actions we might take. You talked about students potentially losing their visas. You talked about students we might inadvertently be discriminating against and some of the what could or could not be emerging from the literature there. This part about wanting to make a writing or really it’s a creation process more broadly, more visible, and those of us that are in varying disciplines as somebody who makes a podcast, as somebody who’s written, the the the we know and we trust that the labor that goes into the creation process is both challenging and worthwhile. And that this is all part of our attempt to have students be able to see that. And it’s very hard to have anyone ever see value in something being challenged in that way until they actually experience it. Yet every time I I know so many people are concerned. I’ll I’ll link to this in the in the show notes, but the author’s name isn’t coming to mind because I just saw it on LinkedIn this morning.

Bonni Stachowiak [00:25:05]:

But just the ways in which surveillance

Christopher Ostro [00:25:09]:

Yeah.

Bonni Stachowiak [00:25:09]:

Is seen as a panacea for, now I can see your labor, and it’s just another way that I get to not just me as a educator, but but society gets to the powers and principalities get to have another look into your life. I’m thinking George Orwell here. You gotta Yeah. Gotta think and so I’m hearing you. I don’t know you very well, Chris, but I already just have such a sense of you as a human being, and I imagine learning from you and you being my professor that that I would feel likely to trust you. And and so you seem like someone who wants to be trusted as an educator. Yeah. I mean, just the very fact that you’ve created this form, and, hey, if you wanna use AI, you know, disclose it to me and and you’re not gonna get reported to the which for people not familiar with honor codes, that’s a disciplinary thing in and of itself, but it’s still to me, I just you start to say it and I can feel my body tensing up of that it would not be as easy for some students and their experiences in an educational context to trust you even though you seem like a trustworthy person to me.

Christopher Ostro [00:26:15]:

A 100%. And I thought I would say that’s part of why I make the Google Doc and the Office 365 thing optional.

Bonni Stachowiak [00:26:20]:

Yeah. Right? I don’t want students

Christopher Ostro [00:26:22]:

to feel like they have to give me that level of insight. I do encourage students to use these things, though, because these tools still create false positives with some minor regularity.

Bonni Stachowiak [00:26:29]:

Mhmm.

Christopher Ostro [00:26:29]:

And I do think having a Google doc link that if push came to shove, if you were actually formally accused, you could point to would be helpful. Yeah. It’s it’s a bummer to say, but I I think you’re right. I think the future of work is going to have some amount of that surveillance built in. All the tools that we have developed in the past 15 years for work do that tracking. You know? And I’m not generally in love with that, but it is the reality of it. I also wanna say because I I feel like I get this pushback with some faculty often that, like, I am not a cop teacher. I am not someone who likes catching cheaters.

Christopher Ostro [00:26:59]:

I’m not someone who wants that to be a big part of my job. Honestly, it’s the least fun part of teaching, but it’s also it is still a part of the job. I used to teach high school as well. Right? And it’s a similar thing there. You have to there are a few students who are gonna cheat just because they’re young, they made some bad time management mistakes, and, ultimately, they do need some minor consequences to teach them that lesson that, like, in the future, I gotta set aside more time for stuff. Right? And college students are are not really that different. I don’t think that any student I’ve ever caught cheating is, like, a villain. I think it’s just an 18 or 19 year old who went to a CU Buffs game rather than doing their paper that night.

Christopher Ostro [00:27:33]:

And I never wanna ruin that student’s career, but, also, it is important that there be some amount of that accountability because I think that in and of itself is a lesson. Here, they do it, and it’s a small slap on the wrist. The honor code process at CU, the first violation is literally just having to go to a class to make sure you know what the rules are. Right? It doesn’t even if they never get reported again, it doesn’t even show up anywhere on their history or their transcript. But I think that’s important because if they made that same mistake in a job, they are fired. Right? I’m thinking actually very simply with a student I had. You know, I teach a class on comics and graphic novels.

Bonni Stachowiak [00:28:04]:

Mhmm.

Christopher Ostro [00:28:04]:

And I had a student who used AI to help him do the outline for this whole presentation he gave, and it hallucinated a bunch of comics that didn’t exist. And when he and I talked about it, I ended up honor coding him. He thought it was a huge overreaction, I think was his wording. And part of what I tried to explain to him was like, if you did this as a job, you would not have a career. Right? Because he wants to go into marketing. I’m like, if you are if you worked in a marketing firm and you advertise a product for a company, a client that didn’t exist, you would be in trouble. Right? I don’t know how else to tell you this. That’s kinda what you did here, and I’m trying to give you the lowest stakes.

Christopher Ostro [00:28:35]:

For what it’s worth, I gave that student the also the opportunity to rewrite their paper because what I want more than anything is for them to succeed. I wanted him to do the work and to use AI to help do that work and figure out a way to do it that still produced high quality work and saved him time. Right? And that’s, I think, where the kind of goal is here. I’m I’m a big believer in a lot of, like, Maha Bali’s writing about AI that these things, we can really teach students new ways of using these things that are productive and enlightening and exciting. We can use these tools to have students reflect on who they are. And, like, what is this if AI is some form of mirror, what does this tell us about us and society and all that? There’s all these really cool reflective things we can do with AI, but I can only do that if I’m actually sure students are doing at least some parts of the work. And I think that’s the place where AI detection has some level of a spot.

Bonni Stachowiak [00:29:20]:

Thank you so much for that example. It’s really helping me and and, also, I think just helping listeners get a better sense of you. I was instantly having this flashback to probably something like 20 years ago where, some person, new in their career, didn’t realize that you couldn’t just take the grateful dead and put them on a t shirt and sell them. But somehow that got past whatever, you know, executives may have been responsible for, you know, double checking the work, that kind of thing, and it resulted in a lawsuit, etcetera, etcetera. And and just the way that you phrase that, I think, I’m hearing out of your example, the idea of authentic assessment that you really are the the care and time and effort and communication that you’re putting into it is designed around a set of values that are consistent with that. And and and, I mean, because because the without that care, without the concern, then why would you even need probably AI detection in the 1st place or in any intermediary? So yeah.

Christopher Ostro [00:30:22]:

Let’s be very clear. There are faculty that I work with in the business school who do not care about AI detection because they do not care about how their students use the work. Right? There are faculty I have seen where 100%, they view the development of these AI tools as just an excuse for grade deflation. What used to be a b is now a d, and you have these tools that do all this work for you. I’m expecting that your work look incredible. And, you know, I had a meeting with one of these business professors, and we had this long discussion about what are your learning outcomes, and it makes total sense in his course. Right? He is teaching a course that is entirely about just teaching students to make the best product possible in this marketing context. And if you’re using AI tools or not, they do not care.

Christopher Ostro [00:31:00]:

They just care about the quality of that output, and AI just allowed them to deflate that great average in a way. And that’s I I really do think that different faculty, different departments, different divisions should be thinking about the learning outcome goals in that way. I don’t think that professor is wrong. It makes sense to do it that way. The unfortunate thing and, you know, this is going back to the metaphor Maha Bali often uses about the what type of role you know, if the university’s goal is to teach someone how to run a bakery, what kind of course are you teaching? Are you teaching the baking course, the bookkeeping course, the property value, you know, how how to find a good location course? Right? Running a bakery takes a bunch of different things. The unfortunate reality is that it being the program for writing and rhetoric, my course teaches the baking part. Right? Like, I am specifically teaching process, so I have to get all involved. Right? A 100%.

Christopher Ostro [00:31:44]:

When I used to teach classics, I do think I would have a different approach. I fully believe that, but I I don’t. I teach I teach writing and rhetoric courses, and what I need to do is make sure my students are developing some type of functional writing process and creation process for later in their life. Yeah. And I think that’s the way where I find so going back to the so what that we talked about before with AI detection, to me and I guess I should say one last thing actually too. One last little piece of research I wanna highlight. So, yeah, one other thing I wanna draw people’s attention to that gives me a lot of hope is this Kaggle competition that happened in 2023 into 2024. And, basically, Kaggle is a an online website.

Christopher Ostro [00:32:20]:

It’s all websites are online, but, still, it’s a place online where people do competitions and put forward big money prizes and people code to compete to get to those prizes. And there was a huge one on AI detection. And, ultimately, what ended up coming out of this was a number of teams were able to develop totally independent models that got above 97% accuracy. And, again, these were just random teams of, you know, people putting together things that they were just curious if it would work. What that tells me, though, is that this is going to be probably a continual arms race. I think we are going to keep seeing AI tools come out because there are a ton of AI tools that are explicitly aimed at cheating. Right? I don’t know if you’ve seen Ithor. It’s author, but AI Thor.

Christopher Ostro [00:33:00]:

But, like, its entire thing is writing college level papers that are undetectable. That is literally what it’s marketed as. Right? There are tons of these tools. They’re gonna keep coming out, and those are the tools I don’t really wanna see my students using. And I’m hoping that by shuttling them more towards these productive tools, I think, are maybe more above board, that can be a healthy thing.

Bonni Stachowiak [00:33:17]:

Mhmm.

Christopher Ostro [00:33:17]:

But in terms of the so what, we asked, you know, at the very beginning, what are things faculty should take away from this lit review? What I personally have taken away from it is that I think AI detection has a place, but its place is limited. I don’t think it should ever be the sole reason a student is getting honor coded, 1st and foremost. If you are any sort of, I think this student cheated. This AI detection report makes me think that I’m gonna report them. I think that it’s too short of a process. I know people would want it to be that short, but it’s not. But on the other hand, I do think they are reliable specifically for catching what I would call lazy use of these tools. If I have if I have a report from a student that’s over 60%, it’s probably because that student did, in fact, just copy paste output from big parts of their paper.

Christopher Ostro [00:33:57]:

And that is something I do wanna intervene on, not because I’m some weird cop teacher, but because that’s a bad process they’re developing. That is a writing process that is maladapted and will hurt them later. Like, that is not an effective way. If ultimately, the thing I love to tell my students is if all you’re doing with these tools is taking the output and submitting as your own work, you don’t have a job. Right? That’s something every company can do and they can automate. Right? The thing that’s gonna give you a career, the thing that’s gonna help you flourish professionally and personally, the thing that’s gonna actually allow you to do something is putting yourself into that work, and you need to learn how to actually leverage your skills, your knowledge, your expertise, your background, whatever, into that AI output to make it better. Because if you can’t, it can just be automated by anyone.

Bonni Stachowiak [00:34:40]:

Before we get to the recommendation segment, just tell us a little bit about what that conversation looks like. Like. How do you structure it? What any tips that you have about the that, difficult conversation?

Christopher Ostro [00:34:50]:

Yeah. You’re totally right. Right? It is a difficult conversation. Sorry. I should say, my process now is I run things through ad detection. For what it’s worth, if you were using GPT 0 or anything other than Turnitin, you should 100% be removing identified student data from that because we do not know what happens to that data, and technically speaking, that could be a FERPA violation. So if anyone is hearing this who uses GPT 0, remove student info from that. But I run it through there.

Christopher Ostro [00:35:13]:

Then if I get a high score, I reach out to that student, and I have a whole conversation with them. You if they have an AI disclosure, honor coding is off the table. If they don’t, it’s in play, and we have a discussion where it’s me just trying to meet with them and talk about their paper. How did you write this? I am suspicious of these parts. Can you tell me about this? How did you get these ideas together? It also depends what I saw on the paper. If I saw a bunch of hallucinations, there’s really not another excuse for that. Right? You know, if you have hallucinations in your paper, it it means you used AI, which in and of itself is not a sin, but it means you used it poorly. Right? It means you didn’t edit it.

Christopher Ostro [00:35:43]:

It means you didn’t even reread your work. You didn’t fact check the output. That’s a problem. That is, I think, again, another maladaptive. That’s not a good use of those tools. So those conversations, I generally try to I think it’s really important, in fact, to start from a place of questioning and caring. If you come into it guns blazing like, You cheated and I know you cheated, you’re the bad cop and there’s no good cop in the room. And that’s not good.

Christopher Ostro [00:36:02]:

Right? That’s just going to scare this poor student. And I generally think it’s pretty easy to get students rattled and, you know, admitting to stuff they didn’t even do because they’re just terrified of authority. I don’t want that. I’m not interested in that. What I wanna know is, legitimately, how did you write this piece that this is what I got and these are the flags I’m getting? Because very often, I’ll be honest, the number one most common conversation I have is the student used AI tools and just either forgot to use, an AI disclosure or hoped I wouldn’t notice. So what usually happens is in that first paper, I have a bunch of these little conversations. And then by the second paper, most of my students are giving me an AI disclosure, and we’re kind of past that phase. And, yeah, I I find that asking students a lot of questions.

Christopher Ostro [00:36:44]:

How did you write this? Why did you do it this way? Oh, this is interesting. I I this wording is weird. Can you tell me why would you phrase this this way? Right? Pretty often, my experience is the kind of students who are, you know, using these tools unethically, they’re doing it because they’re rushing. They probably didn’t reread the paper, and they’re not, like, seasoned liars. Right? They’re not they’re not lifelong criminals. Most students, in my experience, are just very quickly like, yeah. Look. I I’m gonna save you some time.

Christopher Ostro [00:37:07]:

I over relied on this tool here. Can what can we do? Most students get it, and the few that haven’t are actually some of the more interesting ones. Right? Because the few times a student doesn’t do that, we end up having a whole interesting conversation where they’re showing me exactly how they wrote these things. So I’m learning a bunch about their writing process. I’m learning a bunch about how AI might be miscategorizing things. That’s valuable to me too. But you should one 100% be having those conversations. They’re difficult, but also it shows a certain amount of caring to your students.

Christopher Ostro [00:37:34]:

I should say that I’ve had 3 of those students later reach back out and ask for a letter of recommendation. Right? Because ultimately, we had this really heated at first conversation that led to this place where I think we were really understanding each other’s workflows and priorities more. And then by the end of it, ultimately, I think I’m viewed as more helpful than I hope. Fingers crossed. This is what I tell myself to sleep at night, but it was viewed as more helpful than disruptive.

Bonni Stachowiak [00:37:57]:

And tell me a little bit about logistically how you get that meeting to happen. I I I don’t wanna answer the question for you, but I’m thinking of things like you do somehow to get the student to recognize that there is something amiss here and that they need to reach out to you? So what’s logistically how you set that up?

Christopher Ostro [00:38:18]:

That’s an awesome question. Yeah. So I should say I teach both in person and online entirely asynchronous courses, and so it really matters on the course. Right? Either way, it starts the same way. I shoot the student email. Usually, a very friendly, albeit a bit vague email. Hey. I was reading through your paper, and I just had a few questions on parts of how it was written.

Christopher Ostro [00:38:36]:

If you could send me that link to that Google Doc if you want, but, also, I’d love to schedule a meeting. Very often, they’ll send me the link to the Google Doc. I’d say half the time, that just solves whatever my question was. And if that doesn’t, okay. Let’s get that meeting on the book for Monday. If it’s an in person class, I might just even offer, like, hey. Can you guys just hang after class for 5 minutes? I don’t think it has to be a long conversation. My experience is it rarely is.

Christopher Ostro [00:38:56]:

We talk after class. If it’s an in person class, easy peasy. I actually just do my office hours after class anyway, so it’s easy. With the online classes, it’s a little harder because my experience with online courses is very often the students who are using these tools recklessly enough to get in trouble for them are students who are having other stuff in their life kind of cascade. Right? But I I have a lot of sympathy for those students. And usually what ends up happening is I email them a few times, they ghost the email, so then I just refuse to grade the assignment till they meet with me. I will literally put a note in it that says, I need to see you at office hours or schedule a 1 on 1 meeting. I will regrade this then.

Christopher Ostro [00:39:32]:

And that usually, eventually, after a week or so, they see that. They’re like, okay. I’m gonna reach out. Yeah. It’s annoying, but it is also one of the least fun parts of teaching, but part of teaching.

Bonni Stachowiak [00:39:43]:

And you used the word regrade. So I am assuming that there’s a 0 sitting there to get their attention.

Christopher Ostro [00:39:51]:

Depends on it depends on how long it’s been waiting for what it’s worth. If it’s just been, like, a few days or a week, I’m not putting a 0 in there right away. If it’s been more than a week that they’ve ghosted my email, I’ll put that 0 in there because, ultimately, it takes me less than a second to change it, but it does send that message to a student that, okay. I can’t just ignore this forever. Right? And that’s usually what happens. I should also say, I don’t think there’s a single time I have reported a student to the honor code and not eventually given them a chance to redo the work for at least most credit. Yeah. Right? There are there’s what the only reason I’m even, like, kind of waffling there is there’s actually just one time it happened where I would just tell a student, I will not let you redo this, because it was the 3rd time in a row that I had caught them, you know, lying about their AI use.

Christopher Ostro [00:40:32]:

Right? And I I didn’t think it sent a good message to be like, no. You can keep lying. That’s fine. But generally speaking, over the last year, I have honor coded 15 to 20 students, and that’s way more than normal. Right? I mean, that’s more than the entire 5 years before that combined. But, also, a 100 per all literally one student has insisted I got it wrong, and he was a student whose paper was full of hallucinations. Every other student has ended up owning up to it, and we did a regrade where it ultimately didn’t really hurt their grade. I don’t want this to be a punitive thing.

Christopher Ostro [00:41:03]:

I don’t, again, I don’t wanna view myself as, like, authoritarian cop teacher. But I also think that you can brandish that a little bit to get students to actually put more effort and work in to for them to realize, like, oh, no. I’m gonna get this 0 if I don’t actually respond to this teacher’s email. And, like, maybe I need to be more thoughtful on how I’m using these tools. It’s like, that’s what I want. Like, that’s ultimately the goal. And if authority can be sort of vaguely alluded to in a way that gets them to just do the work, that is ideal and be communicative and transparent with me. More than anything, that’s what I want.

Christopher Ostro [00:41:33]:

Right? The moment that we’re on the same page talking about how they’re using these tools, not that they’re using them. I don’t care that they’re using them, but how they’re using them. I’m at least being given an opportunity to do something productive with that student again.

Bonni Stachowiak [00:41:44]:

This, I could keep going.

Christopher Ostro [00:41:46]:

But No. That’s alright.

Bonni Stachowiak [00:41:47]:

I have to I have to behave myself and move us to the recommendations. This is the time in the show where we each get to share our recommendations. And for the first time that I can remember, although we’ve had a lot of podcast episodes, but the first time I can remember, I’m gonna recommend not to do something. And then I do have another more positively oriented recommendation as usual. I am going to recommend, don’t buy one of those tiny sticker printers even though they look cute. We were in a situation over the holidays where our son had asked for something rather large, and our daughter had not been at all specific about what she wanted, so we were trying to make it even and fair. And so I think consumerism just reigned the day. And on Amazon, I had seen these adorable little sticker printer thingies, and it was like a rainbow color and so shiny.

Bonni Stachowiak [00:42:42]:

And I was like, this is just perfect, and so I bought one. It turned out that not only was do they have different models, some of them actually print stickers and some of them just print slips of paper, which I didn’t even realize. So I had bought her the wrong thing. But even then, she had just remember when we’re she she was so so grateful about her gifts, and please don’t, like, let me characterize her in a way that’s not at all accurate. Such a kind, sweet girl that just was happy to spend time with her family. But as we were discussing this and my husband’s having her download the app that went with it, he was he was trying to, like, say it behind her, like, totally shady app, and you had to be over 18 for the app. And it’s like it’s clearly geared toward children, not a good thing. And then she was just like, I just don’t even know why I would use this.

Bonni Stachowiak [00:43:28]:

And as soon as she said that, I was like, yeah, because you wouldn’t because, you know, I was just trying to make it even and fair. So anyone who’s worried that she didn’t get justice, she certainly got justice and got to have the money and, you know, other other things that she’s interested in, but don’t buy one of those tiny sticker printers. There’s a lot of other ways you could be creative and a lot of other cute tech gadgets. That’s one I’m gonna definitely recommend against. And then I wanna recommend a television show. My friend recommended this. It’s called a man on the inside. It’s on Netflix, and it stars Ted Danson.

Bonni Stachowiak [00:44:02]:

And I’m gonna read the description from Internet movie database. Charles, a retired man by the way, I’m not giving anything away that you wouldn’t see in the 32nd preview of it. Charles, a retired man, gets a new lease on life when he answers an ad from a private investigator and becomes a mole in a secret investigation in a nursing home. I loved the show so much. I pinched it over, like, 2 days. It’s just a single single season. I think it was something around 8 8 episodes, so it doesn’t 8 30 minute episodes doesn’t take that long to go through, and it’s just wonderful about aging. There’s some themes around Alzheimer’s.

Bonni Stachowiak [00:44:40]:

People know that have been listening for a while know that that’s been a source of pain and grief in our family and continues to be so today, and it’s just a wonderful heartwarming look at aging, at family, at friendship, at the meaning of life. It’s a beautiful show. I really, really enjoyed it. And I am gonna pass it over to Chris for whatever you’d like to recommend.

Christopher Ostro [00:45:06]:

Honestly, that’s a perfect transition, but I do have to ask her. How is Ted Danson still making so much good? Like, he is constantly producing, like, banger TV shows. I love the Good Place. He’s one of the best parts of it. He is such a good, like, straight man comedic character. Yeah. You know? Like, anyway but so the reason it’s a good transition is I’m actually gonna recommend a novel. And this is a graphic novel I reread every, I don’t know, every 5 ish years because I don’t know.

Christopher Ostro [00:45:31]:

I I get something new out of it every time, and it’s Day Tripper by Fabio Moon and Gabriel Ba. I don’t know if you’ve ever heard of this, but it’s these 2 Brazilian twin brothers are the author and one of them the other one is the actual artist. And the entire gimmick of it, it’s 10 issues. At the end of each issue, the main character dies, and it is just about how would they have been remembered at this point if they died at this point in their life. Right? Things they accomplished, things they regretted, things they loved, things they hated, how they are remembered by the people in their lives. And then the next issue is like, okay, but if they didn’t die then, okay, we go forward more in their life, and then, okay, if they died in their thirties now. Right? And so each issue is this kind of pensive look at aging and mortality and the the legacies we leave, the lives we affect, and how that changes. You know, one of the I find it to be a pretty you know, the first time I read it, I was a young, young, angry 18 year old and remember reading it and seeing so much myself in one of the issues.

Christopher Ostro [00:46:24]:

And now I’m, I think, hopefully still young, but, less young 37 year old. And I I read it and I get a different issue. A different issue really appeals to me in a real way. And the last time I read it, I was 29. It was a similar thing of, like, I read it and was like, that’s interesting. Right? Like, you it grows with you in this way because of the actual story it’s telling. I’m trying not to give spoilers here, but I think it ends up being, like, a pretty beautiful meditative look on aging, both youth, like, the youth lost, but also the best and worst things that can happen in youth, and also how we reflect from those things. How, at the time, we don’t maybe even think about how cool or exciting or bad or detrimental this is.

Christopher Ostro [00:47:03]:

And then 10 years later, we’re looking at those consequences, and we’re really reflecting on, thank god this thing happened that at the time I thought was a disaster or whatever. And I I really appreciate the book for that. It’s it’s instilled a certain amount of humility in my own life to me.

Bonni Stachowiak [00:47:14]:

Oh, you’ve gotten me so curious about it and so curious about It’s a

Christopher Ostro [00:47:18]:

great read. Only 10 issues. Highly recommended.

Bonni Stachowiak [00:47:19]:

So curious about so many things. I’m so grateful to be connected to you, Chris, and to have you come on the show to share about this big picture look at AI detection tools and what a joy it is to know you.

Christopher Ostro [00:47:30]:

Yeah. It was awesome being on, Bonni. Thank you so much for having me.

Bonni Stachowiak [00:47:35]:

Thanks once again to Christopher Ostrow for joining me for today’s episode of Teaching in Higher Ed. Today’s episode was produced by me, Bonni Stachowiak. It was edited by the ever talented Andrew Kroeger. Podcast production support was provided by the amazing Sierra Priest. This is your moment if you’ve yet to sign up for the weekly show notes that will come into your inbox as well as some other resources that don’t show up in those notes. Head over to teachinginhighered.com/subscribe, and you’ll start to receive those. And this one’s gonna be really good. We’ve got lots of great links coming from Chris.

Bonni Stachowiak [00:48:15]:

So thanks so much for listening, and I’ll see you next time on Teaching in Higher Ed.

Teaching in Higher Ed transcripts are created using a combination of an automated transcription service and human beings. This text likely will not represent the precise, word-for-word conversation that was had. The accuracy of the transcripts will vary. The authoritative record of the Teaching in Higher Ed podcasts is contained in the audio file.