EPISODE 617: How Today’s Agentic AI Changes What and How We Teach

Bonni Stachowiak [00:00:00]:

Today, on episode number 617 of the Teaching in Higher Ed podcast, how today’s agentic AI changes what and how we teach, with Teddy Svoronos. 

Bonni Stachowiak [00:00:15]:

Production Credit: Produced by Innovative Learning, Maximizing Human Potential. 

Bonni Stachowiak [00:00:24]:

Welcome to this episode of Teaching in Higher Ed, I’m Bonni Stachowiak, and this is the space where we explore the art and science of being more effective at facilitating learning. We also share ways to improve our productivity approaches, so we can have more peace in our lives and be even more present for our students. There’s a Charles Babbage quote that today’s guest, Teddy Svoronos uses to think about AI literacy. And I’m reading from one of Teddy’s blog posts now, and reading as he quotes Charles Babbage here about his mechanical computer quote: On two occasions I have been asked, Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out? I am not able rightly, to apprehend the kind of confusion of ideas that could provoke such a question.

Bonni Stachowiak [00:01:34]:

In 2026, that confusion is everywhere, and Teddy is grappling with that confusion and doing wonderful writing and collaboration with his colleagues. Teddy Svoronos is a senior lecturer at the Harvard Kennedy School, where he teaches statistics and public policy. He’s a returning guest today and has also been a guest on other podcasts, including a beloved podcast of mine, the Mac Power Users. And he’s been writing about what does it mean to work with agentic AI? What is that changing in terms of within his disciplines, that he and his colleagues teach in? And just broadly speaking, within our own use or not of these tools. Some of these changes that he looks at are not just a chat that’s asking a question, doing a search, but now pursuing tasks that we assign it that, yes, might include searching the web, but also might include other tasks that we ask them to do. And he has a concept for the risk that comes from leaning on them, that you’ll hear him talk about in our conversation, and that is cognitive debt. All right, let’s get straight to it. This is a great conversation, and I want to welcome to the show Teddy Svoronos.

Teddy Svoronos [00:02:57]:

Thanks so much, Bonni. It’s great to be here.

Bonni Stachowiak [00:02:59]:

I fall for those posts all the time that tell me that I’m doing things wrong. And I have to also tell you, Teddy, Mike Caulfield, who’s been on the show many times, has also told us that we’re either doing things wrong, or we’re thinking about things wrong. And specifically when it comes to artificial intelligence, and in his most recent visit to Teaching in Higher Ed, he talked about how the way that we’re conceiving of AI, you’re thinking about it with its old capabilities, or lack of capabilities. He made this huge distinction about when search got added, how that changed things. And he, he’s said many times, stop thinking about these things as an answer machine. Think about it instead as a discourse map. I was so excited to hear from you, and to learn about some of the things you’ve been wrestling with, and experimenting with and thinking about. And today you’re really going to help us do that, specifically around two things.

Bonni Stachowiak [00:03:55]:

One is agents, and also what we’re thinking about, AI literacy. But let’s start with agents, Teddy, what do we need to know about them? Particularly given that this term gets thrown around to mean 12,000 different things. So what’s happening?

Teddy Svoronos [00:04:12]:

And honestly, in your and Mike defense, the reason we’re always doing it wrong is because it keeps changing so quickly that I think we need to keep updating not just like what they’re capable of, but even our mental models of how they work. So, you know, I’m constantly realizing that I’m understanding this wrong, or using it wrong in some way. To the point about agents, I totally agree. I feel like, for some reason, what an agent means to a lot of people is like, once this thing can book a flight for me, then it’s an agent. And I don’t know why, maybe it’s the travel agent thing, I don’t know. But that’s not, I think, the best way to think about what agentic AI is. So agentic AI actually relates a lot to what Mike was talking about, I think, of agent,

Teddy Svoronos [00:04:52]:

I use the definition that this sort of software developer and, you know, big voice in the space Simon Willison uses, which is an AI agent is an LLM that runs tools in a loop to achieve a goal, and each of those parts is really important. And so, web search is a tool that these LLMs can use, right? They predict words, but then they can decide that they should go out and do a web search and get results. They can also do things like they could run code if you gave them a spreadsheet. They can go to a browser and like click around as you in your Chrome browser, or whatever. There are all these new tools and capabilities that these models have. And that’s a really important part of this. But the next part that I think is really important is the loop, which is that, these models can now iterate in ways that are really new and interesting.

Teddy Svoronos [00:05:38]:

So to get back also to the idea of web search, one functionality that all these tools have now is called deep research, where you ask for a topic and it writes a big report for you. And that came out a year, year and a half ago. And in my mind, that’s like the first real agentic thing because you give it a task and it runs a tool. The tool in this case is a web search, but then, based on the results of its web search, it runs another web search. And then based on what it learns from that, it runs a new web search. And it keeps iterating in this way until it decides it has done enough web searches to get the information it needs to write your report. And that process of having a task, write a report, use a tool, web search, and do it over and over again until you feel like you’ve gotten the full sort of spectrum of things, that, I think, is what an agent really is. And so deep research, I think, is a very sort of simple version of an agent, because the task is write a report and the tool is just web search.

Teddy Svoronos [00:06:34]:

A thing that has happened in the past couple of months with these newer models. So Claude Opus 4.5 and 4.6, ChatGPT released 5.2 and Codex, these things now have a bigger spectrum of things that they can be agents about. So the big thing that people talk about now is Claude code, where the goal is to literally write functional software. The tool is write and execute code, and the loop is really important here. So it’s not just writing code, it’s running it, and it’s seeing where it goes wrong, and it’s correcting it, and then it’s changing it, and then it’s writing a test for itself to see if the code passes the test. And if it doesn’t, then it keeps iterating. That process, that’s what I feel like is really new about agentic AI in the past couple of months, that it can keep iterating in this way, to make things better and better.

Teddy Svoronos [00:07:26]:

Now, finally, I think things are starting to get to even bigger and more abstract tasks. So Claude now has Cowork, and the task could be: Help me plan a conference. And it keeps track of all the attendees, it writes documents for you and keeps them in folders. It reads those documents when it’s necessary to make sure it can update stuff. It can now be applied to a much broader array of tasks. And part of how it does that is by doing that iteration.

Bonni Stachowiak [00:07:53]:

One of the things, and by the way, I apologize for I keep going back to Mike, but I just feel like this is such a great addition to that conversation. But he talked about then, that we also need to stop thinking about it. And especially I think he mentioned more people newer to artificial intelligence, which could be in those cases, our students. Of course, not in all cases. But he talked about how people needed to stop thinking about, okay, it’s prompt engineering. You just write the perfect prompt, and then you’re done. That he wanted us to think about it. I’m not sure if he used the word conversation, but that’s how I remember it.

Bonni Stachowiak [00:08:26]:

He wrote this custom GPT to assist us with fact-checking. And one of the custom things that you can type in as a person engaging with this custom GPT is called another round. And he found it helpful in his fact-checking efforts. And I have also subsequently found it in mine. Is that even just that another round, then it went and created that. If I’m understanding what you just said correctly, that might not be as important, if I’m thinking about growing my own literacies, because thinking of it as needing that another round, if it’s operating as a loop, is that still as important, Teddy, for me to be thinking about it as a conversation and that interaction that I’m having, or are there other things that are emerging? Or does it kind of depend on what I’m trying to do? I don’t know if this is making any sense.

Teddy Svoronos [00:09:19]:

It really is, and it’s a really good question. I think a lot of what we think of as AI literacy, at least from my perspective, is changing a lot. So I teach a course with Dan Levy and Sharad Goel at the Kennedy School on Generative AI. And it’s like when it comes to actually how to use these tools, we focus on prompting a lot. And I use this framework of your task, and your instructions, and your context. And that’s like the way that you write a really good prompt. I think the idea of prompting as effectively as possible is less important than it was. And having this other round thing is now becoming,

Teddy Svoronos [00:09:53]:

I think it’s still extremely important, but it’s becoming, kind of inherent part of the process, by which I mean a lot of the times now what these models are doing are based on your input. They are basically creating their own plan or prompt, based on your prompt. Sometimes these models will write a prompt for a separate agent to go do something while it works on what it’s working on. And then when the other agent is done, it comes back to it. So actually, I don’t think I’m spending as much time as I used to on the prompts itself. I’m thinking more of the, kind of the infrastructure of what’s in place, right? So like what is the agent capable of, how do I want it to think about things long term? And how do I give it the best feedback to make sure it’s accomplishing what I want? 

Teddy Svoronos [00:10:40]:

So in one sense, the conversational part I still think is quite important, and in another sense it’s kind of taking on a different tone than maybe it used to.

Bonni Stachowiak [00:10:44]:

And you’re helping bring up, I think it’ll be helpful maybe if we come up with an example. So do you have one where you’ve done some experimenting or should we do this one on planning a conference? What do you think might be more helpful for people?

Teddy Svoronos [00:10:57]:

Sure, so let me think of an example. So I’m actually planning a convening right now of a bunch of faculty to talk about the future of quantitative methods education, now that these agents can do what they do. And I have a window of CLAUDE code open, and I give it the names of the people in my conference, and I have it help me draft an email, like more sort of standard stuff that you have as an LLM. And then in the process, I’m basically helping it write a document to itself. The document is called Claude.md, it’s a markdown file. And it’s just high level information about what I’m trying to do, what steps I like to take, what issues need to happen, and every time I go back to it, it references those instructions, includes the new information that it has for me, and then it iterates on that. I can then have it do what are called CLAUDE skills, where there’s a thing that I have it do pretty often, and so I want it to be able to do that quickly.

Teddy Svoronos [00:11:57]:

So I said, hey, can you make a skill where I give you all the new information about responses, and you use it to update this tracker of who’s responded and why and how and all that. And it said sure, and it writes, I can look at it, but it writes a pretty elaborate prompt for what steps it should take when I ask it to update the tracker, and that now exists within that project as well. And so all I have to do is say update the tracker. It references the skill file that it itself wrote, takes in my new information, takes the original document of who has responded versus not, updates it with new information, and then gives me suggestions on maybe people I should reach out to who haven’t responded yet and things like that. So the agent is maintaining multiple files, some of which have context it needs, and some of which is just for me to make sense of what’s happening. And that’s the kind of iteration I’m doing with it. And so what I have now isn’t like a chat log.

Teddy Svoronos [00:12:50]:

It’s a folder filled with documents, text files, images, and things as I put together rosters and stuff like that. And it just has that and has this sort of, in its working knowledge as I interact with it.

Bonni Stachowiak [00:13:02]:

I’ve got this friend who has been encouraging me to look more retrospectively at things I’ve been doing. I’ve got some crises that have been, have been going on, and I tend to be a person who lives in the future. And that’s really where I have my greatest strength. What are the possibilities? What could be out there? And then what might it take to get there? And she’s been really encouraging me, so I could be more gentle with myself. You know, maybe you could put together, like a captain’s log of the things that you have done in this last month as you’ve been navigating through some of these crises, and that really resonated with me. 

Bonni Stachowiak [00:13:37]:

And so I went in, I actually recently had switched over to Claude, and not exclusively, but is a fairly new tool for me as far as my tool set goes. And it’s hard for me to wrap my head around a little bit. And you had just mentioned this in terms of the, the files. Some of these files end up living on your computer. Although in my chat with Claude, it was saying, well, this little captain’s log, which was the friend’s suggestion, make, you know, look at this last month, yes, you’ve been navigating a ton of crises, but perhaps you could be a little gentle with yourself, and see with a little bit more clarity the things that you have been able to do. And so it was explaining… 

Bonni Stachowiak [00:14:20]:

Because I, I build things, and then if it was an exercise for students, okay, let’s work with AI and build a concept they have trouble understanding. Okay, can you help me build this little simulation, a little game or exercise? I like to build humor into those things. And then I’ve been uploading those files into GitHub. And so now I’m realizing, okay, what lives where? Where do my files live? What lives in GitHub? What stays in a tool like Claude? So that’s, to me, that’s a big part of my own fluencies, which feels like it’s also continually changing what lives where. And just to make it even more complicated, what the heck is going on with security Teddy? Because I don’t feel comfortable with, where did these people’s names and information live? So that was like about 20 questions. So I’m going to stop talking and let you pick the one that you want to answer and explore more.

Teddy Svoronos [00:15:10]:

Let’s see. So I think one thing that’s worth thinking about when you talk about GitHub, for example, I now keep a huge amount of things that I work on, both the code-related stuff and sort of like more day-to-day stuff in GitHub. And the reason is that it’s a very, very nice way of having version history of what was changed and what wasn’t in your notes, in your work or whatever. And so with my agent, at the end of the day, I don’t have to tell it everything that I did, it looks at my different GitHub repositories, it sees what activity I did in those things, and it says, here’s my understanding of what you worked on today. Is there anything else that you did that I can sort of give more color for? And that process of just kind of like creating, kind of like passively created data that this agent can use to kind of make sense of what I was doing, makes it so that I don’t have to be constantly giving it new and new context. The flip side of that though, I should say with respect to security is it’s a real issue. So I think right now these agentic tools in particular, to get the most out of it requires you to give up, I think, a fair amount of control. And so I don’t have Claude code touch any student data,

Teddy Svoronos [00:16:21]:

unless I’m doing anything that I use Harvard infrastructure to interact with. I am giving it more access to personal information about it. And Anthropic has a very, they’re pretty, I think, transparent and committed to the sort of privacy things that they have, and turning off model training and all of that. But it’s still true that I’m giving up information to these agents. And on one hand, it’s different from how things were before, where you would upload all your documents, and it would create this searchable database of it. The way these agents work is actually closer to how a human works, which is to say it just runs searches on your files in your computer, and based on what it surfaces, it chooses what documents to read. So it’s not like everything you’re doing is now being sent somewhere, but it’s sort of like picking and choosing based on what it finds in the same way that a human would. Which is an interesting wrinkle to the question of, I think, privacy and security.

Bonni Stachowiak [00:17:18]:

What are some of the different repositories that you keep in GitHub?

Teddy Svoronos [00:17:22]:

Sure, so because I’m on sabbatical now, which is really excellent, I have a notes database in an app called Obsidian. Obsidian is like a plain-text notes app. It’s similar to the Notes app that you might have on your phone, but it uses all plain text files. And so in there I have, you know, meetings that I run, any notes that I take, I have a daily note where I just add stuff that I’m working on. As I have ideas for blog posts, I do it. I have a separate repository for my blog, where I’ve been writing on my website and on my Substack, so it can sort of see what writing I’ve been doing for a more public audience.

Teddy Svoronos [00:17:58]:

Then there are my actual research projects, where I’m like, writing code and writing software that I never really thought I would be able to do or write. And maybe I’m not actually doing it, I’m just having these agents do it for me. But those are projects that I’m kind of pushing along, ongoing with every passing day. And each one of these has its own sort of little Claude project that I use within CLAUDE code. And I have sort of a more centralized one that has been given access to each of those repositories to find more commonalities as I sort of work with it.

Bonni Stachowiak [00:18:27]:

And you mentioned Obsidian, which has been recommended at least twice that I know of on the show, and I would do it a third time if I could. I try not to recommend the same thing twice. I’ve failed at that before, but my husband and I use it extensively. But they have their own syncing service. So this is a little bit helping clarify some things for me. You’re using GitHub in place of their sync, or is it in addition to their sync?

Teddy Svoronos [00:18:52]:

I’m using it in place of it so that I have a more visible version control that CLAUDE can use. This is kind of an interesting thing that I think that I’ve been spending a lot of time thinking about is like, there are some things that LLMs are much better at working with than others. Plaintext is one of them. GitHub is one of them, because these models are trained to work with GitHub for, like, coders and stuff like that. And so, I’m finding myself starting to prioritize tools that I know LLMs are good at working with, maybe even more than I’m prioritizing things that I’m good at working with, because these LLMs are now becoming kind of like this intermediary between me and the actual content. And so I’m optimizing in a different way than I used to.

Bonni Stachowiak [00:19:35]:

That’s fascinating. So come back to, you mentioned GitHub. You have some things that live in GitHub. What are some things that live locally on your computer, and how do you go through that decision making process of what makes sense to give access to and what doesn’t? Take us through an example, and if it relates to your event planning, that’s great. Or maybe a different example.

Teddy Svoronos [00:19:55]:

Yeah, so I mean the event planning is a good one because I’m only really giving my, the model access to publicly available information about my speakers and stuff like that. I might put snippets of an email that are related to it, or things like that. But I have to be careful about what I consider to be okay for a model to look at versus not. But it’s a very weird hard middle ground right now, which I don’t think we have really good best practices on. It’s a little bit of the wild west right now because in the past couple months, I would say before November, the only people who had LLMs working with local, working with your actual files were developers, who had a code base that they were having working with. And now the idea of an agent working with your actual files on your computer is now I think a pretty common thing that’s going to become more and more common. And so the question of what are the best practices and stuff now are kind of being developed I think as we go.

Bonni Stachowiak [00:20:54]:

You’ve talked a little bit about how prompting has evolved somewhat in your own use case. Tell us about then skills. Could you give some examples? You gave a few earlier, but just a couple other examples in terms of, what do we even mean by skills? And how if somebody was convening a group, what sorts of skills might come up as helpful in those endeavors?

Teddy Svoronos [00:21:14]:

Yeah, so it’s kind of interesting because like as we talk in parallel, the thing that I’m planning is all about what skills actually are for quantitative analysis now. And so I’m kind of balancing these two things, both in terms of what I’m trying to do as I plan this event, but also in terms of the actual substance of the event. But I think there’s actually a lot of really interesting parallels. So one that I think is really important, and seems really basic, but I think is going to become a really important part of working with these tools, well, is having a very understandable and well-documented trail of how you got to where you got. So we’re used to kind of just having back-and-forths with LLMs, and maybe we have the thread, and we can share the thread in some way, but with these agents now, in a given session, I’m working with an agent for two hours, and I have this incredibly long conversation. And so it’s really important, and I actually include this in the CLAUDE instructions that I give it to, whenever it does something, it just adds a line to a text file of what it did so I could trace back what actually happened.

Teddy Svoronos [00:22:20]:

And having that kind of documentation is, I think, really important, certainly in quantitative analyses, which is what I’ve been thinking a lot about. But even in terms like this, like why did I phrase it in a particular way to somebody? Or where did I get the idea that this person was free on this day and not that day? Just having a way to actually check that is quite important. As the tasks these things are working on become longer and more complex, having it be human, understandable is quite important. And an interesting thing about that is these tools are really good at making documents now, and even like decks, like slide decks. There’s one faculty member, Scott Cunningham, who’s been writing a lot about CLAUDE code, and he has Claude make essentially PDF PowerPoints for him, periodically through a session so that he can understand the steps that Claude took without having to read through it. So, Claude now has this instruction where it is told, at the end of a particular important sprint, make a deck that fully summarizes each step of what we did, so that he can just browse through it if he wants to make sense of it. And so thinking about how you want things to be documented, I think is one skill that’s quite important.

Bonni Stachowiak [00:23:26]:

I am curious because you mentioned earlier markdown files, and Markdown has been talked about on the show previously, but I know from talking to people off-air, they still say it confuses them or intimidates them. And I’m always like, no, oh my gosh, don’t let it do that to you. And I feel like plain text is just such an important literacy for us. So even in my own work, just as recently as this morning, I was starting to think, do I need to keep going with the Teaching in Higher Ed episodes? We do pay a podcast editor. We also pay a podcast production support person. So I was thinking, do I need to keep- have that person having a plain text version of Transcripts on the Teaching and Higher Ed website, in addition to having the PDF, because of course we know PDFs not as accessible as plain text.

Bonni Stachowiak [00:24:11]:

So from an accessibility standp, they’re also- it is extra work. It is being done mostly manually now, and I’m thinking, how are people even? Part of the problem is when you don’t understand how someone, whether it’s a student taking your class, or if you’re convening a group of scholars, if you don’t understand how people might use whatever it is you’re doing, it’s hard for you to then decide whether it makes sense. But so talk about this. And again, you’re going to have to talk for Scott. So I imagine that might be hard, but, like, why might a person choose to make a PDF of one’s PowerPoints in this instance, versus just give me a plain text outline of what I said? I’m curious if you have any thoughts about that.

Teddy Svoronos [00:24:49]:

Yeah, yeah. So in this case, what Scott is doing is he’s using plain text to generate the files. So he’s using LaTeX, which is a different kind of syntax to markdown, and it creates these nice presentations. So he’s using plaintext as well. And I would say in terms of how these agents and stuff currently work, the hierarchy really is plain text, like you say, PDFs, which at least have, for the most part, content that these things can read. And then way lower on that is any kind of Office document, PowerPoint, Excel sheet, because these more proprietary formats, so much of what the LM ends up working on is taking apart the package of what a Word document includes, sticking to the style syntax that Microsoft makes, and trying to approximate in various ways. Basically, there’s all this work that’s being put into making it a thing that we are now very used to, which is a Word document.

Teddy Svoronos [00:25:43]:

And so it’s very, very funny that the thing that we’re most used to, which is make a couple slides in PowerPoint or send me a Word document, is actually quite far, I think, from what these tools can work with more effectively, which is plain text files and, to a lesser extent, PDFs.

Bonni Stachowiak [00:25:58]:

My first job out of college was as a computer instructor. And this is why, I think this is one of the reasons why I’m able to think about these things, because it’s hard. We, as human beings, we like to sort things, we like to categorize things. And the example would always be, you know, somebody, for example, who would try to use, I don’t know, Microsoft Word as a spreadsheet. And it’s kind of like, well, you can calculate things in Microsoft Word, you’ve been able to do that for decades now, but is that really what you want for your intended purpose? And same thing with Excel, like, oh, I’m going to use Excel as a database. Well, you can, and certainly there’s some things that Excel, but there is a point at which you cross over to really needing a relational database in the truest sense. So it’s hard for people to navigate.

Bonni Stachowiak [00:26:40]:

And I feel that way about my own AI fluencies, is trying to figure out what, you know, what makes sense in which particular case. So I’m gonna come back to our example, of this conference planning, because it’s so much fun, and ask you a little bit about, how might you also then think about integrations? Because I’m realizing you just mentioned Office Document, and just the other day, when I logged in to Claude, which I’m still very new to, it said, oh, look, we now have this integration with Excel, and now we have it with PowerPoint. And I was thinking, I’m not even ready to try that yet because I’m still such a beginner in all of this. What examples, what kinds of things are you seeing in your work, or might we see in this event planning example that we’re using where, yeah, you might need to be integrating with these different kinds of tools? I’m not trying to make this a Claude only show, but just in general, what ways might we have artificial intelligence integrating with other tools in order to get these things accomplished?

Teddy Svoronos [00:27:40]:

Yeah, so this is the thing that I think is- I can’t tell how much this is a temporary thing versus a longer-term thing. And here’s what I mean. The current iteration of Agentic tools were initially developed to help developers make software. So they’re optimized toward plain text, they’re optimized toward things like GitHub. They’re optimized to do those things. And now we’re trying to figure out how can we make them work with Word Documents or Google Slides or things like that, right? We’re trying to shift in this way.

Teddy Svoronos [00:28:10]:

So it may well be that these companies, and I think they have a strong commercial incentive to do so, are working really hard to make that shift. And you gave Claude as an example, and I think ChatGPT has a similar one with Excel, where they’re making these sidebar applications that live in an app that is good at working with that file format. So Claude has a PowerPoint one. Claude and ChatGPT both have an Excel one. And those, I think, are they’re trying to train the models to work better with those kinds of things. And I think that’s really important because I think AI stuff is moving incredibly fast. I also think people’s path dependence is maybe even stronger than the progression of AI. And so I don’t think people are going to all switch to start using GitHub and stuff like that.

Teddy Svoronos [00:28:53]:

So I think that kind of adaptation is important. In the short term, it means that, first of all being used to particular integrations with AI. So like you said, Claude has PowerPoint in Excel. Claude also has particular skills for different types of document formats. So you can actually add a Word Doc skill where it has a whole bunch of instructions and scripts that it uses to make- to manipulate Word documents. The most clunky way to do this is to have it do this browser use tool. So one way that these AI tools work is they can literally open a Chrome browser, take screenshots of your browser window, and click the mouse in particular places to do stuff. And that’s what a lot of people think of when they think of agents.

Teddy Svoronos [00:29:37]:

But again, using the framework from the beginning, that’s- the tool is browser use, and the loop is whatever. Book a flight for me or something like that. You can also have these tools work in a Google Doc by just clicking around the Google Doc, and inserting text as it works, right? Now, Google is going to go very far out of its way to make Gemini really good at working with Google Docs because it’s all within its same ecosystem. But the sort of like, the thing of last resort, which a lot of folks are using now, is just having the AI literally use your computer the way a person would.

Bonni Stachowiak [00:30:09]:

I recommended this book on a recent show. I’m learning how to use Procreate and also learning about digital art. I’ve been having so much fun. But the reason I recommended was scratched a certain itch for me Teddy, because I liked that it was more project-oriented. Oh, you could print stickers, oh, you could make a print a tote bag. You could make a piece of artwork. And I really liked that. And one of the things that I keep coming back to about your work, you have inspired me so much in so many ways, is the idea of metacognition, our teaching, thinking about our teaching, and reflecting on it with regularity.

Bonni Stachowiak [00:30:43]:

And when you were on the show previously, you talked about that. And I would love to sort of close this part of the episode with you sharing what, what has changed since we spoke with how you collect those thoughts and reflect on them and make use of them, and to any level that you want to be specific about where stuff gets saved, what, and then how you revisit it? I’m just excited to sit back and take notes.

Teddy Svoronos [00:31:07]:

Yeah, totally. I mean, I think metacognition has been talked about a lot as the best way to learn. I think a lot of people, including myself, feel that it’s the only hope we have of people taking learning seriously, as tools are more able to do the learning for us, that the only way to really build it is to think very seriously about your own learning and what you’re getting from things and what you aren’t. And so I continue to do that, and I actually have Claude facilitate that in various ways. So I have these skills that I activate at the beginning and end of my day, where it looks at what I have on my calendar and what I was doing the previous day based on what I committed to GitHub, and it says, here’s where you’re at. Are you getting what you want to be getting out of today? And this is all prompted by me. And I give it a bunch of thoughts, comments, it helps me turn that into an operational plan, and at the end of the day it has me reflect on it.

Teddy Svoronos [00:32:00]:

I think that metacognition actually relates also to a second type of AI literacy skill that I think is really important, that I think is new to these AI agents. So we talked about the importance of documentation and so forth. I think there’s another one that relates to metacognition, which is in the developer community, there’s an idea called technical debt, which is that if you’re building a software tool or something, and you have to cut corners, you do so. And every time you do that, you’re basically adding a little bit of debt that eventually you’re going to have to pay back because you’re making all these sort of small- you’re giving up all these small things that you have to kind of get back to it, and that builds up, and eventually you have to address it. I think there’s an analogy with these tools that I’ve been thinking of as cognitive debt, which is that as you offload to them, there are things that they’ll do that you won’t quite understand, so they’ll push you beyond what you know. They’ll use whatever software libraries that you’ve never heard of or,

Teddy Svoronos [00:32:57]:

try some technique you don’t know about. And every time you give that up, you are building a little bit of cognitive debt, in that you now have a thing that you think you know how it works or what happened, but you don’t quite know, right? And so figuring, managing, I don’t think cognitive debt is a thing that goes away. I think it’s just a part of living, and it’s a thing that happens as a result of using these tools. And so the ability to manage cognitive debt, well, by which I mean figuring out how much do I need to understand for me to be confident that this is a thing I can sort of put my name behind? To what extent am I making assumptions that might come back to bite me later in terms of what actually is happening? Figuring out how to actively manage that cognitive debt, which I think happens through metacognition, is a newer skill that I think we don’t really incorporate into AI literacy stuff now, that I think we’re increasingly going to have to. Because the idea of offloading stuff sure is a thing that happens, but there’s stuff that you get in return.

Teddy Svoronos [00:33:57]:

And so thinking of that trade-off and what you give up and what you need to keep is, I think, a really, really important thing we need to be teaching.

Bonni Stachowiak [00:34:03]:

Years ago, I had Maria Anderson on the show, and her background is in math. I hate to oversimplify her background because people who know her are going to be like, seriously, you just went with math. But I’m trying to share her discipline with you, Teddy, so you’ll have a sense of. But she came up with this scale, and she’s really encouraged again for more than a decade now that we need to be getting together like this thing that you’re convening, and we need to be- we need to stop thinking about our learning outcomes as so dichotomous. So she has a whole scale and one of, on one extreme of the scale is you just need to know that this thing exists. So for me, Teddy, I’m going to tell you, maybe I’m understating it. I feel like I just barely know that GitHub exists.

Bonni Stachowiak [00:34:47]:

I have an account, I know how to upload stuff. But even with these experiments I’ve been doing where I pick a concept that’s difficult for students to understand, I try to bring humor and playfulness in. I build a little thing, and then I upload a file. But I’ll often forget even the step you’re not done once you’ve uploaded it. Like I’m used to Dropbox, oh, something’s uploaded. I right-click, I choose share that link, and now you can see it.

Bonni Stachowiak [00:35:09]:

And it’s like, no, no, there’s pages in there, you got to put it on the pages. And then you have to make this- So I have, I literally have a checklist for myself that. And this goes all the way back to the checklist manifesto. Because I’m thinking, I really do just barely need to know, okay, GitHub exists. You have an account, log in, upload your stuff, and there’s these steps that you’re not going to remember because you don’t do them that often.

Bonni Stachowiak [00:35:32]:

And so go to your little thing, and then do those steps. But remember that there is a step that if you forget, it is going to be really important to anyone ever being able to see that game. So as I think about this event that you’re convening, and then I think about this scale that I love so much, where it’s everything from know that this thing exists to, like the overused analogy of, yes, be able to take the clock totally apart, or even just build a clock from little spare parts in your garage, you know, and we act like so many of us within our disciplines that, for every single thing we know, we need to be out in the forest and be gathering, you know, things and be able to build. And it’s- that’s just not, it’s not realistic. And we lose our credibility with students when we try to act as if every single thing that may ever come up in their learning needs to be that way. So I’m curious, as I just shared that scale with you and you’re thinking about this convening, are there one or two concepts that are the most debated in teaching quantitative research skills that you think would create the most discourse and debate around which end of the scale, or somewhere in between may happen? 

Bonni Stachowiak [00:36:43]:

Like what what would maybe generate the most debate for the scholars and educators you’re going to be bringing together?

Teddy Svoronos [00:36:48]:

Awesome. So I should say briefly, just that your experience with GitHub, I think, is like such a nice version of what I was sort of saying, which is that you now can. You’re now using a thing you haven’t used before.

Teddy Svoronos [00:37:01]:

It is empowering you in lots of ways, you don’t “fully understand”, if you were just typing everything by hand, you probably would have a lot of trouble actually getting GitHub to work with the right syntax and stuff like that. But you need to understand it well enough to make sure that when, at the end of the day, what you think happened, happened. And so you are figuring out ways to manage this, a checklist is a really nice idea of a way to do it. But that is the kind of managing of cognitive debt that I think is really messy. And that’s part of why I think people are so quick to either say you need to understand everything, or it doesn’t matter if you understand anything, right, because the middle is really messy. But I think is what we need to kind of be worrying about.

Teddy Svoronos [00:37:43]:

And so in the quant methods area, I think the biggest one that has sort of really come up in the past couple months, because of all these models, is whether we should be teaching students, or how we should be teaching students how to conduct statistical analyses themselves. So, most courses now you learn statistics, you learn how they work, but you really want to be like doing statistics. And to do that, maybe you use a point and click thing like Excel, maybe you use a programming language like R or Python, or to a lesser extent, Stata to do your analyses. And when you do that, you have to learn to code, you have to learn what the functions are to make data do a certain thing. You have to figure out how to specify models in a way, so that they’re being evaluated in the right way. And it’s sort of interesting because it’s both mechanical, it’s both getting the code right, but there’s a lot of knowledge bundled into that. So why you order things in the right way? Like, first you only focus on individuals with a particular characteristic and then analyze them. That one two-step has a lot of stuff about it with respect to conditionality and conditional probability and stuff like that.

Teddy Svoronos [00:38:50]:

And so the challenge here, from my perspective, is I personally, for the kind of students I teach at the Kennedy School, don’t think I need to be making students really good at coding manually. There’s an extent to which I think people are wondering if anyone needs to be really good at coding manually anymore, because that syntax is now basically doable by a lot of these models. But then what does it mean to direct an AI agent well, to do that analysis, right? What do you need to know? Do you need to know? Do you need to be able to specify every step of the process? Do you need to be able to tell it what functions to use when it does that particular analysis? And what do you lose by saying, figure it out? I want to make sure we’re only focusing on this subgroup, and then run the following model on that subgroup, and then letting it kind of go with that. There’s some middle ground that I think is hard to figure out, because get too granular and you’re not actually using these models, these LLMs, for what they’re capable of doing. And you’re kind of making your analysis analyses more constrained than they should be. But if you get too abstracted and basically let these models do the analysis themselves, you’re going to get conclusions that you think are because of one way of doing it. And it’s actually because of something totally different. And so this debate now about what the sufficient level of abstraction is for code, when it comes to statistical analysis is, I think, a really central concern right now, and one that I’m thinking a lot about.

Teddy Svoronos [00:40:14]:

Because one interesting thing about all of this is that, these models now I think, can do a huge number of things that I myself have spent my career doing. I literally think parts of my job are now fully doable by these models. On the other hand, I guess the good part is expertise still does really matter even with these new models. And I can feel it when I use CLAUDE code to do things I don’t know that much about, the output is worse. There are more mistakes, there end up being bugs that I didn’t even think were there, or I didn’t even think to think about. And so the ability to direct it well still matters. But I’m not sure if we can develop that kind of intuition, expertise, judgment without the more granular getting in the weeds thing. That I think is the tension that we’re kind of struggling with now.

Bonni Stachowiak [00:41:01]:

All right, well, this is the time in the show where we each get to share our recommendations. And I’ve just got a quick one, I wanted to mention that I’ve continued to really enjoy on LinkedIn some of the conversations that are happening there, and many time past guest Jose Bowen recently posted about, he created an AI detector false accusation calculator, and he was using Claude in this particular case. I’m reading from him here: When you move the sliders for a false positive rate, class size, and estimated prevalence of student AI use, it provides numbers of students using AI who might be misled, or who might be falsely accused. And people who’ve been listening to Teaching in Higher Ed for a while are probably familiar with the debate. How much of a failure rate is enough for you, if you falsely accuse one student? Is that okay? 2, 7, 19. You know what would be okay with you? And people have very differing opinions on that. And I just love because, that’s one of the things that I’ve been long fascinated by, is data visualization. And there’s just something else about seeing that rather than just just it is all hypothetical, of course, but rather than just thinking about it and talking about it, well, go look at it.

Bonni Stachowiak [00:42:15]:

Do the calculator for your class that you’re teaching right now. Because I would hope you’re thinking about your actual students and not just hypothetical ones. And what would be too much to falsely accuse someone? I just thought it was a really compelling thing. And then what I loved about it, when you come together in community. Someone who I do not know, Emily Pacheco, came into the conversation. She took his version of this calculator, very visual calculator, and adjusted it for the enrollment admissions world. And then she shared it with a professional organization that she belongs to, the AI in college admission community. And what I was so excited about is I prepared to get to have this follow-up conversation with Teddy, and just so continue to be so curious about his work, is just when you get together, and you start to have conversations like the one he and I are getting to have today.

Bonni Stachowiak [00:43:05]:

And then some of these conversations that are happening out there in the public, is just that we can shape each other’s curios and imaginations in ways that are particularly powerful to me. So I just wanted to share that, encourage you to check it out if you’re not already following Jose Bowen. He is one that sometimes people will get, you know, he’s really kind of trying lots of things, and he’ll just kind of throw it out there, and sometimes people will be energized and excited, and other times they’ll go, what about this? What about that? It’s just kind of fun to watch. And his humility himself of, you know, as an experimenter and someone who’s curious is fun to follow. So, Teddy I’m going to pass it over to you for whatever you’d like to recommend.

Teddy Svoronos [00:43:41]:

Awesome. This is a really good recommendation, mine’s a little bit more nerdy, I guess, which is, we talked earlier in this conversation about privacy and security, and I wanted to make the case for what are called open models. So open models are large language models that you can download and run on your machine, and it does all of the prediction processing, everything entirely locally. You don’t need any Internet access to do it, and nothing ever leaves your computer. These models are, you know, they require your processor to be pretty fast. They’re slower than the best commercial models that exist right now, and they’re a little bit less capable, but they’re really good when you’re really concerned about privacy. So I use an application on my Mac called MacWhisper, and basically, what it does is it downloads any number of very high-quality open weight models that are about transcription.

Teddy Svoronos [00:44:32]:

So OpenAI makes some, Qwen is one other one, Jama is one other one, and you can choose the model, and you can dictate, and it’ll both take your dictation and it’ll process it to format it based on your specifications. So if you want to make sure it uses bulleted lists or whatever, when you start kind of talking about a list, it’ll do that, and it’ll do it in a way that’s way better than any other dictation I’ve used, and it will not touch any server. It’ll never leave my computer. And so using these kinds of open weight models, I think, can be very effective for sort of more narrow tasks. And I have found MacWhisper to be quite a good way to kind of get started with them in a very user-friendly way.

Bonni Stachowiak [00:45:12]:

So helpful. Well, I could keep going for hours and hours. We’re going to have to make it not so long before you come back again, so we can keep going, because I know there’s so much more you could share. Thank you so much for coming back on Teaching and Higher Ed.

Teddy Svoronos [00:45:25]:

Absolutely. It was great talking to you, Bonni.

Bonni Stachowiak [00:45:29]:

Thanks once again to Teddy Svoronos for joining me on today’s episode, and thanks to all of you for listening. Today’s episode was produced by me, Bonni Stachowiak. It was edited by the ever-talented Andrew Kroeger. And if you’ve not recommended, or shared about the show with colleagues in a while, I would love to have you rate it or review it in whatever service it is that you use to listen, or just suggest it to a colleague or a friend. I appreciate your support of the show, and I look forward to seeing you next time on Teaching in Higher Ed.