155 - Craig_Zilles

===

Craig Zilles: Last fall we had 65 different courses on campus using the facility. We have four rooms that contain a total of 260 computers. We run them 12 hours a day, seven days a week. So our, our centers are open from 10:00 AM to 10:00 PM seven days a week. Last fall we ran over 130,000 exams for those courses. So it's a thing that, that does work at scale, and it does work for a lot of people and, and there's definitely a bunch of people that have started doing retake exams who wouldn't have in any other context.

Boz: Welcome to The Grading Podcast, where we'll take a critical lens to the methods of assessing students' learning, from traditional grading to alternative methods of grading. We'll look at how grades impact our classrooms and our students' success. I'm Robert Bosley, a high school math teacher, instructional coach, intervention specialist, and instructional designer in the Los Angeles Unified School District and with Cal State LA.

Sharona: And I'm Sharona Krinsky, a math instructor at Cal State Los Angeles, faculty coach, and instructional designer. Whether you work in higher ed or K-12, whatever your discipline is, whether you are a teacher, a coach, or an administrator, this podcast is for you. Each week you will get the practical, detailed information you need to be able to actually implement effective grading practices in your class and at your institution.

Boz: Hello, and welcome back to The Grading Podcast. I'm Robert Bosley, one of your two co-hosts, and with me, as always, Sharona Krinsky. How are you doing today, Sharona?

Sharona: I am doing excellently. I am in a little bit of "Is this really happening?" Because there were a couple times this week where my name got thrown around with names like Robert Talbert and Jesse Stommel as references to look into grading stuff, and I'm like one big fish in our small pond was referring to me with other big fish, and I thought that was really cool. So I'm a little bit, flying a little high on that. And we're getting to record a lot this week, so I'm just in a really good mood.

Boz: Good.

Sharona: Yeah. I'm not gonna lie. How about you?

Boz: At the time of this recording, I am still in session. I'm not sure if I will be by the time it comes out, but I am still in session. We're doing a bunch of really big end of the year going into next year PDs, so I'm working with teams of people on these presentations. They're done, except now I'm going back and doing all the formatting so it looks like one product instead of four different people worked on it. So yeah I'm been staring at a computer screen for the last day and a half. I'm going cross-eyed. I am so happy to be here recording right now and not dealing with Google Slides or anything like that.

Sharona: That sounds like a good choice. So we're just gonna go with we love doing this podcast. And that's just super fun.

Boz: And we are not doing this podcast alone today. We do have a third person in the virtual studio. So who do we have with us, Sharona?

Sharona: So I'm really excited to welcome to the pod today a new guest, Craig Zilles, and I think I'm saying that correctly. Okay. I always should check that beforehand. Is a professor... He's a professor of computer science at the University of Illinois at Urbana-Champaign, and his current research focuses on applying computing and data analytics to education, particularly including the development of Illinois' computer-based testing facility, and that's a lot, hopefully what we're gonna talk about today because we know one of the big challenges with adopting this at big institutions with large classes is being able to do these multiple chances at testing. He routinely teaches large in-person undergraduate classes. And by the way, he came up with really high-end technical credentials, like undergraduate degree from MIT, first grad degree and second grad degree, including his PhD in computer science from University of Wisconsin-Madison. Now he's teaching at University of Illinois, but he's teaching large in-person undergraduate classes from 300 to 800 students, and that's led him to focus his research on how to do that effectively, and approaching it with an engineering mindset.

He's won a number of teaching awards, including the IEEE Education Society's 2010 Mack Van Valkenburg Early Career Teaching Award, and the University of Illinois College of Engineering's Rose Award and Everett Award for Teaching Excellence, and also has an NSF Career Award. So another heavy hitter, another big fish in a small pond person, so welcome to the pod, Craig.

Craig Zilles: Great. Thanks so much.

Sharona: You are welcome.

Craig Zilles: Great to be here.

Boz: So we are thrilled to have you on. There's a lot of things that we wanna talk about with you. Probably not gonna get them all today, but one thing that we always like to do with our new guest when they come onto the podcast for the first time, is ask you how did you get involved in this crazy world of alternative grading and alternative assessments?

Craig Zilles: Yeah. I think my story goes back to when I was still an assistant professor and I had to do a little research to dig it up, but a guy named Craig E. Nelson from Indiana University came to Illinois and gave a talk, and one of the things he was advocating for was what I now, I call second chance testing. I don't know if that's the term that he used, but he was an advocate of running every exam twice because there's a lot of reasons why students can't demonstrate the material the first time they take a test. And so giving them that optional opportunity to review the stuff they hadn't yet mastered and demonstrated on a second exam is really motivating for students to learn the material.

And I took that to heart and I introduced that in my undergraduate class, which was probably 150 students at the time, and it was a lot of work, but it was a thing that I could see clearly was benefiting the students and the students found very valuable. And so I think the thing we're gonna talk about is the technology that we've built to, to enable that to scale to large classes and to the point where it's not quite the massive undertaking that it can be in those large classes.

Sharona: So when you say second chance testing, are they retaking the exact same test or is it just a test on the same content?

Craig Zilles: Yeah. So I think you can formulate it in a bunch of different ways, but so it's not the same test, so we will write new questions. Back when I was doing it on paper I let them on a per question basis there's some number of slots on the exam for questions and, each question is testing some learning objective. Back when we were doing it on a paper because we were grading them manually, we'd let them check, put a check mark on the second chance exam of which questions they wanted us to grade, and we would only grade that subset. The way we do it now, we basically ask them to retake the whole test at the sort of test granularity. But, there's different ways you could do it. But it's definitely new questions that we want to test their ability to, do the skill, not just have you gone back and reviewed it and now you know the correct answer, but can you do it, that same learning objective on a new problem?

Sharona: And when was this that you started doing this approximately?

Craig Zilles: Do

Sharona: you know what year?

Craig Zilles: At least 15 years ago, so it's been a while.

Boz: So we're talking mid 2000 teens. 2000-

Craig Zilles: Yeah. Yeah ... 15, 18. Probably late

Boz: 2000s. Yeah. All right,

Craig Zilles: so- I started teaching in 2002, and this probably happened sometime in the first, five or so years of me teaching.

Boz: All right, so you-

Sharona: But I hate to tell you that, that's 20 years ago.

Craig Zilles: Yeah. Yeah. I'm an old guy.

Sharona: Just letting that out. I still got you beat by quite a bit, but still.

Boz: But it's weird how time works. I think, something in the '80s yeah, it was about 20 years ago. But you said when you started this, you started it with classes that had 150 students in them?

Craig Zilles: Yep.

Boz: One class?

Craig Zilles: One class, yeah. So- Wow ... we tend to teach large sections that, that instead of breaking courses into many small sections, we'll typically run a large lecture and then have discussion sections for, it's like breakout and ... But yeah and now I routinely teach classes that are, 300 or 600 students, depending on the class.

Boz: See, and that's one of the biggest questions we get. That's one of the hottest topics every year in the grading conference is how you do this with large classes. In fact, one of our keynotes last year was addressing just that. That is an issue that I personally have never had to experience. In my high school world, I've got a union contract, limits my numbers to 36, 38. At Cal State LA, most of my classes, have been under 30. So I have never personally had to deal with that issue. But the thought of it just blows me away of I know how much work it's for me now. I can't imagine doing this with 150, 300, 600 students. And you said you started off on paper.

Craig Zilles: Yep.

Boz: All right. So I would imagine it didn't take long before you realized, "Okay, this is a lot of work."

Craig Zilles: Yeah. It is a lot of work, and I think that's why, a lot of people in my position, they hear about alt-grading kinds of stuff and they try it for a semester 'cause they're excited, but some have trouble sustaining doing it. That, if it does take a lot more work, it's really hard for them to justify. And so the thing that we've done, I think, one of the nice features of STEM courses is a lot of the things that we're doing in sort of freshman and sophomore level STEM courses is we're teaching procedural knowledge. It's can you do this task? Can you analyze this circuit? Can you do this derivative? Can you know- annotate this free body diagram. So and there's a correct answer or a set of correct answers such that if you can capture the student's work in a digital format, then we can write computer programs that will grade them and give feedback. So we can automate the feedback process and the grading, and that, that really pays off in two dimensions. So on homework, we can do mastery-oriented homework that allows a student to attempt a problem, through a web-based interface. And again this, this is done in a number of contexts, but, solve a problem, submit the answer, and get immediate feedback on whether their answer is correct or not. I think we, I think we have a tool that lets us ask maybe a broader range of questions than, some people are used to in this context. But the first thing is being able to auto-grade a range of problems.

The second piece is being able to write question generators. So another feature of STEM and the sort of procedural knowledge that we're trying to teach is there's many questions that I can ask that allow you to test whether somebody can do something, that there's a lot of functions you could write that I can ask the students to take a derivative of, and we can also write computer programs that generate those, random functions. And one of the things that we've done is, again, is to put these two things together in the context of homework that allows a student to attempt a problem, get immediate feedback on whether they can do it or not, and then they can press a button that gives them another instance of that problem. So they can repeat the problems until they've, demonstrated mastery of that, and at that point, move on to, the next problem. So they have as much practice as they want, and we're not say, "Hey, do this small set of problems and submit it and get the feedback a week later." That in the moment they're getting the feedback and they can practice until they've mastered the skills.

Sharona: So how did you get from 2000 and whatever on paper to where you are today? What were some of the intervening steps? I just wanna kinda finish the origin story, so to speak, before we get to where we're at today.

Craig Zilles: Yeah. So we had shifted into doing this sort of mastery-based homework, so the tool that my colleague wrote. I wrote a very, simplistic version of this kind of mastery-based practice tool. My colleague Matt West, who's also a professor at the University of Illinois wrote a much better one what's called PrairieLearn. PrairieLearn I think I like to call it the most flexible question-asking platform in the universe. But basically anything you can do in a web browser, you can turn into a problem that you can potentially auto-grade. And so it's a very flexible problem-asking platform. Again, like we can, have students draw free body diagrams, and we can auto-grade those and give them feedback. We can have them, do symbolic math and auto-grade that. We can have them draw, graphs as-- both in the terms of drawing a function on an XY graph, but also in computer science, there's a notion of what's called a finite state machine, which is a graph of nodes and edges, that, that has a certain meaning, and we can provide students a little CAD tool that lets them do that. And again, because we're capturing their work in a digital format, we can write code that checks properties of the artifact that they produce and give them immediate feedback and auto-grade it.

So we had built that and so that had streamlined the-- That allowed us to do mastery-oriented homework, but the big bottleneck was exams, that exams we were still doing on paper, and that was really, a bunch of the effort in teaching the class was running the exams. And so my colleague in physics, had this sort of offhanded remark, which was like, "Oh, it's such a pain getting all the students in the same place at the same time to take an exam. Wouldn't it be convenient if we just had a room where whenever the student was free, they go to this room, and they can take the exam whenever they want?" And he was thinking, pencil and paper exams, but I said, "Oh, with computers, we can do this really well." And so back in 2014 we started piloting what we call the computer-based testing facility. So it's exactly that. We took a computer lab. We had it part-time. We only had it in the morning at the time. And we locked down the machine, so basically we control the networking on the machine, so we know the students aren't talking to other students and they're not accessing AI now or looking things up on the web, and we can-- We used PrairieLearn to deliver the exam. And We run the exams asynchronously, so we typically now give the students a three-day window. They will make a reservation for a time that's convenient for them, so they fit it into their schedule wherever they can. They show up to our facility. It's proctored rooms, so the proctors will check the IDs of the students to make sure they're the right person. The students sit down, and at that point, because we have all these question generators we generate for them a unique exam.

And if I'm taking it on Tuesday and Sharona's taking it on Wednesday, I can tell Sharona what was on my exam, but Sharona's exam's gonna be different. And we deal with that sort of what we call collaborative cheating potential by using a lot of randomization. And the fact that the scheduling is automated, the students are, using a web-based service to make their reservation. We're not dealing with conflict exams and all the emails flying back and forth when a student gets sick. They just make a reservation, once they get healthy. The proctoring we've centralized, so at this point we have, we're running the testing facility for a bunch of classes on our campus simultaneously. So I don't need to coordinate my TAs to show up at various rooms and proctor the exam, that the exams are being proctored, by a group of staff at the university And the fact that because, again, that we're capturing the students' work in a digital format, that we can do a lot of auto-grading.

This drastically reduces the amount of effort it takes to run an exam. And so it really lets me and other faculty deploy the exams in ways that are pedagogically good for students. The two things that, that we really advocate for is many small exams. So in most of my classes that every two weeks they'll take a 50-minute exam. So you you learn a unit of content, and then you get tested on that content, and then the opportunity to do retake exams. So, in my classes, I typically will do what, again, what we call second chance testing, where I give them one retake opportunity. And again, that's, that seems to be really valuable for the students, that the amount of extra learning that I think I'm getting, between the first chance and the second chance, I don't think I have to convince you guys of the value of retakes, that it really seems to allow a group of students to excel that may not have excelled if we gave them one shot at doing well on the exam.

Boz: So I, I've got tons and tons of questions, but first, if I'm one of your students, and let's say I take quiz one and I don't get proficiency or mastery at it, so I'm gonna need to take the second chance quiz. When do I do that? Is, how much time do I have to get my first quiz back, look at whatever kind of feedback there is, try to do that feedback loop to where I can actually, learn from those mistakes, gain that new knowledge? Now I think I'm at proficiency, like how much time do I have from one to the second?

Craig Zilles: Yeah. So I can tell you what I do, which isn't a requirement of how the CBTF is used, but what seems to work pretty well for a bunch of our classes is basically we do it a week later. So for example if my exams are running Wednesday through Friday, it would be the following Wednesday through Friday that they could sign up to take the retake exam. And our exams are actually graded interactively, so that after a student answers a problem, they can say, "Grade this problem right now," and they get immediate feedback. They can solve a bunch of problems and save the answers and grade them in bulk, but most of the students will attempt a problem and immediately have it graded because another feature of our exams is we will allow students to reattempt the same problem multiple times, potentially for fewer points.

So one thing that an instructor has under their control is how many attempts a student is allowed on a problem and what fraction of the points they can earn on subsequent attempts. So what this interactive grading means is by the time the students leave my exams for anything that's auto-graded, they know their score and they know which problems they got right and which problems they got wrong, so they can immediately start studying the things that they got wrong. They can immediately decide, "Okay I wanna study and retake the exam," and they know which things they, they need to start working on.

Boz: So do they-- 'cause you said they have the option of like immediately getting that graded. With that auto-grading, are they also getting any kind of feedback? I- is it just that, yes, you got this right or yes, you got this wrong? Or is there additional things that, the student would know that would help them get from where they don't understand it in one to when they do understand it in two?

Craig Zilles: Yeah. So it has to do with how you choose to write the problem, the kind and amount of feedback is really up to whoever writes the problem. So I have some problems that they actually give hints, when a student answers it, it'll come back and say, "Oh, your answer's not correct, and here's a little nudge towards the correct answer." So an example of that, again going back to my sort of finite state machine question, which again, I apologize, the audience probably don't know what this is, but we tell them something about the structure of the graph that they've built, that is incorrect, so they can go and potentially fix that and, resubmit it. And so we'll give them fewer points for that resubmission. Then typically after they've run out of attempts, we will show them the correct answer for that question. And so they can spend time during the exam if they have time left over to review those or, after the exam's over, they can meet with a member of my course staff and go over the exam.

The other feature I think that's pretty useful in how we do things is we typically provide students a practice exam generator. The way my exams are built is that a lot of it is, pools of question generators. So we for every slot on the exam we're typically trying to test a particular learning objective at a particular difficulty, but we might have multiple ways of doing that. And these question generators again, the question generator generates a bunch of instances of a particular question We often put those same question generators on homework and practice exams that, that we have enough of them that I'm not worried about the student, knowing what's on the exam and only studying that because I have so many ways of asking, all these questions that to study, like they have to study all the material in order to do well on the exam because they don't know what subset they're going to get when they show up.

And so we provide them a practice exam generator that has a significant overlap with the actual exam. And so what most of the students do is they'll remember "Oh I got the problem, on topic X incorrect," and they'll go back to the practice exam generator and they'll generate more practice exams and they'll practice that particular kind of problem. And so I think it's less important that they know exactly what they did wrong on the exam. As long as they know what the learning objective they're failing on, they can go practice it and then they can go to office hours if they're struggling with it to learn how to do it

Sharona: So I'm curious if, because we do something similar, not nearly as sophisticated as you, but we do have generators and those kinds of things. But I was doing this with my pre-calculus over the last couple of years, where I literally generated three examples of every one of the generators, gave them the practice exam, gave them the exam, and made a retake available. And I sat in a room, my, me personally, sat in a room all day, and I made myself available to all of the sections because I was the coordinator. And our uptake rate was very low on doing the retakes, and it was very clear that the students, despite having received feedback and having had the practice test, did not do anything between retake one and retake two or between the exam and the retake. So how do you think that people are communicating to students what to do in order to make this valuable? 'Cause my students, despite language that I gave to the instructors and things like that, they, it just, it was a complete flop.

Craig Zilles: Yeah, that's interesting 'cause my experience has been completely the opposite, that , it seems like this second chance is really like a lifeline and that we actually, our experience is that office hours are more utilized between the first exam and the second exam. And when I say that, I'm referring to three years ago back when students would go to office hours instead of going to ChatGPT. But we saw more use of the resources and more metacognition that I think everybody's had the experience where at the beginning of the semester you tell the students, "Okay, here's how to be successful in my class. You should do this and you should do this." And the students are like, "Blah, blah, blah," and they're like, they go and do their thing. It's only when their thing fails that they seem to be willing to be like, "Oh wait, how should I study for this class?" That there seems, so I think, again I can't say enough about retake exams that to me, at least in our population, they seem to be triggering the students to think harder about process than basically anything else we could do. It's super curious to me that you're having a different experience. And I think that's maybe something we should follow up on after the pod.

Sharona: To be clear, because I'm a huge fan of retakes, I personally think that my specific situation had two problems One is I didn't have, for this particular set of courses, instructors who were experienced and bought in to retakes. I had some that were, and the ones that were more experienced with them, they had higher participation rates from their students. So I think the ways that we talk about them in class matter, and they matter a lot.

And the second thing is I was teaching pre-calculus courses at a university, and many of those courses are for students who are underprepared. So I have a feeling that having a lot of underprepared students, they're not just underprepared because their math content is not good. They're underprepared because their study skills and their metacognition and their self efficacy in terms of formulating how to adjust, I think all of those skills are underprepared. And when that happens, I don't care how many times you give them a chance to do it, they need to be taught, and I don't know that you can do all of this at once, but they need to be taught how to learn from mistakes. And I suspect that, I, as a student going in, I went to, as my undergrad to Berkeley, and I struggled in my STEM classes. It was the first time I had ever struggled in school. I was able to learn how to learn, and I don't know that my students are coming in with some foundational skills on learning how to learn. So that is something to take into account. I think that retakes are phenomenal and yet not a magic bullet.

Craig Zilles: Yeah. Yeah, and I think a-another piece that, that may be part of it is those may be the exactly the students that are struggling in their other classes and working a part-time job, and they may just be resource limited as far as like finding the time to restudy and retake exams. And so yeah, I agree that none of these things are a magic bullet and, we gotta do the best we can to sort of meet the students where they are. But, learning calculus, for example, or pre-calculus is gonna take a certain amount of time, and if the student just doesn't have that time, it's gonna be, no matter what we do isn't gonna solve that problem.

Sharona: I agree

Boz: All right. So I've got another question that's just burning a hole in my pocket. You do your, both your original and your second chance asynchronously, correct? Are you st- and you're still doing that since-

Craig Zilles: And our final exam

Boz: ... A- okay. A- and you're still doing that since kind of the blowup of AI and programs like, Goth Math that you can take a picture with your phone and it gives you details on answer keys? Okay, 'cause that's been one of the big things that Sharona and I have been talking a lot about both on and off air. 'Cause that is one of the most impactful things that really came out of COVID for me, was taking my testing asynchronous. Because it gave me so much more of my in-class instructional time back that I could use for actual instruction instead of assessment. And this was something that we made as cornerstone of our statistics class that she coordinated and I taught along with several other professors. But that was one of the cornerstones of it, was this asynchronous testing that gave us so much more. But recently, that has been a divide now in, in this group about whether or not we should continue to do that asynchronous testing because of-

Sharona: Clarification.

Clarification. Asynchronous, unproctored.

Boz: Okay. So that was one of the things I was getting at is, so what are some of those safeguards that you're taking that's allowing you to continue to do the asynchronous testing and have the confidence that the student's work is the student's work?

Craig Zilles: So the next thing is that we have the students take the exams on institutional computers, so we can lock down the networking and the file system and so we can be pretty confident that the students aren't accessing, AI or anything that they're not supposed to be accessing. And then our proctors during the exam are looking for sort of other cheating mechanisms. So the main things that people try to get away with, which again, doesn't happen very much, but the main things that people try to get away with are, using their cell phone during the exam or sneaking in pieces of paper that have, some kind of thing that they're not supposed to have. And so the proctors are circulating the room during the exam just looking for, weird behaviors that would-- might be indicative of that.

At this point in the maturity of our testing facilities, we have video cameras in the room. So if the proctors think something funny is going on, we can go back and review the video camera footage to see "Oh, here's you pulling out your cell phone. Here's you looking down at your cell phone, looking up at the screen, looking down at the cell phone." So when we do observe academic integrity problems, we produce the most bulletproof, academic integrity reports that it's not just, oh, I said, I said this thing, you said that other thing. We literally have video evidence. And so the rooms are pretty secure, and I think they have to be because faculty are delegating their exam security to us. And so if they didn't feel like they could trust the testing facility to do this for them, they wouldn't, It just wouldn't be an option. And so I think the physical security is really important. I think it's really hard to do online exams that are secure, that it's just a, it's an arms race, and if you can't, you don't control the computer and you don't control the environment, there's just too many ways to get by the security that, you know.

We were forced, during COVID, we couldn't have students come to the facility, and we had to do the same Zoom-based proctoring that I think everybody was doing during that time. And empirically, the number of academic integrity violations went by, up by almost two orders of magnitude. So the number of things that we caught went up substantially per capita exams. But I'm sure there's a bunch of stuff that we didn't catch. And I think it's really hard, especially in this day and age of AI to trust online exams.

Sharona: Yeah. This has been a tough thing in our community because up until AI came out, one of the arguments was using alternative grading practices were more authentic ways of measuring learning, and that most students wanted to learn. And so when you took the incentives, the negative incentives built into grades out, the vast majority of students were just not going to cheat, and so you could trust students. And I agreed with that. Until AI came out, there was just enough friction left because I wrote authentic assessments, and even though they were taking them at home and unproctored, I always reserved the right to ask them about their questions, and I got pretty good. I missed a, I missed some, I got pretty good at being able to see. And a lot of times I would have them do their actual exams they would write on paper, like it would be delivered electronically, but they would write on paper and scan it and upload it. And I was also open everything. Yes, use your computer, use your phone, use your whatever. I would write exams that was okay with that. And that worked until AI came along. And now I'm like, the incentives are just too frictionless.

Craig Zilles: Yeah it's just too easy that it used to be you needed to pay some money to some service or know somebody, but now it's free and I think all the students are using AI for valid reasons also, and it's just the temptation is so strong. And I think there's some genuine misunderstanding in the students of what learning is and like why we have them do homework and what's the purpose of the stuff that we're asking them to do that, AI can do derivatives. Why do I have to learn how to do derivatives? It's we don't really care necessarily whether you can do derivatives. We want you to understand there is a mathematics of motion, or, change and, derivatives are a part of that.

Sharona: The problem is that I do think, speaking as a mathematician, that most of the procedural calculus is no longer worth learning now that computers can do all of that in a way that doesn't require coding skills. Like it used to be, even with Wolfram Alpha having been out 25 years or 35 years now, like you had to know enough to be able to put in the formula and the integral you needed, and AI's getting good enough that you don't need that anymore. And exhibit number one is the fact that we don't teach numerical integration anymore, even though that is what computers do. We don't even teach it. Like all of Hidden Figures, the movie, all of that math, we don't teach anymore. So I agree with you that they need to know rates of change and that kind of stuff. I don't agree that they necessarily need the procedural skills that we currently still teach.

Craig Zilles: Yeah. So- what I know about constructivism and, like, how people learn and how brains work suggests to me that you need to do a certain amount of the low-level stuff to build the intuition and the structures that I think it would be hard to learn high-level mathematical reasoning without understanding sort of how math works. That, I feel I'm in dangerous territory talking to mathematicians using math.

Sharona: Examples but- Yeah, I was gonna say I, I'm gonna try not to go down that path, but most of what we believe about why we teach math and the order that we teach it is wrong and false. So-

Craig Zilles: Okay. ...

Sharona: that, and it is not based on the mathematics. So we could do that on online another time. Yeah.

Boz: So that's a whole episode. Yeah. I, well-

Sharona: Yes.

Craig Zilles: Let me poke at one more thing, but so I like to think about the analogy of long division, that we ask students typically to do long division some number of times by hand so that they have an understanding of the process, but then we give them a calculator and never ask them again. Do you think asking them to do it by hand is a waste, or do you think that's potentially valuable for understanding?

Sharona: I think the long division algorithm in all of its forms is probably a waste because what we really care about is do they understand that division is parts and whole. Because if you ask a student to explain what a half divided by two-thirds is the one I'd like to use. What is a half divided by two-thirds or a half divided by a third? That one's really fun. And making them come up with the theoretical concept of that is much more powerful than teaching them to flip and multiply.

Craig Zilles: Yeah. Okay.

Sharona: So basically- And so the long division algorithm is specifically a space-saving way. It doesn't actually teach you division. It doesn't teach you parts and wholes. So- Okay ... but I could go on forever.

Craig Zilles: Yeah. So I would love to have this conversation sometime in the future. But yeah I think, genuinely the AI is really shaking up how we think about teaching, which I think could be a very good thing for us. But we're obviously in that sort of messy middle where we're just trying to catch up with this whole process of like- ... what should we be teaching?

Sharona: But I do wanna go back because I wanna make an argument in favor of your computer-based testing facility from the perspective of the work that we've done for the last 10 years at the center, which is regardless of where faculty are on their spectrum of knowing about grading and the problems with grades and those kinds of things, we wanna get to mass adoption of alternative grading practices. And we particularly wanna get to mass adoption at R1 research institutions because they're training the future faculty. So we wanna get to their undergraduate students and their graduate students with these better practices. And almost all the R1s, the big R1s I know, they all have 300, 400, 500-person classes, and you're not gonna get the vast majority of faculty to agree to do authentic assessment in the form of oral exams or some of these alternative things, 'cause you can't do it at three, four, 500 students. It's just not possible.

So, what I love about what you guys have done is you have let us recapture the class time like Bosley loves, right? Because it is asynchronous, it is proctored, but not in the evil, I'm watching your eyeballs Proctorio way. So it's not that kind of like invasive, personally invasive space.

Craig Zilles: Yep. We're not installing sp- spyware in, on their computers.

Sharona: That if you look away for too long, you're toast. But I do wonder when you catch people with integrity issues, what do you do? What's the sequence? Because I feel like we go from zero to a million with integrity issues without stuff in the middle.

So what do you do with a student when you catch them cheating? What's the process?

Craig Zilles: Yeah, I think that hasn't changed much. I think the things that have changed are there's a lot less arguing about did you cheat or not because we have, the evidence. The policy in my department, which, isn't I think anything special. Typically, like on your first instance, you get a zero on whatever it was and up to a loss of a full letter grade. And then on your second instance, it's an automatic fail in the class. Then the campus keeps track of some count. So if you are like a serial repeater there's consequences on your enrollment in the university.

I like to think that the presence of, if the students feel like the center is secure they're gonna try less stuff. And, the things that I like about alt-grading is I think the students are less desperate. If they know that they have another chance to come back and take the exam in an authentic way, then they don't, if they didn't get enough time to study for the first chance, like it's not the end of the world.

Like we've done studies on test anxiety, and students report significantly lower test anxiety when they know that there's going to be a second chance exam offered. And so my hope is, and I think, like what you said earlier in the conversation, that we're providing the students the tools that they can be less desperate. And I think most students don't go into classes intending to cheat. I think a lot, most cheating is an act of desperation. And so if we can provide the structures where the students can succeed without, having to cheat, then, they would much rather do that. One of my favorite bits of research we did around this is when we introduced the video cameras into our testing centers. We were worried about it feeling like a high security prison. That it was gonna be a really anxiety-provoking space to come in and take exams there. And so we did a survey of students, and first we asked them, "Did you know that you're being recorded?" And even though we by law have signs outside of our center saying, you're going to be recorded, half the students didn't know. And then we said what do you think about that?" And the, like 80% of the students said I think it's good that basically the gist was they want to be in an environment where cheating isn't rewarded. They wanted a level playing field. And I was like, "Oh, my faith in humanity is restored," that the students don't want an environment where they can cheat.

And I think both Stanford and Princeton recently got rid of their honor codes that prevent faculty from proctoring exams because the students know that, they want their degree to mean something. They want to be in an environment where they don't feel like they have to cheat in order to keep up with other students that are cheating

Sharona: I definitely agree with you on the desperation. And what I've done is I've, in my mind at least, have turned instances of cheating into learning opportunities. So because I do retakes, so if I catch someone cheating, it is an automatic zero out on that particular exam, but they can take the next one. But then they have to have an uncomfortable conversation with me, and my first question to them, I don't ask, "Did you cheat?" My question is, "Why? Why did you cheat?" And 99% of the time the student says, "Because." Okay, so I don't even have to have that... And it's out of desperation. And so what I point out to them I'm like, "Look, I'm gonna zero out this one. You're gonna get another chance, but here's why I need you to stop. Not because of me and my class. You're not hurting me. You're not even hurting other students." I said, "You're making poor decisions under pressure, and I need you to recognize that because when you go out into the world and you're under pressure and you have a manager asking you to steal money or cover up the theft of money or things like that, what are you gonna do then?

So recognize now that you're allowing desperation to have you make poor choices. Can we learn..." And some of them it works, and some of them it doesn't, and they cheat again, and they just keep getting zeros and I fail them. But I try.

Craig Zilles: Yep.

Sharona: Boz, I know it's hard when we can't see you. Do you have another question?

Boz: So I wanted-- 'cause w- we've been talking about this a- and you talked a little bit about the platform. We've been talking a lot about the testing center, but we talked a little bit about the platform earlier, and I kinda wanna get back to that. So, you've said the name, but I don't know if we really knew that you said the name. So what is this platform called?

Craig Zilles: So, we're faculty at the University of Illinois. We live in the prairie and so the tool is called PrairieLearn. So it's the word prairie and the word learn put together. Turns out prairie is like one of the hardest words to spell in the world. The, what's the order of vowels? But luckily, Google's good enough that if you put anything that could be pronounced as PrairieLearn, it'll find it for you. It's open source software so anybody can download it and use it if they want. It's under active development both by faculty at the University of Illinois who use it in their classes, but faculty at other universities contribute, or grad students and whatnot.

If there's a feature that you want that it doesn't have, we're very open to incorporating new things into it. My colleague Matt, who wrote it used to support the whole University of Illinois on top of his faculty day job, and that got untenable, and then there started being other universities that were using it. And so we set up a little company called PrairieLearn Incorporated which is if you're familiar with Red Hat Linux, so Linux is open source software, and Red Hat Linux is a company that sells services around Linux. Prairielearn.com or PrairieLearn Inc. Is basically the equivalent thing. So if you don't wanna host the open source software yourself, you can pay prairielearn.com to do it for you, and they will resell AWS time to you to run it, and they do all the work as when universities wanna use a tool, they need to make sure that it's- Yeah fERPA and SOC 2 and accessible and whatnot. And so this company has done all the work to check all the boxes that, universities want, and then they can also provide support. So if something happens or whatnot, there's somebody you can call to, to do that.

But it's a super flexible platform that, recently we've been working with math faculty to get some classes adopted. We've been in collaboration with the Gates Foundation on this, and so we added support for image capture. So now our testing facilities have cameras on every workstation, so students can derive a mathematical expression on paper and then take a picture of it and submit that as part of their exam or, on homework. We have support for what I call deterministic auto-grading, so you write a program that checks properties of a student answer. We can do manual grading, and recently we released support for AI grading. For example, those handwritten things I'm a huge advocate for AI grading on formative assessments, so like homework, because they're giving immediate feedback to the student saying, "Oh, it's on line three of your derivation that you made an error. Maybe go look at this reference that, gives you a hint of how to do this, and then you can try another problem." I think closing that loop on homework is really valuable.

I think there's a number of people that are concerned about using AI for summative assessment, which I can totally understand. And so my stance right now is that it doesn't cost us that much to manually grade the stuff that we can't deterministically auto-grade on exams, and so that's what most of us are doing right now. Yeah I encourage people, if they're interested, to check it out. You can get a free course to play around with. There's a bunch of open education resource questions that you can play around with, and we have a very active user community that both an active Slack and we run weekly office hours. So if you want to show up once a week on Zoom, there's both developers and other users that you can ask questions.

Boz: Okay. So I've got a couple of usability questions, and Sharona, interrupt me and jump in i- if I take up too much of the time we have left. But, you mentioned the capture image. Is-- And I know there's AI software out there. Is your system and your platform set up to where it can auto-grade those image captures?

Craig Zilles: Yes. Yeah. So right now that's done in bulk so that we don't have the immediate feedback loop closed. But I think AI grading is still new enough that it's good to have a human in the loop to make sure everything's going right before you, toss the scores back at the students. And so right now the AI grading is built as part of the manual grading interface. So you capture the student's image, you build a rubric, and the AI can help you build the rubric and refine the rubric. You check as many of the assignments as you want. You can compare your grading to the AI grading, and only when you're satisfied do you then release the scores to the students and the feedback.

Boz: Okay. My, my next question is, let's say I am at one of the 20 universities that are using this. And I did, I just jumped on your website and looked that up. There is-- you said there's a few colleges. There's over 20 universities across the country that are utilizing this now, including a couple out in the area that me and Sharona are at. The UC Davis uses this. So let's say I am a professor at one of these universities. Can I write my own questions? Do I have to use question or question banks that are already there? And if I can write, like how coding knowledgeable do I need to be able to write those kind of questions?

Craig Zilles: Yeah, this is a great question. So you can definitely write your own questions, and this is what most of us have done. Increasingly we're building a large bank of question generators is a lot of work, and so most people don't wanna do that, so we're trying to get people to contribute questions into the public domain so that there is a big pile of questions that you can start with. But definitely you can write your own questions and you can write your own question types.

There's a lot of things that if you're doing a common thing, so if you're writing a question that has a symbolic answer or a numeric answer you don't have to write any code to write that question. If you wanna turn that question into a question generator, there needs to be some code that does the randomization for that. But a lot of it's very formulaic that it's not, you're not doing anything big from scratch. And so even people that don't think of themselves as programmers, when they look at an example question, they're like, "Oh, that's really not that complicated."

Like-

Sharona: So to be clear, Bosley had to teach himself some Python and Sage from scratch because he had to code my question generators.

Craig Zilles: Yeah. Yeah. And it's, you know- He's

Sharona: that person ...

Craig Zilles: it's not rocket science eng- you know, it's oh, pick a number from this set of numbers or from this range of numbers. But what's really neat is that, we talked about the sort of negative things about AI. One of the good things about AI is it's really good at writing code, especially if it's writing code like some code that's already been written. And so a lot of our questions are variations on a theme, and so we've added to PrairieLearn a coding agent that will write the question. So you can describe the question in natural language, and then it will produce the question for you, and you can test it, and then you can have a conversation with it and keep tweaking it until you're done. So even somebody with no coding experience could write question generators for, standard types. It would be difficult to write like entirely, you couldn't You know, before the system had support for image capture, you couldn't use the AI interface to, add the image capture feature. But now that the image capture feature exists, you could, use the AI interface to say, "Hey, I wanna write a new image capture question that looks like this."

Boz: Let's I'm going to use my own experience as an example. Let's say I am a statistics teacher, a base level, intro statistics, and I'm trying to code some questions using, hypothesis testing or just using the Z-scores and bell curve. Can I have that conversation with the AI to try to get that question coded? 'Cause the coding to get something off of a normal distribution curve isn't as simplistic as, saying use a random coefficient between negative eight and positive 14.

Craig Zilles: Yeah. So I'm hesitant to say exactly what it can and can't do, but I am genuinely surprised by what it can do. One fun experience I had was I gave a talk in Finland and I don't know Finnish, but I had, one of the people there write questions in Finnish and of course, the AIs know a bunch of languages, and so it was happy to do that. I think, especially if there's a mathematical way of doing it which I think there is that you can, if there's a Python library that does the thing, you can probably ask it to do the thing and it'll find the Python library and do the thing.

Boz: I just realized I time is running away from us. We are actually coming up on time. Yeah. And maybe even past a little.

Sharona: We're pretty much at time,

Boz: yeah ... Sharona, d- I'm gonna give you an opportunity 'cause I've been talking a lot. Is there any last minute questions or things that you wanted to bring up?

Sharona: I guess the only question is, if someone wants a feature, is it on the website they would put into the website that they want this feature? Or how do they communicate with you that they're looking for a feature that you don't have? That's my question.

Craig Zilles: Yeah. I think the Slack is the best way to interact with the developers. Usually people will go to the Slack and say, "Hey, I wanna do X," and people will say, "Oh, here's how you do X, using the features that already exist." But the other thing is there's a lot of extensibility that can be done on the user side which, ... So again the image capture thing isn't part of PrairieLearn. It was an element that was built on top of PrairieLearn, and so users can write their own elements. That would require more coding experience, but there's a lot of things that, you could do without having to ask permission from anybody because the platform is extensible I think the other thing I, I would wanna touch on just to circle back to Sharona, you were saying that you're excited about the, the CBTF being a tool that our testing facility to enable R1s to do alt-grading things.

I just wanted to give the listeners an idea of the scale of what we've been doing. So again, we started this in 2014, so this is, we've been, our 12th year of operation. Last fall we had 65 different courses on campus using the facility. We have four rooms that contain a total of 260 computers. We run them 12 hours a day, seven days a week. So our centers are open from 10:00 AM to 10:00 PM, seven days a week. Last fall, we ran over 130,000 exams for those courses. So it's a thing that, that does work at scale and it does work for a lot of people and there's definitely a bunch of people that have started doing retake exams who wouldn't have in any other context.

It's also there's a number of universities besides Illinois that have started doing this. You mentioned UC Davis is a PairieLearn user, but UC Berkeley and UC San Diego and NYU and Michigan and University of British Columbia are all examples of other universities that have built computer-based testing facilities on our model. And if people out there are interested feel free to reach out to me because I would love to, to engage with you to help your institution start a computer-based testing facility also.

Sharona: And how do they get ahold of you?

Craig Zilles: Yeah. So my last name is Zilles, Z-I-L-L-E-S. My email address is that word @illinois.edu. But I think if you put Zilles CBTF in a, search engine of your choice, you would find me

Sharona: But we're gonna put you in the show notes to make it really easy.

Craig Zilles: Yep.

Sharona: And I just joined the Slack community as we were talking, so I'm gonna put that join link in the show notes as well.

Boz: All right. I wanna thank you so much, Craig, for taking some time out of your, I'm assuming, very busy schedule to come on with us. There's a lot of things I would love to talk more about. Hopefully we'll have some time to sit down a- and talk a little bit more. But I wanna thank you for coming. And for our listeners, thank you for listening. This has been The Grading Podcast with Boz and Sharona, and we'll see you next week.

Sharona: Please share your thoughts and comments about this episode by commenting on this episode's page on our website, www.thegradingpod.com. Or you can share with us publicly on Twitter, Facebook, or Instagram. If you would like to suggest a future topic for the show or would like to be considered as a potential guest for the show, please use the Contact Us form on our website.

The Grading Podcast is created and produced by Robert Bosley and Sharona Krinsky. The full transcript of this episode is available on our website.

Boz: The views expressed here are those of the host and our guest. These views are not necessarily endorsed by the Cal State System or by the Los Angeles Unified School District.