Boz: this episode about is just how that Math 1000 class is going.

Sharona: So putting me on the hot seat.

Boz: Exactly.

Sharona: All right. Well, I'd like to start, if I can, with a little bit of context-

Boz: Yes, please. ...

Sharona: both personally and for the course. So for the course itself, so we've talked a lot about our statistics course on the pod and how it came out of EO 1110, Executive Order 1110, that removed math remediation from the Cal States in 2018. And we said there were three pathways, and so we talked a lot about our path, which was the quantitative reasoning with statistics, and there's a very obvious pathway, which is the pathway to calculus that many, many, many STEM majors need. We didn't talk a whole lot about the third pathway. We sort of tossed it off as quantitative reasoning for everybody else.

Boz: Welcome to The Grading Podcast, where we'll take a critical lens to the methods of assessing students' learning, from traditional grading to alternative methods of grading. We'll look at how grades impact our classrooms and our students' success. I'm Robert Bosley, a high school math teacher, instructional coach, intervention specialist, and instructional designer in the Los Angeles Unified School District and with Cal State LA.

Sharona: And I'm Sharona Krinsky, a math instructor at Cal State Los Angeles, faculty coach, and instructional designer. Whether you work in higher ed or K-12, whatever your discipline is, whether you are a teacher, a coach, or an administrator, this podcast is for you. Each week, you will get the practical, detailed information you need to be able to actually implement effective grading practices in your class and at your institution.

Boz: Hello, and welcome back to The Grading Podcast. I'm Robert Bosley, one of your two co-hosts, and with me as always, Sharona Krinsky. How you doing today, Sharona?

Sharona: I am doing well, although I'm a little bit nervous because today we are starting into our new cycle at the theater. We have auditions today for our main stage, Frozen, that's gonna be performing in February, and I have been doing this a long time now, and I absolutely hate casting. I love auditions because getting to see everybody is fabulous and they do such a great job, but actually picking people sucks. It's terrible.

Boz: It the picking the people that's terrible or is it the not picking the other people?

Sharona: It's, it's a lot of it because, you know, weird things happen in the room. People have bad days, people have great days, and because we work with the same people over and over again, it's not purely... It's never purely on merit. It's all about a puzzle, who fits with who, who's saying well, who's due for something. So it's just, it's complicated. There's a lot of personalities in the room. But yeah, at the end of the day, what I really know, especially for younger people, 'cause we cast a lot of kids, is they're gonna make the fact that they didn't get something mean something about themselves that it doesn't mean, and it's heartbreaking to have to talk to some of these kids and say, "You did nothing wrong. You just got beat in this specific moment for this specific role." And it's just, it's heartbreaking.

Boz: Well, I do hope you enjoy the actual audition part. You said you love, love that part just 'cause you get to hear and see everybody.

Sharona: Yeah. I get to smile at a lot of people I know. I also don't love the fact that it's really long. It's gonna be like nine hours, and that's after we record today, and we have a faculty learning community today for three hours, so I have like a 15-hour day in front of me So yeah, I'm whining about getting to do the best stuff.

Boz: All right. Well, you did mention one thing, that we are doing later today, which is that ongoing FLC. But what else do we have? Do we have some other things coming up here?

Sharona: Yes. Yes. I am really excited. So there's some real opportunities for people who wanna do some in-person stuff. So the first thing is that you and I are going to be talking on October 24th, which is a Saturday, at the MAA Southern California Section Meeting, which is gonna be hosted at Cal State LA, which was kind of funny 'cause when I found out about it, someone's like, "Oh yeah, it's at Cal State LA." And I'm like, "It is?" And then we were invited to speak, so we're gonna be giving our Grading is the mis-use math talk. We are also going to be having a conversation like that on campus at Cal State LA November 3rd. I don't know yet if you're gonna be there. I'm definitely gonna be there, trying to kick off grading reform on our campus, which is kinda scary for those of you that have tried to do that on your own institution. And then we have some registrations opening.

Boz: We've talked a little bit about it on the podcast before, but we've got our first in-person conference that we are hosting, the Northeastern Regional Conference. It's, it is going to be a, a smaller conference. But registration-

Sharona: Well, we think.

Boz: But yeah, registration, ~um, ~should be opening soon. ~Um, ~you can check out all the details about the conference, who's organizing and some of the actual speakers along with the registration on our website.

Sharona: And I do wanna announce that one of our plenary speakers, and I think he listens to the pod, is going to be Dan Drees from Chapman University. So I'm really excited to get him, up in front of the conference and talking about what he does in alt-grading. So shout-out to Dan if you're listening. And then we have another one opening soon. So that was the Northeast Regional. What's the other registration that's opening pretty soon?

Boz: The big 2027, grading conference, even though, it doesn't come out until June, registration will be opening soon, and we've already got our keynotes listed. So-

Sharona: We do. So yeah, get-

Boz: you wanna

Sharona: share some of those?

Boz: Yeah. Go ahead. Who do we have this year as our keynotes?

Sharona: So I'm really excited for these because we're really getting a wide variety of speakers now to, Actually, I don't know if we've officially announced those now that I think about it. I know we know who they are, but they're not on the center. I'm not sure we can announce those yet.

Boz: All right. So that was a little bit of a tease, I guess then. We will be announcing them shortly.

Sharona: But I will say this, we are definitely continuing to have a breadth of people from both directly within alternative grading as their core, but also some people who are nationally known for some other types of pedagogy that they do, but are actually also alt-graders. And that's something I'm excited about is sometimes we're getting a lot of the crossover now between other types of good pedagogy and the fact that the people who promote those things are also alternative graders, and that's just really exciting. And then we have one more, which we've mentioned before, which is the Joint Math Meetings. So I'm gonna be on site at the Joint Math Meetings. We're gonna be doing talks about alt-grading. We're gonna have the center there. We're gonna have your mini me there. We're gonna have a Bosley there, but not that Bosley. So lots of stuff coming up in the near future.

Boz: And one other thing that's coming up in the very near future is we're gonna have Becky Pepler back on the podcast. And the reason I'm mentioning her, specifically, 'cause we've got a lot of great guests coming up, but we were just on her podcast, on the, Fires and Grading, the podcast that she does with, ~um, ~several of the other people from, First Education. ~First~

Sharona: Educational, yeah.

Boz: They interviewed us a little while ago, and that just came out on their podcast on October 2nd, so just a few days ago. So if you're not already following First Education and, their podcast, Fires and Grading, go check it out. It, it's, it's a great podcast, and it was a fun interview that we did with them.

Sharona: Well, and if you've never seen us in person- ... this podcast is actually recorded, theirs is in video also. So you can actually go to YouTube and see us talk, which I'm not sure you want to, to be honest. But it is out there now. So you can get them on all the major podcast players, but they are also on video on YouTube.

Boz: It's great talking with that whole group of people over there. I, I still say that, Extinguishing the Fires is one of the best when it comes to how-to books. Like, a- anytime I'm doing any kind of planning, that, that book is with me. So yeah, go, go check it out. Go check out the book. They're an absolute great group of people to work around and with. So looking forward to having her back on.

Sharona: Yeah, because they work more in K-12, and even though you're a crossover person between K-12 and higher ed, you're more known in the grading world for the higher ed stuff.

Boz: Yeah.

Sharona: And so it was really nice to kind of cross over and talk about what colleges and universities are doing in their world, and then having Becky come on, and, and Garth and Don and talk about how K-12 is feeding into this. So it's really fun.

Boz: But talking about college, and we've kinda teased this for a little bit, a- and this is really what I wanna talk about today, is you've got a brand-new course this semester, that you've never taught before. And we did an episode or two over the summer about kind of your process of planning for this course. So now that it's going, now that we're, what, six weeks deep into the semester, I wanna revisit that and kind of hear where you fell on some of your grading architectural, decisions, how the class is going, what adjustments you're already, either trying to do this semester or planning if you get to do this class again in the future.

So, if you don't mind, that's kinda what I'd like to talk this episode about, is just how that Math 1000 class is going.

Sharona: So putting me on the hot seat.

Boz: Exactly.

Sharona: All right. Well, I'd like to start, if I can, with a little bit of context.

Boz: Yes, please. ...

Sharona: both personally and for the course. So for the course itself, so we've talked a lot about our statistics course on the pod, and how it came out of EO 1110, Executive Order 1110, that removed math remediation from the Cal States in 2018. And we said there were three pathways, and so we talked a lot about our path, which was the quantitative reasoning with statistics, and there's a very obvious pathway, which is the pathway to calculus that many, many, many STEM majors need. We didn't talk a whole lot about the third pathway. We sort of tossed it off as quantitative reasoning for everybody else. We're not really-- It's not for people who need statistics. It's not for people who need calculus. It's anybody else. The title of the course in the catalog is Quantitative Reasoning for Everyday Life, and that is the most non-descriptive title that exists in the world. Like, what the heck does that mean, right? And I was aware of the course, and I had been told its, its number is Math 1000. We have 1000 and 1002, which are the same course, just one of them is supported for students who are less prepared, one of them's not, but they're really the same exact course, and I'm teaching a section of each.

And I was sort of vaguely aware that it was a combination of the mathematics of voting, some statistics, and some finance. But each of these, like statistics, we do a whole class on it, and finance, I mean, I had a finance class in my MBA program, so, like, that's a huge topic. So I'm thinking, didn't really pay attention to it, and then about a year ago, I helped the coordinator of that class sort of rebrand the class, keep the content, but, like, simplify the way that we talked about it to students. But I still hadn't really thought deeply about the class. And then I was handed it to teach. Now, I've never had the mathematics of voting, but this is a freshman level, university level math class. Like, I'm not gonna have trouble with the math. I mean, I practically have a PhD in math. I can handle the math, whatever it is. But I'm sitting there last summer, and this is what you guys heard me grapple with, going, "I don't understand what this class is." Like, these three buckets, why these three? Why do these matter? Am I just teaching them a little bit of civics and a little bit of personal finance and a little bit of statistics?

And that didn't sit right with me. So I struggled for a long time last summer, and I was working on the learning outcomes, but I was trying to come up with the story of the class. And then, I was procrastinating a lot. Because I didn't know the story of the class. I really, really, really struggled with the story of the class, and it wasn't until after I went to MathFest, and I happened just by chance to go to one of their plenary talks, and a woman by the name of Moon Duchin, who is a mathematician, or a person by the name of Moon Duchin, runs at the University of Chicago, the Data and Democracy Lab. She's a research mathematician, and she does research into the mathematics of voting. And I heard her speak, and I understood the voting piece more And that actually gave me an entry point to the whole class.

So I finally came up with a story, which is how does mathematics allow us to take individual level data, aggregate it, or do math on it to get to group level data, and then how does human decision-making come into choosing which of multiple mathematics pathways is the way we want to treat the data? Because it turns out on individual level data, you can choose multiple correct ways of doing math on that data And so which one do you choose? And that's true both for voting, but it's also true for finance because when you have government policy that comes into, say, how we tax savings versus how we tax income versus how we tax debt, like, people's choices change, and that's not the math,

Boz: We talked, personally even talk a lot about, the fact that statistics really is about making an argument using data. But what data you use, what argument you make, these are all human choices. And, and you can use the same data set and do different statistics and come up with two different arguments. Both are valid uses of statistics, so it's that human choice of which statistic, which, you know, measurement are you looking at, which combination. So, I been your thinking partner on this class all summer, and it really was when you came back and had that human choice in mathematics that, that kind of put this whole course in, in really in motion for you, and that gave you that three very different sounding bucket through line to where you could make one cohesive course instead of, like, three mini courses.

Sharona: Exactly. And one of those ways that statistics comes in is when you choose how to model and weight your electorate in a poll. And so, you know, what, what do you do with undecided voters? What do you do with this? What do you do with that? So again, what I like about this through line is a lot of us who grew up in the path to calculus method of mathematics, there's only one right way to do the math, supposedly, right? This is what we're taught. Whether we agree or not, you're taught this is how you factor, this is how you do this stuff, and the point is there's not one right way to do There, not even a, not one right way to do it, but there are multiple things you can do with a piece of mathematics that will actually get you different answers. My problem is a lot of what we're taught is, yeah, there's multiple ways to get to the right answer on the path to calculus, but there's so much more to math and quantitative reasoning than that.

Boz: I, I don't know if, you know, if you're in the calculus stream of mathematics, which basically is everything that you do in high school up to precalcul- I mean, th- that's one string of, of math. And I wouldn't say that there's one right way to do, but there is usually one right answer.

Sharona: Yeah.

Boz: And that's the big thing with, statistics and with what you've been talking about is not only is there more than one way to come to an answer, there is actually multiple right answers. A- and that's what I love about statistics, and this is why the creating a viable argument and critiquing the argument of others is a big part of any statistics class that I teach.

Sharona: Well, and to clarify, what I have been working with my students on is in any given method we're doing, there's actually one right answer mathematically. It's so we still have if you're starting in the particular method, then there is a right, like the, the math, the computation that you stick in the calculator comes out with one answer. It's just that there's multiple correct maths you can do. Yeah. So it's not even that we're not talking about the fact that there are multiple answers to a single set of mathematics. There's multiple sets of mathematics. Yeah. And that is what I'm trying to get my students to understand.

Boz: Which is a very is a very different thing. Like most of us don't see that in math, that you have a situation, you actually have multiple math pathways that you can take, and each of those are going to lead to a slightly different result. And which one is quote, "right"? Which one is the, the right method? And especially with, I think, the voting, your students are starting to see there isn't one right one. There isn't a correct one. It's a humanistic choice that has to be made, and the value of that choice is coming from the human's perspective. It's not the math, 'cause all, mathematically, all of them are right.

Sharona: Right. But that left me with a problem.

Boz: Okay.

Sharona: What the heck am I putting into my grades? Like, what am I trying to know if they know? Right. Because I didn't want this just to be I'm gonna teach them five different ways to do the math and can they get the right answer. Like, 'cause that's not the point either, right?

Boz: So before we get into some of the specifics of, how you ended up, structuring this, we talked a little bit about the purpose of the course itself. Give a real quick, background, like what is the typical student in this class? We know the demographics of Cal State LA, don't need to get into that, but where the students that are sitting in front of you, where are they coming from? Are they STEM majors? Like, what's that part of this? 'Cause that's, just as important when you're redesigning a course is knowing not just the details about your course and if it lies in a sequence and all that, but also who is your audience.

Sharona: Yeah. So these are, as was the case with our statistics class, these are largely first semester freshmen, and, they are coming primarily from, like I said, majors that do not have a specific quantitative reasoning requirement. So these are students for whom, again, usually they're not on a path to calculus because they just don't need to be. The types of majors that they have don't require it, and although we think that everyone could use statistics, again, their majors don't necessarily specifically require it of them. They're given choices, and therefore they actually come from a breadth of majors. One of the biggest majors at Cal State LA that's not in a STEM pathway or a nursing pathway actually is criminal justice. So I have a lot of, a lot of criminal justice majors, and the reason that that's so big at Cal State LA is we have the Sheriff County Lab on campus. So I think that we just have a very strong criminal justice. I have a few business administration, especially if they're in the marketing pathway, because if in they're finance or if they're in econ, they need the calculus pathway.

I have a very interesting major that I didn't know we have. Okay. We have aviation administration in our campus. So this is people who work for airports and airlines, but not as pilots, as administrators, and that major is actually in the College of Engineering, but yet they don't have a calculus pathway. So I didn't know that existed. I've got some anthropology, some journalism, some communications, some liberal studies, some English, you know, it just a wide variety of majors that are, are really not on a particular pathway to-- Oh, I have a couple of administration, administration people. I didn't realize. I've got an art p-person who's getting a bachelor's in art, some child development, television, film, and media. So just really running the gamut of the non-STEM, non-medical, and non, like, psychology, sociology majors.

Boz: So unlike our statistics class, our 1090s, which was very predominantly nursing- but my classes were always dominated by nursing majors. So unlike that, you really do have a, a wide range, which means most of the things that you do can be kind of geared and targeted to at least someone. But it's also maybe harder to get a lot of people, 'cause when I was doing the statistics class, if I wanted to make what we were doing, you know, real to our students, I just had to do anything around medical, statistics 'cause I had so many nursing. You've got a lot of ways you can go, but you're only, you're only hitting a certain group of people in your class.

Sharona: Exactly, and this is the first time I've designed a class since AI has come out. So that was my other thing I had to think about is: what does quantitative reasoning look like now in a world where AI is available to us, and is there a way to leverage that? Because I still have qualms about the use of AI, but I feel a responsibility to give my students opportunities, if they choose to, to learn how to use AI well, and for those who don't wanna use it, to allow them to not.

Boz: So let's get into it. Let's hear some of the details about your course, starting with just how many learning targets did you actually end up having in this course, and could you give us examples of a couple of them?

Sharona: Absolutely. So I wanna be clear that this is a course where this is the first time I'm designing it, so I've already run into things where I'm like, "Whoops, that didn't go as what I expected." So for example, I think I have a total... I'm gonna double-check this right now, but I have a total of, I wanna say 14. Yeah, 14 learning outcomes. 13 of them are content, and one is my triple P. Two of them are voting two of them are statistics, four of them are finance, and five of them are quantitative reasoning. My first voting one was too big. It's gonna get split into two in the future, so I already messed that up. But I mean, not that I messed it up, but I want to split some of the things that I actually ended up teaching and actually check those.

And one of the problems I was having is I was combining the design of my learning outcomes with how I was gonna assess them, so I didn't know how many quizzes I was gonna do versus other things, and now that I'm going through it, I see how I'm gonna adjust if I get to teach the course again. But for example, my first voting one is, I can determine the outcome out of an election using multiple voting methods and compare how different methods affect the results. So I did have an and in there, which I don't know, I, I'm still grappling with micro versus combination learning outcomes, 'cause what if they can do one versus the other? But because of some of the other decisions I made in terms of how I'm gathering evidence, I'm comfortable with this and, and I can explain that later.

Boz: Okay, so that's not the one you were talking about that you've already decided was too big.

Sharona: Well, ~I, ~it's not that this one's too big. It is the one that I decided was too big, but it was more just because the way I ended up quizzing There was a little too much in there, but it wasn't because of the determine and compare. It's just that there were too many methods that I wanted to check, and I couldn't get them all into a single quiz in the time that I wanted. So I'm gonna have to figure out how to split this one up, but it's not the determine and compare that was the problem.

Boz: Interesting. Okay.

Sharona: It's more that we looked at five or six different methods of tallying votes, and those broke across multiple weeks in the semester, and I wanted to give more frequent quizzing. So it was more of a logistics thing. So I handled it a different way. But what's interesting about this one is this hits right at the heart of the core of the class, because when you have a set of preferences from voters, what are you gonna do with it? Are you gonna require a majority, which means you have to do an initial and then a runoff? Are you gonna do plurality, which is just the biggest vote-getter wins? Are you gonna get multiple preferences like the ranked choice instant runoff stuff that Maine does? Like, there's so many different ways mathematically to run an election, and these different mathematical ways on the same set of individual voter data can have very different results.

Boz: Yep.

Sharona: So we did a lot with that one. And then the second one in the voting is, "I can evaluate voting systems using mathematical fairness criteria and explain the trade-offs among different methods." So again, even though there's an and and two verbs, in this case, they have to do both. You, you have to be able to evaluate them and then justify and explain your answer basically.

Boz: All right, so we've got around 14, standards altogether. You had a breakdown in kinda like four different categories, one of each of the three main, components of the class, and then that one that had, what, I think you said five, that's gonna kinda run the gamut of all parts of the class.

Sharona: Right.

Boz: All right. So there's our learning targets, and we actually talked a little bit in some of the previous episodes about how you came up with and what you were doing with those learning targets. So let's get into the grading architecture. Let's talk a little bit about what did we end up doing? How did we make the structure of the grading? What, what did you end up going with?

Sharona: So I ended up going with what in my head was a model that was the same as my history and math class, which is there are... What I was thinking in my head were three projects. But the grade is a combination of how many learning outcomes you get and how many projects you complete. What I didn't have clear at the time, that are much clearer now, is they're not really projects. They're portfolios, and there's a difference between a project and a portfolio in my mind.

Boz: Okay.

Sharona: And the difference is a project is a single completed artifact. The way I think of it in my head is if I print out a five-page essay and I staple it, that might be a project. But then if I take that and I insert it into a file folder with a bunch of other pieces of paper, that might be a portfolio So what happened is I was giving quizzes to check for things, but then I also wanted to do these investigations where students had to really take what we were learning in a unit and apply it to a situation that was customized to them. And that was where a lot of the evidence for the quantitative reasoning was gonna come through.

Boz: Okay.

Sharona: ~So, ~so they still have three things, but those things they have to turn in are these portfolios. And in these portfolios, they're putting evidence that includes the quizzes they've been taking, which have both revisions and retakes included, plus some of their homework that they think demonstrates their work, plus the investigation that they had to do, and some reflection. So all of those things go into the portfolio as sort of a bucket of evidence. So I'm grading all this stuff based on the individual learning outcomes, but also on did they complete and give me a complete set of this evidence.

Boz: And one of the things I love about portfolios, and I've used them in the past, not to the extent that I would like to, but what I really do like about those is it gives students some choice of what they want to present to you to say, "This is what I've learned in this, in this topic or in this learning, outcome." So it really does give students some choice. And when they explain why they chose it, you really do get to get a deeper understanding of just how well they know the material, and what kind of personal connections or insights they have to it. And I, I... If you are thinking about doing a portfolio, first, I encourage you to jump in and try it. Second, have that kind of, student choice aspect, but have them explain why they chose what they chose.

Sharona: Well, and the thing I like about that, not only does it give me more insight, but one of the things that we've been harping on for a few years is if we're gonna grade, it should be in support of learning. So making them do the reflections is not only helpful to us, but it actually is how they learn and solidify their own learning.

Boz: Yeah.

Sharona: So they themselves are gonna come out of this process as better learners and better students, because one of the things that you and I said on the Fires and Grading podcast that just came out, is we keep getting asked, "How can K12 schools prepare their kids for college?" And we both said, "Just teach them how to learn and how to reflect on their own learning." Well, if we're gonna tell K12 educators to do that, we better be doing that ourselves.

Boz: A- amen.

Sharona: So although it's not technically a learning outcome in my class, the point of the portfolios, in part, is to get them to reflect on their learning. Because I'm trying to teach them how to learn, and I'm trying to teach them how to reflect on what they learn and become metacognitive about it. So I haven't gotten the portfolios yet. They're coming in in a couple weeks. We're doing rough drafts of the investigation piece this week. It's gonna be interesting to see what they come up with ~and, ~and I'm excited to see how much they grow as learners.

Boz: All right. So that's kind of the first decision in your grading architecture, is once you have your learning targets, you know what you're going, to be basing your grading around, that first decision is how are you going to assess students on those learning targets? So you've got a nice mix. You've got some quizzes, you've got some investigations, and then you also have this portfolio that kind of encompasses multiple pieces of it.

Sharona: Mm-hmm.

Boz: Second decision is how do you grade these things? And I'm going to assume that because you have different components of assessment, do they all have the same way that you're grading or do they, do you grade them differently because, like you said, you've got investigations, you've got actual traditional quizzes, and you have this overarching, portfolio?

Sharona: Yeah, no, all three have different proficiency scales, with, with nuance. So the, the quizzes, ~uh, ~or rather I should say it, it's both the artifact that I'm doing differently, but it's also the learning outcomes. So the content learning outcomes, the six content learning outcomes, have a proficiency scale that's basically retake, revise, or complete. They're my, my emojis, although I, I've ended up grading on paper a lot, and I can't draw emojis, so I am using- ... check, revise, and retake, so I'm writing the words out. But basically if their quizzes get close enough, then they revise those quizzes. And if they don't, they have to retake another version of the quiz. What's interesting about the quizzes is I am trying what I had learned about from somebody called the multi-part quiz. This was presented at MathFest. So I bought these, like, multicolor pens the, the ballpoint pens that you click different colors down. And I make my students use those to take the first version of their quizzes, and then as soon as they're done taking the quiz, literally, like, I don't even collect the quizzes, and they're supposed to trade papers, switch colors, and try to correct the, each other's papers. And then they switch colors again, take their own papers back, and make corrections based on what their friends said. It is not working as well as I would like They are not, which, which-- Well, let me rephrase that. It's not making the quizzes as much better as I would like. I think it is teaching them an important skill, which is they're really having trouble correcting each other's papers. Yeah. They're really having trouble doing error analysis, even when somebody else's answer doesn't answer what they answered. And I'm telling them, "Put your heads together. Talk." But like, this is not done in isolation. They're sitting in table groups. Like, they have five or six people around a table, and they're supposed to be collaborating and putting their heads together and saying, "Well, you said this. Why did you say this? I said this," da da da, and talking, and they're just not doing it. It's like pulling teeth to get them to do this. And, so we're working on it. But that's also an important skill. This is something they're going to need. Right. So, it's fine-

Boz: That's one of the 21st century skills. That's one of the four, you know, the four or five C's, whatever you wanna call it, that communication and collaboration. That's a skill. There is almost no, you know, real careers out there anymore that you don't have to be able to collaborate either within the organization or between multiple organizations.

Sharona: Yeah. So again, not one of my learning outcomes, I'm not assessing their ability to do this collaboration, but they are getting to do this process. And they were a little bit better the second time than they were the first time. They've only done it twice so far because I don't do it on the retakes. I just don't have time, ~uh, ~because it takes time. But, it's been interesting getting them to do it, so we'll see. Right.

Boz: So that's the quizzes. What about some of the other aspects?

Sharona: So then the investigations are definitely gonna be on a revision cycle. Like, we weren't ready to do an investigation before we got to this point in the semester, and the portfolio is supposedly due in week eight. Really, it's gonna be due, and then I'm gonna kick it back for revision. So I do anticipate revision cycle, at least one on the investigation And then I anticipate that at the portfolio level, so the portfolio's just gonna be complete or incomplete. And so that's not so much the portfolio as a whole gets revised, it's just whatever pieces are missing, they're gonna be told to get them to me. Or if they're not-- like the investigation needs to be revised, they have to revise it. So it's really, I'm using a two-point, a three-point, two-- or two different two-points and a three-point proficiency scale.

Boz: All right, so that's how we're assessing the artifact that the student is using to show their proficiency. So that third decision is really how do you take all that evidence and decide when a specific learning target has been met? When has the student shown enough evidence that you can say, "I feel confident that the student is proficient at learning target one"?

Sharona: So this is a little interesting, and this is where when you mess up, you get really creative. According to my syllabus, they only need to show proficiency once on a learning outcome. However, because I messed up that first voting systems one and had to do the quiz in two parts, I kind of sneakily made it two times so that they had to do each quiz correct. But they, they can't see that. So as far as they know, they have to get it once, and I just called it VS1 Part A and VS1 Part B. So I kind of fudged that one, but really it's they have to get it to a sufficient level once because again, there's so much I'm trying to accomplish, and my thought was that ultimately they would... it's the content ones that I'm only doing on the quizzes. So those two voting, two, ~um, ~statistics, and four finance, those are on quizzes, I think. So that's the only one I'm doing that way. Everything else is being done through the portfolio. So yes, it's once, but you will have to have sufficient evidence, and I haven't defined that for myself yet. We'll see once those portfolios get turned in. One of my problems with doing a brand-new course that I have not taught before is I don't know what sufficient evidence is until the first time I look at it, which is terrible, but like this is the messiness that is this, you know? I don't get to rely on did they get 70% of the random number of points I put on this assignment.

Boz: All right. So, one of the things that I absolutely love, you know, and why I really wanted to, to do this episode, you and I have been doing this for a while, and you have done this now with multiple courses, either for yourself or, us helping other people design classes.

Sharona: Yep.

Boz: You didn't get it right the first time.

Sharona: Pfft. Not even close. Like, I'm lucky to survive the first time.

Boz: But to the point that you are making in you know, in semester changes, like it, it's, you can't even wait till the next time you do this. Y- you're making these changes now, and I, I'm calling you out on this not to make fun of you at all, and you know that, but just to show, especially anyone that's listening that might be newer to this or might have tried this and really gotten frustrated, this is something that we usually don't get right on the first try. I mean, we've talked to, you know, dozens and dozens of, if not hundreds of people. We've heard of maybe one or two that, you know, got it really well on the first try, and that's usually because they had a very close mentor helping them through the process.

Sharona: Well, and also, I mean, the difference between me and a lot of people though, to be fair, is most of the time when I'm doing my designs for myself, it's on a course I have not taught or have not taught in this structure. Because even though I've been teaching off and on for over 30 years, a large chunk of that I was not in a higher ed classroom. So I only came back to the higher ed in 2015, and when I came back, in rapid succession, we went from the quarter system to semesters, then we went from semesters to coordin- to dual enrollment, then I went from dual enrollment to course coordination with a brand new course. So I don't know, other than the statistics class and the linear algebra, I don't think I've taught the same class more than twice in the last 12 years. I mean, I've taught pre-calculus, once since I've gone to... No, twice since I've gone to alt-grading, but only once since it was redesigned in this Pathways program. I've taught Calculus 2 maybe twice or three times. I taught Calculus 3 once. I taught Calculus 1 once. I taught History of Math once. So like I don't have 15 years of in my head knowledge about how a course goes. So if it is a course that's very familiar to you, it will hopefully go better, but even then, it's very different than being able to slap some points on it and go, "Ah, this is good enough."

So I'd say the big difference between me and other people is I do have enough teaching experience to hide my own uncertainty from the students. So I sound really confident, or I'm also sometimes I'm honest. I'm like, "Hey guys, this is the first time I'm doing..." This is what I told them about the voting investigation, 'cause they asked me for an exemplar. And what I told them is I said, "That's a really good idea, and I should be able to give you one. I can't. I don't have one. After this semester, you all are gonna give me ones for future semesters-" "... but I don't have one right now." And I was just honest about it, 'cause I'm not worried about what they're gonna think about me, you know? They're either gonna like me or they're g- not gonna like me, and I want them to like me, but it is what it is, so.

Boz: Yeah, but also being honest with your students and letting them know, "Hey, I don't have everything figured out," and that's okay 'cause that's one of the things about learning is... mistakes allow thinking to happen. We learn from our mistakes. So being upfront and honest with your students about it doesn't do anything but help the relationship you have with the students, so.

Sharona: Exactly.

Boz: All right, so we talked about the first three. Let's get into the last one. How do you take these 14 learning targets and decide what's their final grade?

Sharona: So I've learned after 12 years of doing this, or 10 years or however long, is I like simplicity. So this is not quite as simple as when I just do standards, but it's very straightforward. Basically, I decided if you wanted an A in this class, you had to complete all three portfolios. If you wanted a B, you had to complete two. If you wanted a C, except for the C minus, you had to complete one. And other than that, I just stepped it down the learning outcomes. So you need at least 13 learning outcomes and three portfolios completed to get an A. You need 12 and three to get an A minus. You need 11 and two to get a B plus. ~Um, ~and really, you still are gonna need to turn these portfolios in because otherwise you're not gonna be able to get to, say, 11.

Boz: Mm-hmm.

Sharona: So 11 and two to get a B plus, 10 and two to get a B, nine and two to get a B minus, nine and one to get a C plus, eight and one to get a C, eight and zero to get a C minus. And if you have fewer than seven learning outcomes, you're not gonna pass the class.

Boz: Okay. So-

Sharona: It's a very straightforward

Boz: list

Sharona: all the way down.

Boz: It is on the simplistic side, but it's not a straight up end or, just count like we do in our statistics class.

Sharona: Right.

Boz: And one of the things that we always talk about when we're helping someone redesign is to be careful of the multiple components of a grade wrap-up, or have a way to watch for those, those edge cases. So I'm gonna give you an example, and I want you to kind of explain what the grade would be and what the rationale is. So-

Sharona: Okay ...

Boz: let's say I've done all three portfolios- Yeah ... but I only end up with eight learning targets. What would my grade be, and what's the rationale?

Sharona: So if you completed all three portfolios and you got eight, you would, by the grading scale, have a C.

Boz: Mm-hmm.

Sharona: Because these are minimums, right? So if you have eight, learning outcomes, you're supposed to get a C as long as you have at least one portfolio. So then the question is, is it possible to only get eight with three completed portfolios? And in my mind it's not, because I'm going to kick these back for revision if you don't have enough quantitative reasoning evidence in them. So they're kinda tied together in that way, ~um, ~but I haven't gotten there yet, so we'd have to see.

Boz: So, and that's kinda one of the things I wanted to ask that question specifically for, because I knew the answer already. We've talked about this off air and stuff. But one of the things that we hear all the time, and it's specifically homework, like this is where this comes up the most, is a student isn't going to get, you know, nine out of 10 learning targets if they're not doing the homework. So having those dual enrollments, those, oh, what happens if the kid gets all 10 learning targets but only has 40% of the homework? Oh, that can't happen. Students have to have the homework to... And I am living proof it can happen.

Sharona: Right.

Boz: I didn't-- until I got to Calc 2, I didn't need to do crap to get high marks on my test.

Sharona: Right.

Boz: That's something we always warn our, our new practitioners that we're helping redesign. That's something we always warn, because oftentimes we see when we talk about the warnings of, you know, being overly complex, it's usually this decision for where we see the most of the overly complex components come in. But this one is a little bit different with yours. The portfolio and the ways that you are assessing the, the other ways you're assessing aren't mutually exclusive. It's not like the homework and quizzes and stuff where those really are mutually exclusive. Your portfolios, for me to have gotten all three portfolios checked, like you said, there's other learning targets I would've had to have gotten proficiency along the way, otherwise I wouldn't have gotten that, that portfolio checked off as complete. It would've been kicked back to me. So when you say it's not possible, it's very different than that homework argument.

Sharona: Well, yes and no. It's, it's not possible in the scenario you gave me, which is a low number of learning outcomes but a high number of portfolios completed, because the portfolio completion is dependent on the learning outcomes. I definitely could see it the other way around, though.

Boz: Yeah, I,

Sharona: I was gonna- So I could definitely see...

Boz: Yeah. ~Yeah, I was, uh, uh, ~that's why I specifically asked the way I did.

Sharona: Right.

Boz: Because I, I did wanna switch it around as well.

Sharona: Okay.

Boz: My point was the fact that they're not completely mutually exclusive l- um, like homework and quizzes are It does make it actually almost impossible to get the portfolios checked and not have the learning targets.

But okay, let's switch it. So let's say I am that, that great test taker that I literally really am. Right. Let's... So I'm in your class. You have got the 19-year-old or 20-year-old version of me- god bless you, in, in your class, and I am, I'm, I'm killing every quiz you throw in front of me. I, I can knock those out of the park easily. Is there a path where I could have, you know, 12, 13, even 14 of these learning targets and zero or maybe just one portfolio checked? And if so, what's my grade?

Sharona: So first of all, the quizzes only cover eight of the learning targets.

Boz: Okay, so there's number one.

Sharona: I am not checking the quizzes by themselves for the quantitative reasoning learning outcomes. Those are only checked inside the portfolios, okay? So the maximum you could get would be, if you didn't actually turn in any portfolios, you could pass the class, in theory, because you could get the eight quizzes and you could get the triple P, and that's nine, but that'd be zero, nine and zero, so you'd get a C-minus instead of a C-plus. The bigger issue is there's a difference between I didn't do the portfolio and I didn't complete the portfolio, because I'm gonna be a bit of a stickler to say, "Hey, you gotta get your reflections in. You gotta get these things." So I could see a situation where you got 12 learning outcomes, you did your investigations, but you didn't turn in the reflections-

Boz: Okay

Sharona: or you didn't turn in your sources, and I kicked it back for revision, and you just didn't get them in.

Boz: Okay.

Sharona: I'm probably gonna give you either the grade you earned on the learning outcomes or maybe a third of a grade less, but I'm not gonna tell you that until the end of the semester. Like, it's not in my syllabus. I want you to do these other things. I want you to do these reflections, but I'm not going to hold a student to a very low grade because they couldn't get their reflections in. I'm just not gonna do it. So I always reserve the right to give you a better grade than what is listed in my syllabus, because I can defend a grade dispute if I give a student a higher grade than what's in my syllabus

Boz: Okay. But isn't that then like trying to control behavior through grades, which is what we talk a lot about on here that we don't wanna do, So isn't that trying to control behavior?

Sharona: No, I don't think it's controlling behavior. I'm trying to teach students a wide variety of skills, and some of those skills are how to learn, and big part of learning is reflection. So I intend for it to be part of the grade. And I also don't like being overly punitive on any one element of the grade, which is why my learning outcomes are designed the way they are. So I'm not trying to control their behavior, but I'm also trying to not penalize students.

Boz: Yeah, and that's why I wanted to ask the question, 'cause again, we've talked about this, I know the rationale, but I didn't want someone listening to this going, "Wait a second, you're controlling grades just like when I take off 10% every day something's late." You're not controlling the behavior, you're trying to wrangle the learning, because the reflection that Meg had gotten to piece is such an important piece. And we see this again ~and, ~and again, you know, from Hattie's study on to so many others, that that reflection is what really stamps the learning. It's what brings it in and makes it stick. So you're not controlling behavior with that decision, but at the same time, we tell everyone this, always have in your syllabus that out of, "I reserve the right to increase," "to give you a higher grade than what's outlined in the syllabus." So you've got that in your back pocket and, you do come across a knucklehead like I was at that age, and you're like, "Yeah," you get it, okay, yes, ~you, ~you don't think I actually deserve the C, and raise the grade.

Sharona: But the other hand is you might, because I might be getting a lot of the quantitative reasoning evidence from those reflections.

Boz: Yeah.

Sharona: And so there's a chance that you can't get to 12 without it. 'Cause remember, 12 means you've gotten almost all the quantitative reasoning, and if you're not giving me the reflections, I don't know if you can make those arguments. So it's, it's a hypothetical that, yes, could happen. I do see it happening potentially, and I have thoughts on each way. I think the low number of outcomes with high number of portfolios is unlikely because of how much I control that. Yeah. But I think the other way around, where it flips into the student's control, that's where I think you're gonna have a lot more of the issue, is when part of the result is based, is on the student's control.

Boz: So it, it's unlikely to have-- It's, it-- Not unlikely, it's almost impossible to have the student that is gonna have a high number where I've just blown the, the portfolio assignments off. What could happen is I've done them, I've maybe done, you know, the first round of all of them. I've given some of the reflection pieces, but something else gets kicked back, and I just don't do the revisions, and I never actually resubmit. So I've got aspects of it that maybe we're able to check off, but, you know, the completion... And, and you're right, that's m-my choice as a student of whether or not I'm doing those, you know, resubmissions after you kick a portfolio back to me. And you're aware of it. You're aware that out of the kind of three outcomes, you're aware that one could happen, and if it does, you're prepared to really take a look at that and make the professional judgment that only you, the educator, the professional educator, can make about have I, as the student, shown you enough of my proficiency and my understandings to warrant the grade that you ultimately end up giving me.

Sharona: Exactly. And then the other thing I like about this in general is ~it, ~it has built-in flexibility where I don't have to make exceptions.

Boz: Yeah.

Sharona: So if a student, if I know a student runs into a health issue at the end of the semester or whatever and can't get some of that last bit of stuff in, I still have all the evidence I need to give them a grade, and one I can justify if I get called on it by my administration.

Boz: All right. So I know we are coming close to time, but there's one last thing I kinda wanted to ask you about. So you've said this has been your first time teaching it. I know h-hopefully you'll get to teach it again. You know, we don't have control over what we, what we're assigned. But as of right now, we're about a third of the way through, a little over a third of the way through. What, if any, major changes would you look at for next year? And I'm not talking about the things you've already changed, going forward into the course, but if you're teaching this again, what's, if any, major changes that you're already looking at doing?

Sharona: So far I don't have any major changes that I would do. I just finished the voting unit, and I'm a little bit less certain exactly which content I wanna put in some of the other units. I have a general idea. I think the voting unit went very, very well overall, and so it would really be around the edges. It would be mostly on my side, on the prep work. Now I at least have first drafts of quizzes, I have first drafts of investigations. There's some paperwork that I'd change, change the way I give directions on the portfolio, but nothing major yet. I'm just really tinkering. I don't know that that'll be true once I actually see these portfolios, and I haven't had to grade them yet, so that's gonna be the next big hurdle is what am I actually, when paper meets the pen, what am I looking for in evidence on some of this?

Boz: Well, I wanna thank you for being willing to kinda be put in the hot seat on this. I would love to hear come December, come January, love to hear how the course ended up finishing out, , what the grades look like and just how successful overall you thought the course was. But really do wanna thank you for kind of sitting down with us and talking about all those architectural decisions and just how the course is going overall.

Sharona: Well, you're welcome, and wish me luck, everybody.

Boz: And for everyone else, you've been listening to The Grading Pod with Boz and Sharona, and we'll see you next week.

Sharona: Please share your thoughts and comments about this episode by commenting on this episode's page on our website, www.thegradingpod.com. Or you can share with us publicly on Twitter, Facebook, or Instagram. If you would like to suggest a future topic for the show or would like to be considered as a potential guest for the show, please use the Contact Us form on our website. The Grading Podcast is created and produced by Robert Bosley and Sharona Krinsky. The full transcript of this episode is available on our website.

Boz: The views expressed here are those of the host and our guest. These views are not necessarily endorsed by the Cal State System or by the Los Angeles Unified School District