{"id":1112,"date":"2010-03-14T19:09:44","date_gmt":"2010-03-15T00:09:44","guid":{"rendered":"http:\/\/mikeconley.ca\/blog\/?p=1112"},"modified":"2023-12-20T16:25:17","modified_gmt":"2023-12-20T21:25:17","slug":"lessons-from-peerscholar-an-approach-to-teaching-code-review","status":"publish","type":"post","link":"https:\/\/mikeconley.ca\/blog\/2010\/03\/14\/lessons-from-peerscholar-an-approach-to-teaching-code-review\/","title":{"rendered":"Lessons from peerScholar: An Approach to Teaching Code Review"},"content":{"rendered":"

We Don’t Know How To Teach Code Review<\/h4>\n

If you go to my very first blog post about code review<\/a>, you’ll discover what my original research question was:<\/p>\n

Code reviews. They can help make our software better. But how come I didn\u2019t learn about them, or perform them in my undergrad courses?\u00a0 Why aren\u2019t they taught as part of the software engineering lifecycle right from the get-go?\u00a0 I learn about version control, but why not peer code review?\u00a0 Has it been tried in the academic setting?\u00a0 If so, why hasn\u2019t it succeeded and become part of the general CS curriculum?\u00a0 If it hasn\u2019t been tried, why not?\u00a0 What\u2019s the hold up?\u00a0 What\u2019s the problem?<\/p><\/blockquote>\n

I have mulled the question for months, and read several<\/a> papers<\/a> that discuss<\/a> different models for introducing code review into the classroom.<\/p>\n

But I’m no teacher<\/em>.\u00a0 I really don’t know what it’s like to run a university level course.\u00a0 Thankfully, two course instructors from our department gave their input on the difficulty of introducing peer code review in the classroom.\u00a0 Here’s the first:<\/p>\n

The problem is that is completely un-assessable. You can\u2019t get the students to hand in reports from their inspection, and grade them on it, because they quickly realise it\u2019s easier to fake their reports than it is to do a real code inspection<\/em>. And the assignment never gets them to understand and internalize the real reasons for doing code inspection \u2013 here they just do it to jump through an artificial hoop set by the course instructor.<\/p>\n

What we really need to do is to assess code quality<\/em>, and let them figure out for themselves how the various tools we show them (e.g. test-case first, code inspection, etc) will help them achieve that quality. Better still, we give them ways of measuring directly how the various tools they use affect code quality for each assignment. But I haven\u2019t thought enough yet about how to achieve this.<\/p>\n

So, I\u2019ve long since dropped the idea of a specific marked assignment on code inspections, but still teach inspection in all of my SE courses. I need to find a way to teach it so that the students themselves understand why it\u2019s so useful.<\/p>\n

(From Steve Easterbrook<\/a>, commenting on this post<\/a>)<\/p><\/blockquote>\n

And here’s the second:<\/p>\n

1. How many different tasks can we ask students to do on a 3-week assignment? I think students should learn to use an IDE, a debugger, version control, and a ticket system. We have been successful in getting students to use version control because that\u2019s the only way they can submit an assignment.<\/em> We have had mixed success getting students to use IDE\u2019s and debuggers, partly because it is hard to assign marks for their use. We have been even less successful in convincing students to use tickets because a 3-week assignment isn\u2019t big enough or long enough to make tickets essential.<\/p>\n

2. If the focus of my course is teaching operating systems, how much time (and grades) should I devote to software development tools and practices that aren\u2019t centered on operating systems?<\/p>\n

(From Karen Reid<\/a>, commenting on this post<\/a>)<\/p><\/blockquote>\n

All of this swirls around a possible answer that Greg Wilson<\/a> and I have been approaching since September:<\/p>\n

What if peer code review isn’t taught in undergraduate courses because we just don’t know how to teach it?\u00a0 We don’t know how to fit it in to a curriculum that’s already packed to the brim.\u00a0 We don’t know how to get students to take it seriously.\u00a0 We don’t know if there’s pedagogical value, let alone how to show such value to the students. <\/em><\/p>\n

If that’s really the problem… Greg and I may have come up with a possible solution.<\/p>\n

But First, Some Background<\/h4>\n

In 2008, Steve Joordens<\/a> and Dwayne Pare<\/a> published Peering into Large Lectures:\u00a0 Examining Peer and Expert Mark Agreement Using peerScholar, an Online Peer Assessment Tool<\/a>.<\/p>\n

It’s a good read, but in the interests of brevity, I’ll break it down for you:<\/p>\n

    \n
  1. Joordens and Pare are both at the University of Toronto Scarborough, in the Psych Department<\/li>\n
  2. Psych classes (especially for the first year) are large.\u00a0 For large classes, it is generally difficult to introduce writing assignments simply due to the sheer volume of writing that would need to be marked by the TAs.\u00a0 Alternatives (like multiple-choice tests) are often used to counteract this.<\/li>\n
  3. But writing is important.<\/li>\n
  4. The idea:\u00a0 what if we let students grade one another?\u00a0 There’s research showing the benefits of peer evaluation for writing assignments<\/a>.\u00a0 So lets see what kind of grades peers give to one another.<\/li>\n
  5. A tool is built (peerScholar<\/a>), and an experiment is run:\u00a0 after submitting their writing assignments, show students 5 submissions from other students, and have them grade the work (with specific grading instructions from the instructor).\u00a0 Then, compare the grades that the students gave with grades from the TAs.<\/li>\n
  6. A significant positive correlation was found between averaged TA marks and average peer marks.\u00a0 More statistical analysis shows that there is no significant difference between the agreement levels of TA and peer markers.<\/em><\/li>\n
  7. To ensure repeatability, a second experiment is run – similar to the first.\u00a0 Except, this time, students who receive the marks from their peers are able to “mark the marker” and flag any marks that seem suspicious (a 1\/10, for example, if all the other students and the TA gave something closer to a 7\/10).<\/li>\n
  8. It looks good – numbers were closer this time.<\/li>\n
  9. Conclusion:\u00a0 the average grade grade given by a set of peer markers was similar to the grade given by the TAs in terms of overall level and rank ordering of assignments.<\/li>\n<\/ol>\n

    This is a very interesting result.\u00a0 Why can’t we apply it to courses in a computer science department?\u00a0 What if students started marking each others code?<\/p>\n

    What they’d be doing would be called code review.<\/em><\/p>\n

    The Idea<\/h4>\n

    Let’s modify Joorden and Pare’s model a little bit.<\/p>\n

    Let’s say I’m teaching an undergraduate computer science course where students tend to do quite a bit of coding.\u00a0 Traditionally, source code written by students would be collected through some mechanism or another<\/a>, be marked by TAs, and then be returned to students after a few weeks.<\/p>\n

    What if, after all of the submissions have been collected, each student must anonymously grade 5 submissions, chosen randomly from the system (with the only stipulation that students cannot grade their own work).<\/p>\n

    But here’s the twist:<\/p>\n

    Instead of just calculating a mark for students based on the peer reviews that they get<\/em>, how about we mark the students based on the reviews that they give<\/em> – specifically, based on how close they are to generating the same marks that the TAs give?<\/p>\n

    So now a students mark will be partially based on how well they are able to review code.<\/p>\n

    Questions \/ Answers (or Concerns \/ Freebies)<\/h4>\n

    I can think of a few initial concerns with this idea.<\/p>\n

    Q:<\/strong> What if the TA makes a huge mistake, or makes an oversight?\u00a0 They’re not infallible.\u00a0 How can students possibly make the same mistake \/ give the same mark?<\/p>\n

    A:<\/strong> I agree that TAs are not infallible.\u00a0 Nobody is.\u00a0 However, if a TA gives a submission a 3\/10, and the rest of the students give 9\/10’s, this is useful information.\u00a0 It either means that the TA missed something, or might signal that the students in general have not learned something crucial.\u00a0 In either case, this sort of problem can be easily detected, and sorted out via human intervention.<\/p>\n

    Q: <\/strong> What if students game the system by just giving their peers all 10\/10’s, or try to screw each other by just giving 0\/10’s?<\/p>\n

    A:<\/strong> Remember, students are being marked on their ability to review.\u00a0 If the TAs gave a more appropriate mark, and a student starts behaving as above, they’re going to get a poor reviewing mark.\u00a0 No harm done to the reviewee.<\/p>\n

    Q:<\/strong> I’m already swamped.\u00a0 How can I cram a system like this into my course?<\/p>\n

    A:<\/strong> I’m one of the developers on MarkUs<\/a>, a tool that is being used to grade source code for students at the University of Toronto and the University of Waterloo.\u00a0 It would not be impossible to adapt MarkUs to follow this model.\u00a0 Through MarkUs, a lot of this idea can be automated.\u00a0 Besides some possible human intervention for edge cases, I don’t see there being a whole lot of course-admin overhead to get this sort of thing going.\u00a0 But it does<\/em> mean a little bit more work for students who have to review the code.<\/p>\n

    Q:<\/strong> This is nice in theory, but is there any real pedagogical<\/em> value in this?\u00a0 And if so, how can I show it to my students?<\/p>\n

    A: <\/strong>First off, as a recent undergraduate student at UofT, I must say how rare it is to be given the opportunity to read another student’s code<\/a>.\u00a0 It just doesn’t happen much.\u00a0 I would have found it interesting – I’d be able to see the techniques that my peers employed to solve the same problems that I was trying to solve.\u00a0 It would give me a good informal measuring stick to see how I rank in the class – and students always want to know how they rank in the class<\/strong>.<\/p>\n

    Would they learn anything from it though?<\/p>\n

    That’s a good question.\u00a0 Would students learn anything from this, and realize the benefits?\u00a0 Remember – that’s what Steve Easterbrook says was the major stumbling block to introducing peer review…we have to show them that it’s useful.<\/p>\n

    The Questions<\/h4>\n