But there are some other cool advantages to doing code review as well.<\/p>\n
- \n
- It helps to train up new hires.\u00a0 They can lurk during reviews to see how more experienced developers look at the code.\u00a0 They get to see what’s happening in other parts of the software.\u00a0 They get their code reviewed, which means direct, applicable feedback.\u00a0 All good things.<\/li>\n
- It helps to clean and homogenize the code.\u00a0 Since the code will be seen by their peers, developers are generally compelled to not put up “embarrassing” code (or, if they do, to at least try to explain why they did).\u00a0 Code review is a great way to compel developers to keep their code readable and consistent.<\/li>\n
- It helps to spread knowledge and good practices around the team.\u00a0 New hires aren’t the only ones to benefit from code reviews.\u00a0 There’s always something you can learn from another developer, and code review is where that will happen.\u00a0 And I believe this is true not just for those who receive the reviews, but also for those who perform the reviews.<\/li>\n<\/ol>\n
That last one is important.\u00a0 Code review sounds like an excellent teaching tool<\/strong>.<\/p>\n
So why isn’t code review part of the standard undergraduate computer science education?\u00a0 Greg and I hypothesized that the reason that code review isn’t taught is because we don’t know how to teach it<\/a>.<\/p>\n
I’ll quote myself:<\/p>\n
What if peer code review isn\u2019t taught in undergraduate courses because we just don\u2019t know how to teach it?\u00a0 We don\u2019t know how to fit it in to a curriculum that\u2019s already packed to the brim.\u00a0 We don\u2019t know how to get students to take it seriously.\u00a0 We don\u2019t know if there\u2019s pedagogical value, let alone how to show such value to the students.<\/p><\/blockquote>\n
The idea<\/h4>\n
Inspired by work by Joordens and Pare<\/a>, Greg and I developed an approach to teaching code review that integrates itself nicely into the current curriculum.<\/p>\n
Here’s the basic idea:<\/p>\n
Suppose we have a computer programming class.\u00a0 Also suppose that after each assignment, each student is randomly presented with anonymized assignment submissions from some of their peers.\u00a0 Students will then be asked to anonymously peer grade these assignment submissions.<\/p>\n
Now, before you go howling your head off about the inadequacy \/ incompetence of student markers, or the PeerScholar debacle<\/a>, read this next paragraph, because there’s a twist.<\/p>\n
The assignment submissions will still be marked by TA’s as usual.\u00a0 The grades that a student receives from her peers will not directly affect her mark.\u00a0 Instead, the student is graded based on how well they graded their peers.<\/strong> The peer reviews that a student completes will be compared with the grades that the TA’s delivered.\u00a0 The closer a student is to the TA, the better the mark they get on their “peer grading” component (which is distinct from the mark they receive for their programming assignment).<\/p>\n
Now, granted, the idea still needs some fleshing out, but already, we’ve got some questions that need answering:<\/p>\n
- \n
- Joordens and Pare showed that for short written assignments, you need about 5 peer reviews to predict the mark that the TA will give.\u00a0 Is this also true for computer programming assignments?<\/li>\n
- Grading students based on how much their peer grading matches TA grading assumes that the TA is an infallible point of reference.\u00a0 How often to TA’s disagree amongst themselves?<\/li>\n
- Would peer grading like this actually make students better programmers?\u00a0 Is there a significant difference in the quality of their programming after they perform the grading?<\/li>\n
- What would students think of peer grading computer programming assignments?\u00a0 How would they feel about it?<\/li>\n<\/ol>\n
So those were my questions.<\/p>\n
How I went about looking for the answers<\/h3>\n
Here’s the design of the experiment in a nutshell:<\/p>\n
Writing phase<\/h4>\n
I have a treatment group, and a control group.\u00a0 Both groups are composed of undergraduate students.\u00a0 After writing a short pre-experiment questionnaire, participants in both groups will have half an hour to work on a short programming assignment.\u00a0 The treatment group will then have another half an hour to peer grade some submissions for the assignment they just wrote.\u00a0 The submissions that they mark will be mocked up by me, and will be the same for each participant in the treatment group.\u00a0 The control group will not perform any grading – instead, they will do an unrelated vocabulary exercise for the same amount of time.\u00a0 Then, participants in either group will have another half an hour to work on the second short programming assignment. Participants in my treatment group will write a short post-experiment questionnaire to get their impressions on their peer grading experience.\u00a0 Then the participants are released.<\/p>\n
Here’s a picture to help you visualize what you just read.<\/p>\n
![\"Tasks](\"http:\/\/mikeconley.ca\/blog\/wp-content\/uploads\/2010\/08\/workflow.png\" "\\"Group")
<\/a><\/p>\nSo now I’ve got two piles of submissions – one for each assignment, 60 submissions in total.\u00a0 I add my mock-ups to each pile.\u00a0 That means 35 submissions in each pile, and 70 submissions in total.<\/p>\n
Marking phase<\/h4>\n
I assign ID numbers to each submission, shuffle them up, and hand them off to some graduate level TA’s that I hired.\u00a0 The TA’s will grade each assignment using the same marking rubric that the treatment group used to peer grade.\u00a0 They will not know if they are grading a treatment group submission, a control group submission, or a mock-up.<\/p>\n
Choosing phase<\/h4>\n
After the grading is completed, I remove the mock-ups, and pair up submissions in both piles based on who wrote it.\u00a0 So now I’ve got 30 pairs of submissions:\u00a0 one for each student.\u00a0 I then ask my graders to look at each pair, knowing that they’re both written by the same student, and to choose which one they think is better coded, and to rate and describe the difference (if any) between the two.\u00a0 This is an attempt to catch possible improvements in the treatment group’s code that might not be captured in the marking rubric.<\/p>\n
So that’s what I did<\/h3>\n
So everything you’ve just read is what I’ve just finished doing.<\/p>\n
Once the submissions are marked, I’ll analyze the marks for the following:<\/p>\n
- \n
- Comparing the two groups, is there any significant improvement in the marks from the first assignment to the second in the treatment group?\n
- \n
- If there was an improvement, on which criteria?\u00a0 And how much of an improvement?<\/li>\n<\/ol>\n<\/li>\n
- How did the students do at grading my mock-ups?\u00a0 How similar were their peer grades to what the TAs gave?<\/li>\n
- How much did my two graders agree with one another?<\/a><\/li>\n
- During the choosing phase, did my graders tend to choose the second assignment over the first assignment more often for the treatment group?<\/li>\n<\/ol>\n
- Comparing the two groups, is there any significant improvement in the marks from the first assignment to the second in the treatment group?\n