Mike Conley's Blog

A major part of my Master’s degree requirements was my research paper.  If you heard me lament over the past year or so about my “thesis”, I was referring to this research paper.

Anyhow, after lots of hard work, my research paper was finally signed off by my supervisor, Dr. Greg Wilson, and second reader Dr. Yuri Takhteyev.  A huge thanks to both of them!

Here’s the abstract, followed by a download link for the PDF.  Enjoy!

Abstract

Peer code review is commonly used in the software development industry to identify and fix problems during the development process. An additional benefit is that it seems to help spread knowledge and expertise around the team conducting the review. So is it possible to leverage peer code review as a learning tool? Our experiment results show that peer code review seems to cause a performance boost in students. They also show that the average total peer mark generated by students seems to be similar to the total mark that a graduate-level teaching assistant might give. We found that students agree that peer code review teaches them something – however, we also found they do not enjoy grading their peers’ work. We are encouraged by these results, and feel that they are a strong motive for further research in this area.

Click here to download my research paper

5 people like this post.

Like

Unlike

While the benefits of code review are proven, documented, numerous and awesome, it doesn’t change the fact that most people, in general, don’t like doing it.

I guess code review just isn’t really all that fun.

So a few months ago, I broadcast the idea of turning code review into a game. It was my way of trying to mix things up – “let’s add points, and have reviewers/developers competing to be the best participant in the code review process”.

Well, if there’s one thing that my supervisor Greg has taught me, it’s how I shouldn’t rush headlong into something before all of the facts are in.  So before I decide to do something like game-ifize code review, I should take a look at some prior work in the area…

Enter this guy:  Sebastian Deterding.

In particular, check out the following slide-show.  Flip through it if you have the time.  If you don’t have the time, scroll down, where I get to the salient point with respect to game-ificating code review.

Here’s the slide-show. Be sure to read the narrative at the bottom.

The Salient Point

Sebastian seems to be saying that adding points to apps and trying to incite competition does not make something a game.  If it did, then this should be countless hours of fun.

Without play, there is no game. Points do not equal a game.  It’s not nearly that simple.

Free Pizza and Pop

I’m going to divert for a second here.

Last week, a company set themselves up a couple of booths in the lobby of the Bahen Center where I work.  They were there to recruit university students to work for their company – either as interns, or full-timers.

They were also handing out free pizza and pop.

Needless to say, I wanted a few slices – but I figured it would be polite if I engaged them in conversation before waltzing off with some of the free food and drink they’d brought.

So I sparked up a conversation with one of the recruiters, and he told me about the company.  I’m going to call this recruiter Vlad.

I ended up gently steering the conversation towards code review, and I asked my inevitable question:

“So, do you guys do code review?”

I felt like a dentist asking a patient if he’s been flossing.  Vlad waffled a bit, but the general impression was:

“Not as much as we should.  We don’t have a prescribed workflow. It’d be hard to persuade all of the teams to do it.”

And then we started talking about code review in general.  It turns out that Vlad had worked in a few companies where they’d done code review, and he always felt a little short changed.  He said something along the lines of:

“I never felt compelled to do reviews.  They just sort of happened…and I did it, and it felt like…unrecognized effort.  I mean, what’s the incentive?  Do you know what I mean?  There’s incentive for the software, but I’m talking incentive for me.  And some people did really lousy reviews…but my reviews were treated the same as theirs.  I didn’t get recognized, and didn’t get rewarded if I did a good review.  So it was hard for me to do them.  I want to be recognized for my good reviews, for my good contributions.”

I wish I’d had a tape-recorder running so I could have gotten Vlad’s exact words.  But that’s what I remember him saying.

Feedback and Recognition

Maybe instead of trying to game-ulize code review, I can instead hear what Vlad is saying and work off of that.

With the code review that Vlad participated in, all of the feedback went to the code author, and none went to the reviewers.  And the reviewers are the ones who are doing all of the heavy lifting!  As a reviewer, Vlad also wants feedback, and recognition for code review done well.

There’s a company in Toronto that specializes in feedback like this.  They’re one of the major players in the Toronto start-up scene, and have built a pretty sweet suite of tools to facilitate quick and easy feedback/recognition.

The company is called Rypple.  And maybe that’s the name of the application, too.  (checks website) Yeah, it’s both.

So Rypple has this feature called Kudos that let’s people publicly acknowledge the good work of their team.

Normally, I don’t pimp companies.  And it upsets me when people comment on my blog, and their sub-text is to try to sell their product or service.  However, I think this video is relevant, so I’m posting their demo video so you can see how Kudos work:

Click here if you can’t see the video.

The Idea

So Rypple’s idea is to have a feed that the team subscribes to, and publicly display things like Kudos.  The badges for the Kudos are also limited in how many you can give per week, so they’re a valuable commodity that can’t just be handed out all over the place.  Cool idea.

So there’s one approach – use a service like Rypple to give your reviewers better feedback and recognition.

Or maybe we could build an extension for Review Board that does something similar, and more oriented around code review.

It’s not oriented like a game, like I had originally envisioned.  But somehow, I think this idea has more meaning and traction than just “adding points”.

More on this idea in a few days.  But please, comment if you have any thoughts or ideas to add.

4 people like this post.

Like

Unlike

If you’ve read about my experiment, you’ll know that there were two Python programming assignments that my participants worked on, and a rubric for each assignment.

There were also 5 mock-up submissions for each assignment that I had my participants grade.  I developed these mock-ups, after a few consultations with some of our undergraduate instructors, in order to get a sense of the kind of code that undergraduate programmers tend to submit.

I’ve decided to post these materials to this blog, in case somebody wants to give them a once over.  Just thought I’d open my science up a little bit.

So here they are:

Flights and Passengers

• Assignment Specification
• Marking Rubric
• Starter Code
• Mock-ups

Cards and Decks

• Assignment Specification
• Marking Rubric
• Starter Code
• Mock-ups

Peruse at your leisure.

Be the first to like.

Like

Unlike

Before I start diving into results, I’m just going to recap my experiment so we’re all up to speed.

I’ll try to keep it short, sweet, and punchy – but remember, this is a couple of months of work right here.

Ready?  Here we go.

What I was looking for

A quick refresher on what code review is

Code review is like the software industry equivalent of a taste test.  A developer makes a change to a piece of software, puts that change up for review, and a few reviewers take a look at that change to make sure it’s up to snuff.  If some issues are found during the course of the review, the developer can go back and make revisions.  Once the reviewers give it the thumbs up, the change is put into the software.

That’s an oversimplified description of code review,  but it’ll do for now.

So what?

What’s important is to know that it works. Jason Cohen showed that code review reduces the number of defects that enter the final software product. That’s great!

But there are some other cool advantages to doing code review as well.

1. It helps to train up new hires.  They can lurk during reviews to see how more experienced developers look at the code.  They get to see what’s happening in other parts of the software.  They get their code reviewed, which means direct, applicable feedback.  All good things.
2. It helps to clean and homogenize the code.  Since the code will be seen by their peers, developers are generally compelled to not put up “embarrassing” code (or, if they do, to at least try to explain why they did).  Code review is a great way to compel developers to keep their code readable and consistent.
3. It helps to spread knowledge and good practices around the team.  New hires aren’t the only ones to benefit from code reviews.  There’s always something you can learn from another developer, and code review is where that will happen.  And I believe this is true not just for those who receive the reviews, but also for those who perform the reviews.

That last one is important.  Code review sounds like an excellent teaching tool.

So why isn’t code review part of the standard undergraduate computer science education?  Greg and I hypothesized that the reason that code review isn’t taught is because we don’t know how to teach it.

I’ll quote myself:

What if peer code review isn’t taught in undergraduate courses because we just don’t know how to teach it?  We don’t know how to fit it in to a curriculum that’s already packed to the brim.  We don’t know how to get students to take it seriously.  We don’t know if there’s pedagogical value, let alone how to show such value to the students.

The idea

Inspired by work by Joordens and Pare, Greg and I developed an approach to teaching code review that integrates itself nicely into the current curriculum.

Here’s the basic idea:

Suppose we have a computer programming class.  Also suppose that after each assignment, each student is randomly presented with anonymized assignment submissions from some of their peers.  Students will then be asked to anonymously peer grade these assignment submissions.

Now, before you go howling your head off about the inadequacy / incompetence of student markers, or the PeerScholar debacle, read this next paragraph, because there’s a twist.

The assignment submissions will still be marked by TA’s as usual.  The grades that a student receives from her peers will not directly affect her mark.  Instead, the student is graded based on how well they graded their peers. The peer reviews that a student completes will be compared with the grades that the TA’s delivered.  The closer a student is to the TA, the better the mark they get on their “peer grading” component (which is distinct from the mark they receive for their programming assignment).

Now, granted, the idea still needs some fleshing out, but already, we’ve got some questions that need answering:

1. Joordens and Pare showed that for short written assignments, you need about 5 peer reviews to predict the mark that the TA will give.  Is this also true for computer programming assignments?
2. Grading students based on how much their peer grading matches TA grading assumes that the TA is an infallible point of reference.  How often to TA’s disagree amongst themselves?
3. Would peer grading like this actually make students better programmers?  Is there a significant difference in the quality of their programming after they perform the grading?
4. What would students think of peer grading computer programming assignments?  How would they feel about it?

So those were my questions.

How I went about looking for the answers

Here’s the design of the experiment in a nutshell:

Writing phase

I have a treatment group, and a control group.  Both groups are composed of undergraduate students.  After writing a short pre-experiment questionnaire, participants in both groups will have half an hour to work on a short programming assignment.  The treatment group will then have another half an hour to peer grade some submissions for the assignment they just wrote.  The submissions that they mark will be mocked up by me, and will be the same for each participant in the treatment group.  The control group will not perform any grading – instead, they will do an unrelated vocabulary exercise for the same amount of time.  Then, participants in either group will have another half an hour to work on the second short programming assignment. Participants in my treatment group will write a short post-experiment questionnaire to get their impressions on their peer grading experience.  Then the participants are released.

Here’s a picture to help you visualize what you just read.

So now I’ve got two piles of submissions – one for each assignment, 60 submissions in total.  I add my mock-ups to each pile.  That means 35 submissions in each pile, and 70 submissions in total.

Marking phase

I assign ID numbers to each submission, shuffle them up, and hand them off to some graduate level TA’s that I hired.  The TA’s will grade each assignment using the same marking rubric that the treatment group used to peer grade.  They will not know if they are grading a treatment group submission, a control group submission, or a mock-up.

Choosing phase

After the grading is completed, I remove the mock-ups, and pair up submissions in both piles based on who wrote it.  So now I’ve got 30 pairs of submissions:  one for each student.  I then ask my graders to look at each pair, knowing that they’re both written by the same student, and to choose which one they think is better coded, and to rate and describe the difference (if any) between the two.  This is an attempt to catch possible improvements in the treatment group’s code that might not be captured in the marking rubric.

So that’s what I did

So everything you’ve just read is what I’ve just finished doing.

Once the submissions are marked, I’ll analyze the marks for the following:

1. Comparing the two groups, is there any significant improvement in the marks from the first assignment to the second in the treatment group?
   1. If there was an improvement, on which criteria?  And how much of an improvement?
2. How did the students do at grading my mock-ups?  How similar were their peer grades to what the TAs gave?
3. How much did my two graders agree with one another?
4. During the choosing phase, did my graders tend to choose the second assignment over the first assignment more often for the treatment group?

And I’ll also analyze the post-experiment questionnaire to get student feedback on their grading experience.

Ok, so that’s where I’m at.  Stay tuned for results.

2 people like this post.

Like

Unlike

I just spent the long weekend in Ottawa and Québec City with my parents and my girlfriend Em.

During the long drive back to Toronto from Québec City, I had plenty of time to think about my GSoC project, and where I want to go with it once GSoC is done.

Here’s what I came up with.

Detach Reviewing Time from Statistics

I think it’s a safe assumption that my reviewing-time extension isn’t going to be the only one to generate useful statistical data.

So why not give extension developers an easy mechanism to display statistical data for their extension?

First, I’m going to extract the reviewing-time recording portion of the extension. Then, RB-Stats (or whatever I end up calling it), will introduce it’s own set of hooks for other extensions to register with.  This way, if users want some stats, there will be one place to go to get them.  And if an extension developer wants to make some statistics available, a lot of the hard work will already be done for them.

And if an extension has the capability of combining its data with another extensions data to create a new statistic, we’ll let RB-Stats manage all of that business.

Stopwatch

The reviewing-time feature of RB-Stats will become an extension on its own, and register its data with RB-Stats.  Once RB-Stats and Stopwatch are done, we should be feature equivalent with my demo.

Review Karma

I kind of breezed past this in my demo, but I’m interested in displaying “review karma”.  Review karma is the reviews/review-requests ratio.

But I’m not sure karma is the right word.  It suggests that a low ratio (many review requests, few reviews) is a bad thing.  I’m not so sure that’s true.

Still, I wonder what the impact will be to display review karma?  Not just in the RB-Stats statistics view, but next to user names?  Will there be an impact on review activity when we display this “reputation” value?

FixIt

This is a big one.

Most code review tools allow reviewers to register “defects”, “todos” or “problems” with the code up for review.  This makes it easier for reviewees to keep track of things to fix, and things that have already been taken care of.  It’s also useful in that it helps generate interesting statistics like defect density and defect detection rate (assuming Stopwatch is installed and enabled).

I’m going to tackle this extension as soon as RB-Stats, Stopwatch and Karma are done.  At this point, I’m quite confident that the current extension framework can more or less handle this.

Got any more ideas for me?  Or maybe an extension wish-list?  Let  me know.

Be the first to like.

Like

Unlike

I’ve done it again:  I’ve let dust gather on my blog.

Quick update:

• I’ve finished my courses for this semester, and have gone into full-blown research mode.
• My research proposal is going through ethics review, in order to make sure that I’m not going to blow things up (or hurt anybody if I do)
• While my paperwork is reviewed, I’m refining my procedure and apparatus.  Better and better.
• I’ve been accepted into Google Summer of Code this year – I’ll be working on Review Board.  Details about my project will be the subject of an upcoming post, which I will toss up shortly.
• I may or may not be co-directing a radio play.  I’ll let you know.
• The MarkUs team is about to release version 0.7, and a fresh batch of Summer students will soon be here at UofT to work on it!
• I have not forgotten about the UCDP trip to Poland.  I still have to tell you what we saw and did at Auschwitz.  Cripes – it’s almost a year since I returned, and I’m only half-way through the whole story.  And there’s a ton more to tell.  Coming soon.

Stay tuned.

Be the first to like.

Like

Unlike

The Question

My past few blog posts have been concerned with the usefulness of peer grading.  Steve Joordens showed that peer grading was pedagogically useful for first-year psych students…but what about computer science students?  Would they learn from it?  Would they become better programmers?

We don’t know.

Maybe it’s time to find out.

The Experiment

It’s pretty simple, actually.

I have two groups of students.  Let’s call them groups A and B.

For each student in A, have them complete a simple programming assignment (call it P1).  Once they’re finished, have them complete a second simple programming assignment (call it P2).

For each student in B, have them complete P1.  Once that’s done, have them view 5 or 6 different mocked up submissions, also for P1.  For each submission, have the students fill out a rubric and assign a grade. Once finished, the students then complete P2.

Then, I get some fellow graduate students to mark my mocked up submissions, the group A P1/P2 submissions, and the group B P1/P2 submissions.

If grading made the students better programmers, we should see an increase in the number of marks given to the students in group B for P2.

Bonuses, and Other Concerns

This experiment is nice and simple. And, besides showing if peer grading makes students better programmers, it gives us a couple of bonuses:

• It tells us if graduate students tend to agree on what marks to give to submissions.  If they don’t agree, and the marks wildly differ…we might have a problem
• It tells us if some number of students can, on average, approximate the grade a TA would give on a submission
• It can tell us the average inspection rate for both students and TAs

I’ll have to do randomization here and there to eliminate ordering effects – for example, randomizing the criteria on the rubric, randomizing which assignments go first and second, randomizing the order in which the mock-up submissions are shown, etc.

One thing to consider though:  what effect does simply seeing the rubric have on students?

I’ve been in courses where I’ve not been allowed to see the marking rubric for some assignment.  It’s frustrating.  Seeing the rubric helps me focus on the areas that I’ll be marked on.

So what if just seeing the rubric makes the students “better programmers”?  One way to counteract this would be to have the rubric for the second assignment be quite a bit different than the one for the first assignment.

Statistics

Oh yeah.  Stats.  Not my strongest subject.  I’m going to have to brush up on this (and probably enlist some help within the department) if I’m going to do this properly.  I’m probably not going to get as many participants as I think I will…so I have to accommodate small N.  Hrm.

Anyhow, this is where my summer experiment seems to heading.  What do you think?  I’m all ears.

Be the first to like.

Like

Unlike

I was really happy to see all of the response to my last blog post. Lots of great ideas and suggestions.  Thanks all!

One of the problems that was brought up with my original idea for teaching code review was that it punishes students twice if they didn’t understand a programming concept.

For example, if a student does not understand what pipes are for and how they work, they’re probably going to do pretty poorly on their pipes assignment in a systems intro course.  So there’s one slam for the student.

The second time is when they review their peer’s code.  If they still don’t understand how pipes work, their reviews are going to be pretty trashy.  And they’ll get a poor mark for that.  And that’s the second slam.

The problem here is that the students don’t get any feedback before they go into the peer review process.  For the “weaker” students, this essentially means bringing a knife to a tank fight.

So here’s an idea:

1. After an assignment due date passes, and the students have submitted their code, the students are randomly placed into groups of 3 or 4.
2. Each group is assigned a single random submission from the ones collected from the students
3. Each student in their group individually, and privately, performs a review on their assigned submission.  They fill out a rubric, make comments, etc.  They are not allowed to interact with the other members of their group.
4. After the students have finished their review, they can converse with their other group members.  The group must produce another review – but this one is by consensus.  They must work together to find the most appropriate mark.
5. Finally, after the consensus reviews are in, the groups are disbanded.  Students are then shown their own code submissions.  They must do a final review on their own code by filling in the marking rubric.
6. Student’s marks will be based on:
   1. The mark that the TA gave them
   2. How closely their individual review of the group submission agrees with the TAs assessment
   3. How close the consensus review of the group submission agrees with the TAs assessment
   4. How close the review of their own code agrees with the TAs assessment

From my viewpoint, this model has several obvious strengths and weaknesses.

One major strength is that, even if students do poorly in the coding portion of their assignment, they might still have an opportunity to make it up by learning from their peers during the group consensus review.  They’ll also have an opportunity to demonstrate their new-found understanding by reviewing their own code, and admitting the shortcomings.

A major weakness of the idea is the sheer organizational complexity.  Did you see how many steps there are?  That’s a lot of work.

Plus, the model makes some pretty wild assumptions.  A few off the top of my head:

• It assumes students can actually learn by performing peer review together, independent of what piece of code they’re reviewing
• It assumes students will actually reach a final consensus during the group review.  What about bullies?  What about timid folks?

There are probably more that I’m not seeing yet.

Anyhow, this was an idea I had a few days ago, and I just wanted to write it down.

Be the first to like.

Like

Unlike

We Don’t Know How To Teach Code Review

If you go to my very first blog post about code review, you’ll discover what my original research question was:

Code reviews. They can help make our software better. But how come I didn’t learn about them, or perform them in my undergrad courses?  Why aren’t they taught as part of the software engineering lifecycle right from the get-go?  I learn about version control, but why not peer code review?  Has it been tried in the academic setting?  If so, why hasn’t it succeeded and become part of the general CS curriculum?  If it hasn’t been tried, why not?  What’s the hold up?  What’s the problem?

I have mulled the question for months, and read several papers that discuss different models for introducing code review into the classroom.

But I’m no teacher.  I really don’t know what it’s like to run a university level course.  Thankfully, two course instructors from our department gave their input on the difficulty of introducing peer code review in the classroom.  Here’s the first:

The problem is that is completely un-assessable. You can’t get the students to hand in reports from their inspection, and grade them on it, because they quickly realise it’s easier to fake their reports than it is to do a real code inspection. And the assignment never gets them to understand and internalize the real reasons for doing code inspection – here they just do it to jump through an artificial hoop set by the course instructor.

What we really need to do is to assess code quality, and let them figure out for themselves how the various tools we show them (e.g. test-case first, code inspection, etc) will help them achieve that quality. Better still, we give them ways of measuring directly how the various tools they use affect code quality for each assignment. But I haven’t thought enough yet about how to achieve this.

So, I’ve long since dropped the idea of a specific marked assignment on code inspections, but still teach inspection in all of my SE courses. I need to find a way to teach it so that the students themselves understand why it’s so useful.

(From Steve Easterbrook, commenting on this post)

And here’s the second:

1. How many different tasks can we ask students to do on a 3-week assignment? I think students should learn to use an IDE, a debugger, version control, and a ticket system. We have been successful in getting students to use version control because that’s the only way they can submit an assignment. We have had mixed success getting students to use IDE’s and debuggers, partly because it is hard to assign marks for their use. We have been even less successful in convincing students to use tickets because a 3-week assignment isn’t big enough or long enough to make tickets essential.

2. If the focus of my course is teaching operating systems, how much time (and grades) should I devote to software development tools and practices that aren’t centered on operating systems?

(From Karen Reid, commenting on this post)

All of this swirls around a possible answer that Greg Wilson and I have been approaching since September:

What if peer code review isn’t taught in undergraduate courses because we just don’t know how to teach it?  We don’t know how to fit it in to a curriculum that’s already packed to the brim.  We don’t know how to get students to take it seriously.  We don’t know if there’s pedagogical value, let alone how to show such value to the students.

If that’s really the problem… Greg and I may have come up with a possible solution.

But First, Some Background

In 2008, Steve Joordens and Dwayne Pare published Peering into Large Lectures:  Examining Peer and Expert Mark Agreement Using peerScholar, an Online Peer Assessment Tool.

It’s a good read, but in the interests of brevity, I’ll break it down for you:

1. Joordens and Pare are both at the University of Toronto Scarborough, in the Psych Department
2. Psych classes (especially for the first year) are large.  For large classes, it is generally difficult to introduce writing assignments simply due to the sheer volume of writing that would need to be marked by the TAs.  Alternatives (like multiple-choice tests) are often used to counteract this.
3. But writing is important.
4. The idea:  what if we let students grade one another?  There’s research showing the benefits of peer evaluation for writing assignments.  So lets see what kind of grades peers give to one another.
5. A tool is built (peerScholar), and an experiment is run:  after submitting their writing assignments, show students 5 submissions from other students, and have them grade the work (with specific grading instructions from the instructor).  Then, compare the grades that the students gave with grades from the TAs.
6. A significant positive correlation was found between averaged TA marks and average peer marks.  More statistical analysis shows that there is no significant difference between the agreement levels of TA and peer markers.
7. To ensure repeatability, a second experiment is run – similar to the first.  Except, this time, students who receive the marks from their peers are able to “mark the marker” and flag any marks that seem suspicious (a 1/10, for example, if all the other students and the TA gave something closer to a 7/10).
8. It looks good – numbers were closer this time.
9. Conclusion:  the average grade grade given by a set of peer markers was similar to the grade given by the TAs in terms of overall level and rank ordering of assignments.

This is a very interesting result.  Why can’t we apply it to courses in a computer science department?  What if students started marking each others code?

What they’d be doing would be called code review.

The Idea

Let’s modify Joorden and Pare’s model a little bit.

Let’s say I’m teaching an undergraduate computer science course where students tend to do quite a bit of coding.  Traditionally, source code written by students would be collected through some mechanism or another, be marked by TAs, and then be returned to students after a few weeks.

What if, after all of the submissions have been collected, each student must anonymously grade 5 submissions, chosen randomly from the system (with the only stipulation that students cannot grade their own work).

But here’s the twist:

Instead of just calculating a mark for students based on the peer reviews that they get, how about we mark the students based on the reviews that they give – specifically, based on how close they are to generating the same marks that the TAs give?

So now a students mark will be partially based on how well they are able to review code.

Questions / Answers (or Concerns / Freebies)

I can think of a few initial concerns with this idea.

Q: What if the TA makes a huge mistake, or makes an oversight?  They’re not infallible.  How can students possibly make the same mistake / give the same mark?

A: I agree that TAs are not infallible.  Nobody is.  However, if a TA gives a submission a 3/10, and the rest of the students give 9/10′s, this is useful information.  It either means that the TA missed something, or might signal that the students in general have not learned something crucial.  In either case, this sort of problem can be easily detected, and sorted out via human intervention.

Q: What if students game the system by just giving their peers all 10/10′s, or try to screw each other by just giving 0/10′s?

A: Remember, students are being marked on their ability to review.  If the TAs gave a more appropriate mark, and a student starts behaving as above, they’re going to get a poor reviewing mark.  No harm done to the reviewee.

Q: I’m already swamped.  How can I cram a system like this into my course?

A: I’m one of the developers on MarkUs, a tool that is being used to grade source code for students at the University of Toronto and the University of Waterloo.  It would not be impossible to adapt MarkUs to follow this model.  Through MarkUs, a lot of this idea can be automated.  Besides some possible human intervention for edge cases, I don’t see there being a whole lot of course-admin overhead to get this sort of thing going.  But it does mean a little bit more work for students who have to review the code.

Q: This is nice in theory, but is there any real pedagogical value in this?  And if so, how can I show it to my students?

A: First off, as a recent undergraduate student at UofT, I must say how rare it is to be given the opportunity to read another student’s code.  It just doesn’t happen much.  I would have found it interesting – I’d be able to see the techniques that my peers employed to solve the same problems that I was trying to solve.  It would give me a good informal measuring stick to see how I rank in the class – and students always want to know how they rank in the class.

Would they learn anything from it though?

That’s a good question.  Would students learn anything from this, and realize the benefits?  Remember – that’s what Steve Easterbrook says was the major stumbling block to introducing peer review…we have to show them that it’s useful.

The Questions

• How good are students at grading their peers?  How close to they get to the grades that a TA would give?
  • By study year
  • By their perceived programming ability
  • By their perceived programming experience
  • By their programming confidence
• What happens to students’ ability to review their peers as they perform each review?  Do they get better after each one?  And is there a point where their accuracy gets poorer from fatigue?
• How many student reviewers are needed to approximate the grade that a TA would give?
• How long do students generally take to peer review code? (bonus)
• How long do graduate students generally take to mark an assignment? (bonus)
• Do the students actually learn anything from the process?
• How do the students feel about being graded on their ability to review?
  • Do they think that this process is fair?
  • Do they think that they’re learning anything useful?
  • Do they feel like it is worth their time?
  • Do they enjoy reading other students’ code?
  • If it was introduced into their classes, how would they feel?

Lots of questions.  Luckily, it just so happens that I’m a scientist.

The Experiment

First, I mock up (or procure) 10 submissions for a programming assignment that our undergraduates might write.

I then get/convince some graduate students to grade those 10 submissions to the best of their ability, using MarkUs.  These marks are recorded.

I then take a cross-section of our undergraduate student body, and (after a brief survey to determine their opinions of their coding experience/confidence), I get the students to peer review and grade those 10 submissions.  They will be told that their goal is to try to give the same type of marks that a graduate student TA might give.

After the grades are recorded, I take the submission that they reviewed first, and get them to grade it again.  Do they get closer to the TAs mark than their first attempt?

Students are then given a second survey (probably Likert scales) to assess their opinions on the process.  Would it be fair if their ability to grade was part of their mark?  Did you get anything useful out of this?  Did you feel that it was worth your time?  Did you enjoy reading other students’ code?  How would you feel if it was part of your class?  …

The final survey will (hopefully) knock out the last series of questions in my list.  Timing information recorded during marking will help answer the bonus questions.  Analysis of the marks that the students give in relation to the marks that the TA give will hopefully help answer the rest.

What Am I Missing?

Am I missing anything here?  Is there a gaping hole in my thinking somewhere?  Would this be a good, interesting experiment to run?  For those who teach…if my results are encouraging, would you ever try implementing this in your classroom?

And if this was introduced into the classroom…what would happen to student learning?  What would happen to marks?  How would instructors like it?

So, what do you think?  I’m all ears.

2 people like this post.

Like

Unlike

A little while back, I wrote about an idea that a few of us had been bouncing around:  peer code review achievements.

It started out as a bit of Twitter fun – but now it has evolved, and actually become a contender for my Masters research.

So I’ve been reading up on reputation and achievement systems, and it’s been keeping me up at night.  I’ve been tossing and turning, trying to figure out a way of applying these concepts to something like ReviewBoard.  Is there a model that will encourage users to post review requests early and often?  Is there a model that will encourage more thorough reviews from other developers?

An idea eventually sprung to mind…

Idea 1:  2 Week Games

In a speech he gave a few years ago, Danc of The Last Garden placed the following bet:

• If an activity can be learned…
• If the player’s performance can be measured…
• If the player can be rewarded or punished in a timely fashion…
• Then any activity that meets these criteria can be turned into a game.

Let’s work off of this premise.

Modeled on the idea of a sprint or iteration, let’s say that ReviewBoard has “games” that last 2 weeks.

In a game, users score points in the following way:

• Posting a review request that eventually gets committed gives the author 1 point
• A review request that is given a ship-it, without a single defect found, gives the author The 1.5 Multiplier on their total points.  The 1.5 Multiplier can be stolen by another player if they post a review request that also gets a ship-it without any defects being found.
• Any user can find/file defects on a review request
• A defect must be “confirmed” by the author, or “withdrawn” by the defect-finder.
• After a diff has been updated, “confirmed” defects can be “fixed”.  Each fixed defect gives the defect-finder and author 1 point each.

After two weeks, a final tally is made, achievements / badges are doled out, and the scores are reset.  A new game begins.  Users can view their point history and track their performance over time.

Granted, this game is open to cheating.  But so is Monopoly.  I can reach into the Monopoly bank and grab $500 without anybody noticing.  It’s up to me not to do that, because it invalidates the game.  In this case, cheating would only result in bad morale and a poorer piece of software.  And since scores are reset every two weeks, what’s the real incentive to cheat?

Idea 2:  Track My Performance

I’ve never built a reputation system before – but Randy Farmer and Bryce Glass have.  They’ve even written a book about it.

Just browsing through their site, I’m finding quotes that suggests that there are some potential problems with my two week game idea.  In particular, I have not considered the potentially harmful effects of displaying “points” publicly on a leader-board.

According to Farmer / Glass:

It’s still too early to speak in absolutes about the design of social-media sites, but one fact is becoming abundantly clear: ranking the members of your community-and pitting them one-against-the-other in a competitive fashion-is typically a bad idea. Like the fabled djinni of yore, leaderboards on your site promise riches (comparisons! incentives! user engagement!!) but often lead to undesired consequences.

They go into more detail here.

Ok, so let’s say that they’re right.  Then how about instead of pitting the reviewers against one another, I have the reviewers compete against themselves?

Ever played Wii Sports?  It tracks player performance on various games and displays it on a chart.  It’s really easy to see / track progress over time.  It’s also an incentive to keep performance up – because nobody wants to go below the “Pro” line.

So how about we just show users a report of their performance over fixed time intervals…with fancy jQuery charts, etc?

So what?

Are either of these ideas useful?  Would they increase the number of defects found per review request?  Would they increase the frequency and speed of reviews?  Would they improve user perception of peer code review?  Would it be ignored?  Or could it harm a team of developers?  What are the benefits and drawbacks?

If anything, it’d give ReviewBoard some ability to record metrics, which is handy when you want to show the big boss how much money you’re saving with code review.

Might be worth looking into.  Thoughts?

Be the first to like.

Like

Unlike