Thanks for the input!

I’m concerned that your approach only captures how useful students *think* reviews are. I’m also interested in how useful the reviews *actually* are to the students.

To that end, my experiment has undergone some serious redesign over the last few days. I’ll be posting about it soon!

Cheers,

-Mike

Why not do the following:

For each assignment have two problems – half the students have to solve one, the other half have to solve the other.
Have two deadlines. Deadline 1 – submit your assignment solution to 5 random reviewers (that were selected for you by the system).
Deadline 2 – incorporate feedback from review comments, modify your code, and submit assignment for grading by TA, as well as rating how useful each of the 5 reviews was to improving your code.

(Obviously, the students reviewing your code would be working on a different assignment problem and vica versa.)

This way, the person getting the reviews can rate their usefulness (which is much better than rating “how close is this to the TA’s grade”).

To avoid lots of drama, if you find additional flaws that the TAs did not,
the original person’s assignment mark is *not* lowered (you just get bonus
credit for your code review). However, if you convince people that the code
is not as flawed as the TAs think the original person’s assignment mark *is*
raised; however, other people who agreed with the TAs in their code review
get full credit (since one can hardly blame people for not being better
than the TAs). Both of these should be prominently mentioned in the appeal
process. If you do code review on your own assignment this should apply
to that too, just as with other people’s assignments. Yes, you can game
this last one a bit, but I’d rather encourage people to be honestly harsh
on their own code even at the expense of getting one past the TAs.

(This implies that the assignment and/or the code review guidelines should
clearly document things that the submitted programs do not need to do, so
that people don’t spend all day finding obscure things that the code they’re
reviewing doesn’t handle.)

This should apply primarily to code flaws, not to stylistic issues and
basic correctness (enough comments, tests exist, docstrings are sensible,
etc).

> Instead of a pre-built solution, would a standard marking rubric be
> enough? The rubric criteria would give, essentially, a “checklist”
> of things to watch out for. Contrived example: code has doc-strings
> for each class and method, over most, over some, over few, none,
> etc…

I think that students should get this sort of standard rubric (to the
extent that they didn’t get it at the start of class), but I think it’s
not enough. The kind of cheat sheet I’m thinking of is not covering
style or basics of well written code in general, it’s covering what I
called the dark corners of the assignment: the common subtle coding
mistakes that people make. For example, overlooking some particular race
condition in a concurrent program, or not handling out of range input,
or the like. These are always going to be specific to the assignment,
and you may not be able to figure them out in advance.

(Here I am assuming that the TAs are marking the assignment for more
than just code style, that they’re also marking for correct, complete,
and well written code.)

Also, this is probably obvious: if the assignment is run through
automated tests before being marked by the TAs, the code reviewers
should see the test results if the TAs do. Various failed tests are
quite likely a red flag to look in specific areas for code flaws (or
just knowledge that certain flaws have to exist), so both sides need the
same starting information.

Re: my idea of students reviewing their own assignments. In retrospect,
this is probably not workable for your experiment (unless you can sign
up a chunk of an eager class to do this work). However, it might work to
create a ringer assignment, one with specific subtle flaws, and then see
if student reviewers spot more of the flaws if they see the ringer at
the end of their code review then if it’s the first assignment that they
review. (Getting enough subjects for statistically valid results might
be tricky, since there are so many other variables in play here.)

> I think that if you do the code review straight, as proposed, you are
> effectively testing for two things at once; you’re testing both the
> ability of students to do good code reviews and their ability to fully
> understand the dark corners of the assignment. If I was a student, I
> would feel kind of grumpy about effectively getting hit twice if I
> didn’t fully understand the assignment; I’d lose marks on my solution
> and then I’d lose more marks on bad code reviews (because I would miss
> problems that the TAs would see).

Well put, and a very good point.

> My suggestion is that every student doing reviews should get a cheat
> sheet of ‘common mistakes to look for’. This shouldn’t cover everything
> and wouldn’t reduce code review to a mechanical process, but hopefully
> would get much closer to only measuring how well the students can review
> code and see things in it. The cheat sheet might be drawn up by the TAs
> after marking the assignments or it might be prepared in advance based
> on what the instructor is expecting people to have problems with.

Instead of a pre-built solution, would a standard marking rubric be enough? The
rubric criteria would give, essentially, a “checklist” of things to watch out for.
Contrived example: code has doc-strings for each class and method, over most,
over some, over few, none, etc…

> Also, it strikes me that it might be interesting to have students review
> their own assignment at the end of the review process. Having gone
> through a series of code reviews of other people’s code, have they
> learned enough to see the flaws in their own code? (My naive feeling
> is that this is not gameable, except to the extent that if you submit
> a terrible assignment you can at least get some marks back for being
> able to admit how terrible it is.)

This is a *very* interesting idea. Thank you! Keep your eyes peeled for another
blog post about this!

> It would be interesting to break your test subjects into three groups based on the
> initial TA markings then present the solutions in the following orders for review:
> * Random
> * Decreasing TA marks
> * Increasing TA marks

I agree, that would be interesting. IE – what are the learning effects (if any) of viewing
progressively better code? Worse code?

> I could see how reviewing only solutions that the TAs scored with failing grades
> could be detrimental and a waste of time.

I agree.

> How are you going to provide experience and guidance? For example, a student may not be aware
> that using `strncpy()` is greater than `strcpy()` wrt security. How could you expect them
> to apply that knowledge to their reviews if they couldn’t apply it in their solutions
> as @Severin has mentioned?

Another good question. Keep your eyes peeled for another blog post – I may have come up with
a solution…

> There seems to be a lot of knowledge rolling into an overall scalar mark. Perhaps you would be
> more interested in knowing the students thoughts and method of arriving at the mark.

I probably didn’t make it explicit, but in my head, the grading was done against a rubric. So
instead of x out of 10, it’d be series of marks from 0 to 4 based on various marking criteria.
At least then, we have some more granularity, and insight into *why* certain marks were given.

> It would be neat then to email back the student their assignment with their classmates +/-
> score for each line and any comments. This has risk involved too since student generated
> comments could be wrong, misleading, vulgar.

Yep. So far, I have skirted around the question of whether or not to actually *show* the reviews
to the original author. It really depends, I suppose, on how useful the feedback would be.

Thanks for the feedback!

Good point, re: the time to ponder an assignment. Hadn’t considered that.

Not sure how I could account for that – unless I use a real-world assignment, and my participants are students from a course that recently gave it (ie: they’re already familiar with the assignment).

> Ok, but I see a little contradiction here. Either one doesn’t give students hints
> as to what to look for when reviewing, or students get hints and will be able to
> potentially compensate for what they might have missed while working on the assignment
> when doing reviews. I.e. they’d have a chance to do a good review but get weaker
> marks for their assignment, which, still, could make sense. The question is,
> what is of most interest to you when conducting this experiment?

Another good question.

What I’m most interested in, is: does this kind of peer-grading cause students to learn something measurably useful?

Thanks again for the input!

(I agree that having students miss important things about the assignment
even in code review means that there is a problem, but that information
seems more like something for the instructor to know than something to
cost the students yet more marks.)

My suggestion is that every student doing reviews should get a cheat
sheet of ‘common mistakes to look for’. This shouldn’t cover everything
and wouldn’t reduce code review to a mechanical process, but hopefully
would get much closer to only measuring how well the students can review
code and see things in it. The cheat sheet might be drawn up by the TAs
after marking the assignments or it might be prepared in advance based
on what the instructor is expecting people to have problems with.

Also, it strikes me that it might be interesting to have students review
their own assignment at the end of the review process. Having gone
through a series of code reviews of other people’s code, have they
learned enough to see the flaws in their own code? (My naive feeling
is that this is not gameable, except to the extent that if you submit
a terrible assignment you can at least get some marks back for being
able to admit how terrible it is.)

It would be interesting to break your test subjects into three groups based on the initial TA markings then present the solutions in the following orders for review:
* Random
* Decreasing TA marks
* Increasing TA marks

Then ask the students if they feel they were able to learn by reviewing the solutions.

I would be interested to see how the *quality* of the solutions reviewed by the student will help them learn. I could see how reviewing only solutions that the TAs scored with failing grades could be detrimental and a waste of time.

In first year engineering programming courses – huge volume of submitted assignments, not a lot of good ones, perhaps lots of plagiarism – we had the opportunity to look at the suggested solution (used for marking) in tutorials. The professor would simply comment on areas of common weaknesses or strengths of the class as a whole if the assignment was reviewed at all.

How are you going to provide experience and guidance?
For example, a student may not be aware that using `strncpy()` is greater than `strcpy()` wrt security.
How could you expect them to apply that knowledge to their reviews if they couldn’t apply it in their solutions as @Severin has mentioned?

There seems to be a lot of knowledge rolling into an overall scalar mark.
Perhaps you would be more interested in knowing the students thoughts and method of arriving at the mark. If that is the case, allow students to annotate a line with +/- marks (kind of like social network voting – ie thumbs up/down) with the option of adding textual comments. Then sum them all up for a final result. Two 6/10′s may have the same final mark, but could have different paths of arriving there. Of course this is much more complex and would be overkill if you are only interested in the deviation from the TA’s grade.

It would be neat then to email back the student their assignment with their classmates +/- score for each line and any comments. This has risk involved too since student generated comments could be wrong, misleading, vulgar.

By attributing comments to classmates instead of anonymously (privacy issue) you may be able to add accountability to such a system.

> The experiment is still being designed, but yes, I imagine the graders should know what the assignment is all about – much like the students and markers would.
> And then yes, they would begin grading immediately.

Note: I’m speaking of what I’ve experienced when working on an assignment here:

If the assignment is significant effort/significantly complex then fairly I noticed after spending some time working on the assignment that I hadn’t understood the task(s) in its entirety. Even after several read the specs, read the code, work on implementation, read the specs go to the code again cycles, I was unsure as to what instructors wanted us to be aware of. Then at some point and having had some “distance”, you have your personal “aha” moment as to what this is all about. It could just be me, but if the experiment will be in a way that student subjects read the assignment specs and would immediately start reviewing other student’s code after that I anticipate different outcomes of your experiment as if students had worked on the assignment for a couple of days and, hence, have a clearer idea and do better code review.

Of course this is a rather experiment specific concern. But if you are trying to assess students’ ability to do code review after having worked on an assignment your experiment should reflect this. I could have gotten something wrong.

>> I assume you will be providing students with some guidance (similar to the one grad students get from the instructor) as to what to look for.
>> If this will be provided, how will this influence overall outcome of your study?

> Good question. I imagine it’d be a lot like MarkUs…there’d be criteria, and each criterion level would have a description as to what is expected for that mark to apply.

Ok, but I see a little contradiction here. Either one doesn’t give students hints as to what to look for when reviewing, or students get hints and will be able to potentially compensate for what they might have missed while working on the assignment when doing reviews. I.e. they’d have a chance to do a good review but get weaker marks for their assignment, which, still, could make sense. The question is, what is of most interest to you when conducting this experiment?

Interesting stuff!