Author Archives: Mike Conley

Exploring Peer Review in the Computer Science Classroom: Part 1

Exploring Peer Review in the Computer Science Classroom

by Scott Turner and Manuel A. Pérez Quiñones

I’m new at reading papers, so I’ve gotten used to 5 or 10 page-ers.  This looks like the big one, though – 69 pages.

I assume they have something significant to say.  The title sure sounds interesting, especially considering what I’m looking for.

Anyhow, it’s a big paper, and there’s a lot to go through.  Let’s get started.

Right off the bat, I can see that they’re interested in the same problem that I am:

Peer review, while it has many known benefits (Zeller 2000; Papalaskari 2003; Wolfe 2004; Hamer, Ma et al. 2005; Trytten 2005), and is used extensively in other fields (Falchikov and Goldfinch 2000; Topping, Smith et al. 2000; Liu and Hansen 2002; Dossin 2003; Carlson and Berry 2005) and in industry (Anderson and Shneiderman 1977; Anewalt 2005; Hundhausen, Agrawal et al. 2009), is not as widely used in the computer science curriculum.  This may be due, in part, to a lack of information about what, who and when to review in order to achieve specific goals in computer science.

This next part got my attention:

That is not to say that the literature is silent on these issues.  The studies make these choices but there are few reasons given for the decisions and even fewer comparisons performed to show relative value of those options.

Holy smokes.  A pretty bold critique of those previous papers (at least one of which, I’ve already reviewed).  It sounds like Scott and Manuel were as disappointed in some of the peer code review literature as I was.  If I was part of an audience that was being read this paper aloud, I would “wooo!” at this point.

What is needed is a clearer understanding of the requirements that the discipline imposes on the peer review process so that it may be effectively used.

Cool – I’m looking forward to seeing what they dug up.

Reading onwards through the introduction, I’m seeing the same basic arguments for peer code review that I’ve seen elsewhere.  I’ll summarize:

  • PCR (peer code review)  involves active use of higher levels of Bloom’s Taxonomy:  synthesis, analysis and evaluation, both for reviewers and review-ees
  • PCR prepares students for industry, since code review is (or should be) a common part of professional software development

Soon after, a good point is brought up – PCR is potentially a beneficial learning activity, but it all depends on the goals of a particular assignment.  A particular review structure may be better for improving code quality, and another might be better for increasing student motivation.  These considerations need to be taken into account when choosing a PCR structure.

By PCR structure, I think the writers mean:

  • What is being reviewed – source code vs design diagrams, for example
  • Who is being reviewed – students could review one another, or they could all review a piece of code provided by the instructor
  • When the review occurs – what level of students should take part in PCR?  Is there a minimum grade level that must be achieved before PCR is effective?  And when, during a project, should PCR happen?  Early in the design process, or after-the-fact – like a “code autopsy”?

These are good questions.  No wonder this paper is so long – I think their experiment is going to try to answer them all.

And just when things are starting to get good…they go into a literature review.  Yech.  I know it’s important to lay the groundwork, but I think I just felt my eyes turn to oatmeal.

The literature starts by briefly listing off those sources again – past papers who have tried to deal with this topic.  They categorize them based on what the papers try to deliver, and then they give them a light slam for not having a scientific basis on which to form the structures of their PCR structures.  I heartily agree.

The first part of the literature review discusses the benefits of peer review in other fields.  Papers as far back as the 1950’s are cited.  I think it goes without saying that having other people critique your work can be a great way of receiving constructive feedback (ask any playwrite, for example).  I guess these fellas feel they have to be pretty rigorous though, and really ground their argument in some solid past work.  Power to them.

An interesting notion is brought up – peer reviews over an extended length of time within the same groups helps cultivate “interaction between…peers and for the building of knowledge”.  One-time reviews, however, “[lends] itself more to a cognitive approach…more attention can be paid to the changes in the students’ thought processes”.  Hm.

This fellow named Topping seems to be quite popular with these guys.  Apparently, he came up with something called “Topping’s Peer Review Topology”, and I get the feeling that he is one of their primary sources in figuring out different ways of constructing PCR structures.

Oh, and an important morsel just got snuck in:

We are interested in how the student is affected by the acts of reviewing and being reviewed rather than by the social interaction occurring during the process.

So that sense of community that that other paper was talking about – not being looked at here.

The paper then goes on to chisel down some of the jargon from Topping’s Topology, and make it fit the field of Computer Science.  By doing this, the writers simpy reiterate the variables they’re going to be playing with:  what, who, and when.

The paper then dives into a two page summary of some peer review papers, and the various results that they found.  They note a few instances where peer review seemed to improve student performance, and other cases where peer review resulted in semi-disaster.  There are just as many theories as there are conflicting accounts.  From reading this stuff, it seems that peer review is a vast and complicated topic, and nobody seems to really have a firm grasp on it just yet.

Likewise, rubric creation seems to be a bit of a contentious topic:

While there seems to be a general consensus that rubrics are important and that they improve the peer review activity, there is not as much agreement on how they should be implemented.

However, it is clear that rubrics are useful in peer review for novice reviewers:

Rubrics can supply the guidance students need to learn how to evaluate an assignment.  It provides the needed scaffolding until the students are comfortable witht he process and the domain to make correct judgements.

Makes sense to me.

The paper then asks an important question: what makes a “successful” review in the education context?  Greater understanding?  Learning new concepts?  Improved grades?  Better designs?  Fewer code defects?

The answer:  it really depends on the instructor, and what the course wants from the PCR process.  For example, what is more important – having the reviewer learn to review?  Or having the review-ee receive good feedback?  Or both?  PCR is complex, and has lots of things to consider…

The next section highlights how technology is used to support peer review.  One particularly interesting example, is of a Moodle module that allows for peer reviews on assignments.  Apparently, the authors are fans.  I’ve never used Moodle before, and haven’t yet found the module that they’re talking about, but it sounds worth investigation.

The very next section details their experiment – their method, their data, and their results.  However, this post is getting a bit long, so I’m going to stop here, and continue on in a second post.

Stay tuned for the exciting conclusion!

Integrating Pedagogical Code Reviews into a CS 1 Course: An Empirical Study

Integrating Pedagogical Code Reviews into a CS 1 Course:
An Empirical Study

by Christopher Hundhausen, Anukrati Agrawal, Dana Fairbrother, and Michael Trevisan

Here’s another paper talking about using code review in a first year programming course.  Let’s give it a peek.

The idea of this paper is to introduce a type of “studio-based” learning environment for programmers.  The term originates from architecture, where architecture students bring various designs into class, and then the classroom as a unit critiques and evaluates the design.  In the studio-based learning model, the instructor guides the critiques.  This way, not only does the critiqued student get valuable feedback, but the critics also learn about what to critique.  Remember when I talked about learning how to read code? Studio-based learning seems to be one solution to that problem.

Studio-based learning could also help foster community among the students.  In a field where isolated cubicles are still quite normal (I’m sitting in one right now!), a sense of community could be a welcome relief.

The paper also notes the following:

…the peer review process can (a) prepare students to deal with criticism [1], teach students to provide constructive criticism to others [1], provide students with experience with coming to a consensus opinion in cases where opinions differ [1], and build teamwork skills [8] . All of these are “soft” or “people” skills that are likely to be important in their future careers.

I whole-heartedly agree.

The paper then goes on to give a quick survey of how peer review (not just for code, but peer review in general) is currently used in education:

  • pair programming (which is what I experienced at UofT)
  • “in-class conference” model
  • peer feedback meetings (mostly for project documentation)
  • web-based peer grading solutions (RRAS, and Peer Grader)
  • peer code review

Maybe I’m just getting tired of reading these papers, but I found this section particularly difficult to get through.  I had to read it 3 or 4 times just to understand what was going on.

Finally, they get on with it, and talk about their approach to peer code review – a student oriented version of a formal code review.  Yikes – if by formal code review, they mean a Fagan Inspection, then these poor students are in for some long meetings…

Here is their 5 step outline of their approach:

  1. Plan the inspection of a specific piece of code.
  2. Hold a kick-off meeting with an inspection team to distribute the code to be inspected, train the team on the process, and set inspection goals.
  3. Have members of the inspection team inspect the code for defects on their own time.
  4. Hold an inspection meeting to log issues found by individual members’ inspections, and to find additional issues.
  5. Edit the code to address the issues uncovered in the inspection, and verify that the issues have been resolved.

Yep, that sounds a lot like a Fagan Inspection.  As I’ve already found out, there are lighter, faster, and just-as-effective approaches to peer code review…

After using their approach, their study noticed:

  1. A positive trend in the quality of the students code, and a negative trend in the number of defects found per review
  2. Discussion transitioned from syntax and style to more high-level concepts, such as architecture and design of the system.  Thus, meaningful discussion was generated using their technique.  The study notes, however, that “…we
    cannot provide any objective evidence that students were able to provide helpful critiques of each other’s code within the code reviews themselves.”
  3. There is anecdotal evidence that suggests that this work helped to create more of a community feel among the students in the class

Sounds like a lot of anecdotal evidence.

I don’t know, reading the results of this paper, I was disappointed.  The evidence/data they collected feels light and insubstantial.  I certainly support what the study was trying to do, and maybe I’ve read too many peer review papers, but I don’t find it surprising anymore to hear that peer code review improves students code.  It’s good to have evidence for that, but I was hoping for something more.

The paper closes with the writers almost agreeing with me:

There are several limitations to our results that suggest the need for more rigorous follow-up studies with both larger student samples and additional data collection methods.

I knew it wasn’t just me!

Anyhow, the paper finishes with the writers outlining how they’d do future studies.

And that’s that.

Possible Applications for my Adventure Game Obsession – Part 1

If you don’t know this already, I really dig adventure games.  Seriously.  Just click these words to see how much I dig them.

And I keep running into adventure game stuff in the most unexpected places.  A few days ago, Yuri Takhteyev from the Faculty of Information spoke to the Software Engineering group about his work studying the use and popularity of the Lua language in Rio de Janeiro, Brazil.  When he brought up Lua, I couldn’t help remembering that Lua was used by the GrimE engine to script Grim Fandango…

Wouldn’t it be awesome to find a way of turning my passion for adventure games into something that is useful in the field of Computer Science?

My supervisor has advised me not to think too much about my research paper just yet, and to just peek around to get a feel for what’s going on in the various facets of Computer Science.  I take this advice to heart, and yet I can’t help noticing where my passion for adventure games might be applied…

Here are a few things I’ve come up with:

Storytelling Alice

Storytelling Alice is an attempt to find a fun, intuitive way of teaching basic programming with the Alice language to middle-school students.  It was designed and developed by Caitlin Kelleher as part of her PhD thesis at Carnegie Mellon.  Storytelling Alice is designed to use storytelling as a motivating context to get students to learn various programming techniques.

In Storytelling Alice, students are compelled to learn more in order to tell more of a story.  I wonder if they’d be willing to learn more to reveal more of a story?  This would be very similar to the way adventure games reward players with story after solving a puzzle.

Check out Storytelling Alice here

Storyboarding

I’ve recently started taking Khai Truong’s CSC2514 – Human-Computer Interaction course.  One of the first papers he got us to read this week was one that he’d written on storyboarding.

Put simply, storyboards are used by interaction designers as a low-cost way of testing out designs with their potential audience.  They are similar to the storyboards used in writing/designing movies or television productions, but are instead used to communicate use cases, environment of use, physical embodiment of the system, etc.

Here is a copy of the paper, if you’re interested in reading it.

Here’s something I found interesting:

Commercial products marketed specifically for storyboard creation are available, but they are designed for experts and can be difficult for novices to use … Also, expert designers expressed that the greatest challenge for them is storytelling.  These software products are not designed to support that process and may even be detrimental to it, because they do not provide complete creative flexibility in terms of what can be developed.

Very interesting.  Adventure games are designed from the ground up to tell a story.  I wonder if the tools that adventure games are created with could lend something to these storyboard creation tools?

As my studies continue, perhaps I’ll report more potential uses for adventure game technology.

Until then, I’ll leave you with a clip from a playthrough from one of my favourite adventure games of all time, The Dig.  It might not be Dicken’s, but damned if it doesn’t hold my attention with an iron grip.

Just brilliant.

A Few Things Drama Can Bring to Computer Science

So, yesterday I wrote:

[W]hat can Drama bring to Computer Science?

The easy one is presentation/communication skills.  A CS student might be brilliant, but that doesn’t mean they can present or communicate.  And if an idea can’t be communicated, it’s worthless.

But what else?  Any ideas?  I’m going to think about this for a bit, and I’ll see if I can come up with any more.

I posted the question on Twitter, and on my Facebook.  I was quite surprised by the amount of feedback I got back – apparently, quite a few people are interested in this topic.

Thanks for everybody who posted, or who came up to talk to me about this!  Let me summarize what I heard back:

  • Without a doubt, work in Drama hones movement/body senses.  It also trains us to use and take care of our body, and voice, like a musician would take care of a musical instrument.  Spending too much time hunkered over a keyboard can have detrimental effects on the body over time – I can personally admit to having absolutely awful shoulder tension, no doubt to my constant typing.  I only became aware of this tension, and how to deal with it, thanks to my work in Drama.  The dichotomy between body and mind is, in my humble opinion, a Western myth, and when you stop separating them, and get them to work together, amazing things can happen.  Just ask any contact improviser.
  • Drama is also emotional work.  No, this doesn’t mean we sit in a big circle and cry, and get credit for it.  Emotions are something that we study – how to mimic them, how to summon them out of ourselves, how to describe them, and abstractly represent them.  This is where Psychology, Drama, and Human-Computer Interaction might have some overlap.  In particular, it must be remembered that theatre is a communications medium between the actor(s) on stage, and the audience.  A webpage is also a communications medium.  Perhaps the theatre can teach a website a thing or two about communication.  I wonder what Marshall McLuhan would have to say on all of this…
  • Drama folk are creative, and are used to doing impossible, unreasonable things.  If you ask them to fly, they’ll figure out a way of doing it.  It’ll probably be abstract, and involve crazy lighting effects, but they’ll do it.  Production Managers are used to getting crazy, impossible requests from Directors all the time.  In my opinion, that’s what Directors are for!  Sometimes (usually due to time constraints), the Production Manager just says no to the Director – usually, though, they just go ahead and make impossible things happen – like building a triple layered reflection box.  This thing was a beast, and used a ton of computing power for live, context sensitive visual effects. I’m proud to have been a part of that.
  • In Drama, if the project is no fun, the end result suffers.  I’m pretty sure the same goes for software.  Drama students have a way of finding the “game”, the “jeu”, and the “play” (that’s why it’s called a “play”, people!) in what they’re doing.  The best actors are the ones who are clearly having a great time on stage, and are sharing this with the audience.  I believe this is applicable to software development…
  • If you want to think about complex systems, think about the stage.  At any given moment, n actors are on stage, interacting with various bits of set or props, interacting with each other – and each has their own motivation and personal story.  It can’t be a coincidence that the I* modeling language orients itself around terms like “actors” and “goals”.  It also can’t be a coincidence that many adventure game engines refer to in-game sprites as actors…

But now I want to hit the big one.  There is one thing that I really think Drama can bring to Computer Science.  Drama students are very good at it.  From what I can tell, Computer Science students rarely get exposed to it.

That thing is collaboration skills.

I already know that a few of my fellow Drama students will laugh at that – and say, “there are plenty of people in this department without collaboration skills”.  Yes, this is true.  But they tend not to do very well, or produce anything too interesting.

For me, the best, most exciting stuff comes when I’m with a group, and we’re not sure where we’re going with a project, but we just try things. We all throw a bunch of ideas in the middle, and try to put them on their feet.  The most unexpected things can happen.

Two years ago, I took a course in Experimental Theatre.  We were broken down into groups of 3 or 4 right at the beginning of the term, and given this challenge – show us what you like to see in theatre.  Show us what you think good theatre looks like.

That was it.  A blank canvas.  No script.  No “spec”.  Just each other.  It felt hopeless at first – we’d improv things, trying to get a feel for what our group wanted to do.  Nothing would happen, it’d fall flat.  We were lost.

But slowly, something started to piece itself together.  We found some material that we wanted to play with (The Wizard of Oz), and a subject that we liked – “home”.  What it means to be home, why people leave their homes, why we miss home, why we can’t stand home, what if we can’t get home, etc.  We divided the work up into 4 sections – 1 for each of us:  Dorothy, Cowardly Lion, Scarecrow, Tin Man.

It’s really hard to describe what we did.  The characters and structure from The Wizard of Oz was just a playground for a huge meditation on what “home” meant to different people.

And, wouldn’t you know it, the Robert Dziekanski Taser Incident happened just a week or so before we were to present.  It integrated perfectly into our piece.

When we finally presented it, some people were incredulous, others nauseous, others outraged.  Some were crying.  We had a huge class debate on whether or not it was appropriate to include the film clip of the Taser Incident in our piece.

But a lot of people really got something out of it.  And I believe a bunch of people from that class went to a protest rally about the incident that took place only a few days later.  I heard a lot of really positive things.  We were so excited by it that we almost took it to the Toronto Fringe Festival.

In my opinion, that was one of the most interesting, educational, horrifying, and rewarding art pieces I’d ever been involved in.  And it all started from nothing.

When are Computer Science students grouped up, and told to make whatever they want?  When are they given total freedom to just go crazy, and come up with something beautiful?  Something unique?  When are they given the frightening prospect of a blank canvas?  Maybe I’m being naive – but where are the collaborative creativity assignments in computer science education?

Now, I can imagine someone shouting – “but what about those group assignments!  What about CSC318, or CSC301?  Those were collaborative!”.

My friend, thanks for trying, but there’s a distinct difference between group problem solving, and collaborative creation.  In my mind, for collaborative creation at its best, the ensemble starts with nothing and must create something from it.  It’s the difference between having a script to toy with, and not having a script at all.

And don’t just tell me that an independent study fits the bill.  The word “independent” sabotages the whole idea – the key word is collaborate.

Oh, and did I mention that Artful Making sounds like an excellent book? Why don’t you go to their website, and read the forward by Google’s own Dr. Eric Schmidt.  I found it very illuminating.  I think this is going to the top of my to-read list.

Thanks to Blake Winton, Veronica Wong, Cam Gorrie, Jorge Aranda, Neil Ernst, Peter Freund, Jennifer Dowding, and Yev Falkovich for their input on this.  Yes, those little conversations made an impact!

Process Improvement of Peer Code Review and Behavior Analysis of its Participants

Process Improvement of Peer Code Review and Behavior Analysis of its Participants

by WANG Yan-qing, LI Yi-jun, Michael Collins, LIU Pei-jie
SIGCSE ’08, March 12-15, 2008

If you’ve been following, I’ve been trying to figure out why code reviews aren’t a part of the basic undergraduate computer science curriculum.  The other papers and articles I’ve read so far have had less to do with the classroom, and more to do with code reviews in industry.

This paper got a little bit closer to the classroom, and, more importantly, closer to my particular question.

To begin, the paper introduces some terminology I’m not familiar with – the software crisis.  I’m familiar with the concept though:  writing good software for large systems is not a simple problem, and as computers become a bigger and more important part of our lives, this inability to easily write good code could quickly end up biting us in the collective rear.

Code review is one of several methods that the software industry has adopted to try to “tame” the software crisis.

I like this part:

Even though code reviews are time consuming, they are much more efficient than testing [19]. A typical engineer, for example,  will find approximately 2 to 4 defects in an hour of unit testing but will find 6 to 10 defects in each hour of review code [19].

What more argument do you need?  It’s just a matter of getting rid of that “time consuming” part, right?  Right…

And this is even juicier:

PCR [peer code review] is a technique which is generally considered to be effective on promoting students’ higher cognitive skills [9], since students use their own knowledge and skill to interpret, analyze and evaluate others’ work to clarify and correct it [2].

Wonderful!  I’m in my problem space!

Reading along, it seems that this paper is introducing a new, refined structure for PCR, and will detail results of a study on using that new structure in a programming course.  Cool.

The introduction ends by saying that the new structure seemed to enhance the quality of student’s work, as well as their ability to critique one another.  Great news!

It’s not all sunshine and puppies, though – they also mention that they ran into a few problems, and that they’ll be discussing those too.

So the first thing they’ve done, is tried to make the terminology clearer:

Roles

  • Author:  the student who writes the code that is being reviewed
  • Reviewer:  the person who is reviewing the code
  • Reviser:  the author, after receiving a Comments Form from a Reviewer
  • Instructor:  the teacher or qualified TA who is responsible for the class

Documents

  • Manuscript Code:  the unrevised code that is first submitted by an Author
  • Comments Form:  the comments given from the Reviewer to the Author
  • Revision Code:  the code that is revised by the Reviser after the Reviewer gives the Reviser the Comments Form (whew…follow that?)
  • Reference Solution:  the “answer” to the assignment, held by the Instructor

Now that we’ve got all the players and documents laid out, let’s take a look at the process:

Process

  • Phase 1:  The Author completes the Manuscript Code
  • Phase 2:  The Author emails the Manuscript Code to the Instructor.  Simultaneously, a blank Comments Form and a copy of the Manuscript Code is sent to a Reviewer
  • Phase 3:  The Reviewer reviews the code as soon as possible, filling in the Comments Form.
  • Phase 4:  The Reviewer sends the completed Comments Form back to the Author, and also sends a carbon copy to the Instructor
  • Phase 5:  After receiving the Comments Form, the Reviser (who was originally the Author…oh boy…almost went cross-eyed, there) makes the appropriate alterations to the original Manuscript Code, referencing the Comment Form where appropriate.  The completed Revision Code is emailed to the Instructor.
  • Phase 6:  The Instructor should now have a copy of the original Manuscript Code, the completed Comments Form, and the final Revision Code.  The Instructor should be able to check that the author and reviewer did their work properly.

Wow.  What a convoluted way of saying something simple.  They even included a diagram, with lots of arrows.  Somehow, I think this could be said simpler.  Oh well.

It also sounds like a lot of emailing.  You’re balancing your course on the reliability of the email protocol?  Errr….

Well, let’s see what problems they ran into…

  1. The assumption that all participants would carefully and responsibly carry out each phase of the process was faulty.  This may have been due to “careless authors, irresponsible reviewers and busy instructors in the review process”.
  2. Some students lack the coding ability to either:
    1. Produce code that is readable and reviewable in a constructive way
    2. Review code in a constructive, or informed way
  3. The process is difficult to control due to the reliance on email (no kidding!)
    1. Some students would not submit Manuscript Code or Comment Forms on time
    2. Some students would submit multiple copies of their Manuscript Code, due to an inherent mistrust of the reliability of email
  4. There was opportunity for students to “game” the process to their advantage. In this particular study, there was very little control of who was doing what.  Though a particular Author was supposed to write the Manuscript Code, this wasn’t enforced, and there was an occasion where another student wrote the code instead.  Same with review writing, and revision writing.  Yeah, cheating is always a problem.

The paper then goes into some discussion about the observed behaviour of Authors and Reviewers.  They noted that most students did not enjoy reviewing very poorly written code, and don’t give their best efforts on reviews for such code.  Mere encouragement from the instructor was not enough to compel them to give their best reviews either.  The paper suggests finding some way of making Reviewers review code more carefully; perhaps through awarding bonus marks.

Behaviour of Instructors was also analyzed.  The paper mentioned that Instructors with large class sizes might try to cut down on their workload by only viewing the Comment Forms that the Reviewers had provided.  But this strategy does not give the Instructor the entire story, and is open to manipulation from students.

The paper ends with a discussion about group formations, and how best to diffuse student cheating conspiracies.

At the last moment, they suggest some “web-based [application] with a built-in blind review mechanism” be developed.

Hm.