{"id":740,"date":"2009-09-17T15:49:09","date_gmt":"2009-09-17T20:49:09","guid":{"rendered":"http:\/\/mikeconley.ca\/blog\/?p=740"},"modified":"2023-12-20T16:25:19","modified_gmt":"2023-12-20T21:25:19","slug":"process-improvement-of-peer-code-review-and-behavior-analysis-of-its-participants","status":"publish","type":"post","link":"https:\/\/mikeconley.ca\/blog\/2009\/09\/17\/process-improvement-of-peer-code-review-and-behavior-analysis-of-its-participants\/","title":{"rendered":"Process Improvement of Peer Code Review and Behavior Analysis of its Participants"},"content":{"rendered":"

Process Improvement of Peer Code Review and Behavior Analysis of its Participants
\n<\/a><\/h2>\n

by WANG Yan-qing, LI Yi-jun, Michael Collins, LIU Pei-jie
\nSIGCSE ’08, March 12-15, 2008<\/p>\n

If you’ve been following, I’ve been trying to figure out why code reviews aren’t a part of the basic undergraduate computer science curriculum.\u00a0 The other<\/a> papers<\/a> and articles I’ve read so far have had less to do with the classroom, and more to do with code reviews in industry.<\/p>\n

This paper got a little bit closer to the classroom, and, more importantly, closer to my particular question.<\/p>\n

To begin, the paper introduces some terminology I’m not familiar with – the software crisis<\/a>.\u00a0 I’m familiar with the concept though:\u00a0 writing good software for large systems is not a simple problem, and as computers become a bigger and more important part of our lives, this inability to easily write good code could quickly end up biting us in the collective rear.<\/p>\n

Code review is one of several methods that the software industry has adopted to try to “tame” the software crisis.<\/p>\n

I like this part:<\/p>\n

Even though code reviews are time consuming, they are much more efficient than testing [19]. A typical engineer, for example,\u00a0 will find approximately 2 to 4 defects in an hour of unit testing but will find 6 to 10 defects in each hour of review code [19].<\/p><\/blockquote>\n

What more argument do you need?\u00a0 It’s just a matter of getting rid of that “time consuming” part, right?\u00a0 Right…<\/p>\n

And this is even juicier:<\/p>\n

PCR [peer code review] is a technique which is generally considered to be effective on promoting students\u2019 higher cognitive skills [9], since students use their own knowledge and skill to interpret, analyze and evaluate others\u2019 work to clarify and correct it [2].<\/p><\/blockquote>\n

Wonderful!\u00a0 I’m in my problem space!<\/a><\/p>\n

Reading along, it seems that this paper is introducing a new, refined structure for PCR, and will detail results of a study on using that new structure in a programming course.\u00a0 Cool.<\/p>\n

The introduction ends by saying that the new structure seemed to enhance the quality of student’s work, as well as their ability to critique one another.\u00a0 Great news!<\/p>\n

It’s not all sunshine and puppies, though – they also mention that they ran into a few problems, and that they’ll be discussing those too.<\/p>\n

So the first thing they’ve done, is tried to make the terminology clearer:<\/p>\n

Roles<\/h3>\n
    \n
  • Author:\u00a0 the student who writes the code that is being reviewed<\/li>\n
  • Reviewer:\u00a0 the person who is reviewing the code<\/li>\n
  • Reviser:\u00a0 the author, after receiving a Comments Form from a Reviewer<\/li>\n
  • Instructor:\u00a0 the teacher or qualified TA who is responsible for the class<\/li>\n<\/ul>\n

    Documents<\/h3>\n
      \n
    • Manuscript Code:\u00a0 the unrevised code that is first submitted by an Author<\/li>\n
    • Comments Form:\u00a0 the comments given from the Reviewer to the Author<\/li>\n
    • Revision Code:\u00a0 the code that is revised by the Reviser after the Reviewer gives the Reviser the Comments Form (whew…follow that?)<\/li>\n
    • Reference Solution:\u00a0 the “answer” to the assignment, held by the Instructor<\/li>\n<\/ul>\n

      Now that we’ve got all the players and documents laid out, let’s take a look at the process:<\/p>\n

      Process<\/h3>\n
        \n
      • Phase 1:\u00a0 The Author completes the Manuscript Code<\/li>\n
      • Phase 2:\u00a0 The Author emails the Manuscript Code to the Instructor.\u00a0 Simultaneously, a blank Comments Form and a copy of the Manuscript Code is sent to a Reviewer<\/li>\n
      • Phase 3:\u00a0 The Reviewer reviews the code as soon as possible, filling in the Comments Form.<\/li>\n
      • Phase 4:\u00a0 The Reviewer sends the completed Comments Form back to the Author, and also sends a carbon copy to the Instructor<\/li>\n
      • Phase 5:\u00a0 After receiving the Comments Form, the Reviser (who was originally the Author…oh boy…almost went cross-eyed, there) makes the appropriate alterations to the original Manuscript Code, referencing the Comment Form where appropriate.\u00a0 The completed Revision Code is emailed to the Instructor.<\/li>\n
      • Phase 6:\u00a0 The Instructor should now have a copy of the original Manuscript Code, the completed Comments Form, and the final Revision Code.\u00a0 The Instructor should be able to check that the author and reviewer did their work properly.<\/li>\n<\/ul>\n

        Wow.\u00a0 What a convoluted way of saying something simple.\u00a0 They even included a diagram, with lots of arrows.\u00a0 Somehow, I think this could be said simpler.\u00a0 Oh well.<\/p>\n

        It also sounds like a lot of emailing.\u00a0 You’re balancing your course on the reliability of the email protocol?\u00a0 Errr….<\/p>\n

        Well, let’s see what problems they ran into…<\/p>\n

          \n
        1. The assumption that all participants would carefully and responsibly carry out each phase of the process was faulty.\u00a0 This may have been due to “careless authors, irresponsible reviewers and busy instructors in the review process”.<\/li>\n
        2. Some students lack the coding ability to either:\n
            \n
          1. Produce code that is readable and reviewable in a constructive way<\/li>\n
          2. Review code in a constructive, or informed way<\/li>\n<\/ol>\n<\/li>\n
          3. The process is difficult to control due to the reliance on email (no kidding!)\n
              \n
            1. Some students would not submit Manuscript Code or Comment Forms on time<\/li>\n
            2. Some students would submit multiple copies of their Manuscript Code, due to an inherent mistrust of the reliability of email<\/li>\n<\/ol>\n<\/li>\n
            3. There was opportunity for students to “game” the process to their advantage. In this particular study, there was very little control of who was doing what.\u00a0 Though a particular Author was supposed to write the Manuscript Code, this wasn’t enforced, and there was an occasion where another student wrote the code instead.\u00a0 Same with review writing, and revision writing.\u00a0 Yeah, cheating is always a problem.<\/li>\n<\/ol>\n

              The paper then goes into some discussion about the observed behaviour of Authors and Reviewers.\u00a0 They noted that most students did not enjoy reviewing very poorly written code, and don’t give their best efforts on reviews for such code.\u00a0 Mere encouragement from the instructor was not enough to compel them to give their best reviews either.\u00a0 The paper suggests finding some way of making Reviewers review code more carefully; perhaps through awarding bonus marks.<\/p>\n

              Behaviour of Instructors was also analyzed.\u00a0 The paper mentioned that Instructors with large class sizes might try to cut down on their workload by only viewing the Comment Forms that the Reviewers had provided.\u00a0 But this strategy does not give the Instructor the entire story, and is open to manipulation from students.<\/p>\n

              The paper ends with a discussion about group formations, and how best to diffuse student cheating conspiracies.<\/p>\n

              At the last moment, they suggest some “web-based [application] with a built-in blind review mechanism” be developed.<\/p>\n

              Hm.<\/p>\n","protected":false},"excerpt":{"rendered":"

              Process Improvement of Peer Code Review and Behavior Analysis of its Participants by WANG Yan-qing, LI Yi-jun, Michael Collins, LIU Pei-jie SIGCSE ’08, March 12-15, 2008 If you’ve been following, I’ve been trying to figure out why code reviews aren’t a part of the basic undergraduate computer science curriculum.\u00a0 The other papers and articles I’ve […]<\/p>\n","protected":false},"author":4,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[454,1],"tags":[475],"class_list":["post-740","post","type-post","status-publish","format-standard","hentry","category-code-reviews","category-uncategorized","tag-peer-code-review"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/prmTy-bW","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/posts\/740","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/comments?post=740"}],"version-history":[{"count":8,"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/posts\/740\/revisions"}],"predecessor-version":[{"id":3225,"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/posts\/740\/revisions\/3225"}],"wp:attachment":[{"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/media?parent=740"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/categories?post=740"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/tags?post=740"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}