{"id":787,"date":"2009-09-23T15:03:25","date_gmt":"2009-09-23T20:03:25","guid":{"rendered":"http:\/\/mikeconley.ca\/blog\/?p=787"},"modified":"2023-12-20T16:25:19","modified_gmt":"2023-12-20T21:25:19","slug":"exploring-peer-review-in-the-computer-science-classroom-part","status":"publish","type":"post","link":"https:\/\/mikeconley.ca\/blog\/2009\/09\/23\/exploring-peer-review-in-the-computer-science-classroom-part\/","title":{"rendered":"Exploring Peer Review in the Computer Science Classroom: Part 1"},"content":{"rendered":"<h3><a href=\"http:\/\/arxiv.org\/pdf\/0907.3456v1\">Exploring Peer Review in the Computer Science Classroom<\/a><\/h3>\n<h4>by Scott Turner and <a href=\"http:\/\/perez.cs.vt.edu\/\">Manuel A. P\u00e9rez Qui\u00f1ones<\/a><\/h4>\n<p>I&#8217;m new at reading papers, so I&#8217;ve gotten used to 5 or 10 page-ers.\u00a0 This looks like the big one, though &#8211; 69 pages.<\/p>\n<p>I assume they have something significant to say.\u00a0 The title sure sounds interesting, especially considering <a href=\"http:\/\/mikeconley.ca\/blog\/2009\/09\/09\/code-reviews\/\">what I&#8217;m looking for<\/a>.<\/p>\n<p>Anyhow, it&#8217;s a big paper, and there&#8217;s a lot to go through.\u00a0 Let&#8217;s get started.<\/p>\n<p>Right off the bat, I can see that they&#8217;re interested in the same problem that I am:<\/p>\n<blockquote><p>Peer review, while it has many known benefits (Zeller 2000; Papalaskari 2003; Wolfe 2004; Hamer, Ma et al. 2005; Trytten 2005), and is used extensively in other fields (Falchikov and Goldfinch 2000; Topping, Smith et al. 2000; Liu and Hansen 2002; Dossin 2003; Carlson and Berry 2005) and in industry (Anderson and Shneiderman 1977; Anewalt 2005; Hundhausen, Agrawal et al. 2009), is not as widely used in the computer science curriculum.\u00a0 This may be due, in part, to a lack of information about <em>what<\/em>, <em>who<\/em> and <em>when<\/em> to review in order to achieve specific goals in computer science.<\/p><\/blockquote>\n<p>This next part got my attention:<\/p>\n<blockquote><p>That is not to say that the literature is silent on these issues.\u00a0 The studies make these choices but there are few reasons given for the decisions and even fewer comparisons performed to show relative value of those options.<\/p><\/blockquote>\n<p>Holy smokes.\u00a0 A pretty bold critique of those previous papers (at least one of which, <a href=\"http:\/\/mikeconley.ca\/blog\/2009\/09\/21\/integrating-pedagogical-code-reviews-into-a-cs-1-course-an-empirical-study\/\">I&#8217;ve already reviewed<\/a>).\u00a0 It sounds like Scott and Manuel were as disappointed in some of the peer code review literature as I was.\u00a0 If I was part of an audience that was being read this paper aloud, I would &#8220;wooo!&#8221; at this point.<\/p>\n<blockquote><p>What is needed is a clearer understanding of the requirements that the discipline imposes on the peer review process so that it may be effectively used.<\/p><\/blockquote>\n<p>Cool &#8211; I&#8217;m looking forward to seeing what they dug up.<\/p>\n<p>Reading onwards through the introduction, I&#8217;m seeing the same basic arguments for peer code review that I&#8217;ve seen elsewhere.\u00a0 I&#8217;ll summarize:<\/p>\n<ul>\n<li>PCR (peer code review)\u00a0 involves active use of higher levels of <a href=\"http:\/\/en.wikipedia.org\/wiki\/Bloom%27s_Taxonomy\">Bloom&#8217;s Taxonomy<\/a>:\u00a0 synthesis, analysis and evaluation, both for reviewers and review-ees<\/li>\n<li>PCR prepares students for industry, since code review is (or should be) a common part of professional software development<\/li>\n<\/ul>\n<p>Soon after, a good point is brought up &#8211; PCR is <em>potentially<\/em> a beneficial learning activity, but it all depends on the goals of a particular assignment.\u00a0 A particular review structure may be better for improving code quality, and another might be better for increasing student motivation.\u00a0 These considerations need to be taken into account when choosing a PCR structure.<\/p>\n<p>By PCR structure, I think the writers mean:<\/p>\n<ul>\n<li><em>What<\/em> is being reviewed &#8211; source code vs design diagrams, for example<\/li>\n<li><em>Who<\/em> is being reviewed &#8211; students could review one another, or they could all review a piece of code provided by the instructor<\/li>\n<li><em>When<\/em> the review occurs &#8211; what level of students should take part in PCR?\u00a0 Is there a minimum grade level that must be achieved before PCR is effective?\u00a0 And when, during a project, should PCR happen?\u00a0 Early in the design process, or after-the-fact &#8211; like a &#8220;code autopsy&#8221;?<\/li>\n<\/ul>\n<p>These are good questions.\u00a0 No wonder this paper is so long &#8211; I think their experiment is going to try to answer them all.<\/p>\n<p>And just when things are starting to get good&#8230;they go into a literature review.\u00a0 Yech.\u00a0 I know it&#8217;s important to lay the groundwork, but I think I just felt my eyes turn to oatmeal.<\/p>\n<p>The literature starts by briefly listing off those sources again &#8211; past papers who have tried to deal with this topic.\u00a0 They categorize them based on what the papers try to deliver, and then they give them a light slam for not having a scientific basis on which to form the structures of their PCR structures.\u00a0 I heartily agree.<\/p>\n<p>The first part of the literature review discusses the benefits of peer review in other fields.\u00a0 Papers as far back as the 1950&#8217;s are cited.\u00a0 I think it goes without saying that having other people critique your work can be a great way of receiving constructive feedback (ask any playwrite, for example).\u00a0 I guess these fellas feel they have to be pretty rigorous though, and really ground their argument in some solid past work.\u00a0 Power to them.<\/p>\n<p>An interesting notion is brought up &#8211; peer reviews over an extended length of time within the same groups helps cultivate &#8220;interaction between&#8230;peers and for the building of knowledge&#8221;.\u00a0 One-time reviews, however, &#8220;[lends] itself more to a cognitive approach&#8230;more attention can be paid to the changes in the students&#8217; thought processes&#8221;.\u00a0 Hm.<\/p>\n<p>This fellow named Topping seems to be quite popular with these guys.\u00a0 Apparently, he came up with something called &#8220;Topping&#8217;s Peer Review Topology&#8221;, and I get the feeling that he is one of their primary sources in figuring out different ways of constructing PCR structures.<\/p>\n<p>Oh, and an important morsel just got snuck in:<\/p>\n<blockquote><p>We are interested in how the student is affected by the acts of reviewing and being reviewed rather than by the social interaction occurring during the process.<\/p><\/blockquote>\n<p>So that sense of community that <a href=\"http:\/\/mikeconley.ca\/blog\/2009\/09\/21\/integrating-pedagogical-code-reviews-into-a-cs-1-course-an-empirical-study\/\">that other paper<\/a> was talking about &#8211; not being looked at here.<\/p>\n<p>The paper then goes on to chisel down some of the jargon from Topping&#8217;s Topology, and make it fit the field of Computer Science.\u00a0 By doing this, the writers simpy reiterate the variables they&#8217;re going to be playing with:\u00a0 what, who, and when.<\/p>\n<p>The paper then dives into a two page summary of some peer review papers, and the various results that they found.\u00a0 They note a few instances where peer review seemed to improve student performance, and other cases where peer review resulted in semi-disaster.\u00a0 There are just as many theories as there are conflicting accounts.\u00a0 From reading this stuff, it seems that peer review is a vast and complicated topic, and nobody seems to really have a firm grasp on it just yet.<\/p>\n<p>Likewise, rubric creation seems to be a bit of a contentious topic:<\/p>\n<blockquote><p>While there seems to be a general consensus that rubrics are important and that they improve the peer review activity, there is not as much agreement on how they should be implemented.<\/p><\/blockquote>\n<p>However, it is clear that rubrics are useful in peer review for novice reviewers:<\/p>\n<blockquote><p>Rubrics can supply the guidance students need to learn how to evaluate an assignment.\u00a0 It provides the needed scaffolding until the students are comfortable witht he process and the domain to make correct judgements.<\/p><\/blockquote>\n<p>Makes sense to me.<\/p>\n<p>The paper then asks an important question: what makes a &#8220;successful&#8221; review in the education context?\u00a0 Greater understanding?\u00a0 Learning new concepts?\u00a0 Improved grades?\u00a0 Better designs?\u00a0 Fewer code defects?<\/p>\n<p>The answer:\u00a0 it really depends on the instructor, and what the course wants from the PCR process.\u00a0 For example, what is more important &#8211; having the reviewer learn to review?\u00a0 Or having the review-ee receive good feedback?\u00a0 Or both?\u00a0 PCR is complex, and has lots of things to consider&#8230;<\/p>\n<p>The next section highlights how technology is used to support peer review.\u00a0 One particularly interesting example, is of a <a href=\"http:\/\/moodle.org\/\">Moodle<\/a> module that allows for peer reviews on assignments.\u00a0 Apparently, the authors are fans.\u00a0 I&#8217;ve never used Moodle before, and haven&#8217;t yet found the module that they&#8217;re talking about, but it sounds worth investigation.<\/p>\n<p>The very next section details their experiment &#8211; their method, their data, and their results.\u00a0 However, this post is getting a bit long, so I&#8217;m going to stop here, and continue on in a second post.<\/p>\n<p><a href=\"http:\/\/mikeconley.ca\/blog\/2009\/09\/24\/exploring-peer-review-in-the-computer-science-classroom-part-2\/\">Stay tuned for the exciting conclusion!<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Exploring Peer Review in the Computer Science Classroom by Scott Turner and Manuel A. P\u00e9rez Qui\u00f1ones I&#8217;m new at reading papers, so I&#8217;ve gotten used to 5 or 10 page-ers.\u00a0 This looks like the big one, though &#8211; 69 pages. I assume they have something significant to say.\u00a0 The title sure sounds interesting, especially considering [&hellip;]<\/p>\n","protected":false},"author":4,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[454],"tags":[1218,58,496,493,491,475,492,470],"class_list":["post-787","post","type-post","status-publish","format-standard","hentry","category-code-reviews","tag-code-reviews","tag-education","tag-experiment","tag-manuel-a-perez-quinones","tag-pcr","tag-peer-code-review","tag-scott-turner","tag-study"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/prmTy-cH","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/posts\/787","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/comments?post=787"}],"version-history":[{"count":13,"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/posts\/787\/revisions"}],"predecessor-version":[{"id":798,"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/posts\/787\/revisions\/798"}],"wp:attachment":[{"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/media?parent=787"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/categories?post=787"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/tags?post=787"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}