{"id":1089,"date":"2010-03-09T12:56:33","date_gmt":"2010-03-09T17:56:33","guid":{"rendered":"http:\/\/mikeconley.ca\/blog\/?p=1089"},"modified":"2023-12-20T16:25:17","modified_gmt":"2023-12-20T21:25:17","slug":"turning-peer-code-review-into-a-game","status":"publish","type":"post","link":"https:\/\/mikeconley.ca\/blog\/2010\/03\/09\/turning-peer-code-review-into-a-game\/","title":{"rendered":"Turning Peer Code Review into a Game"},"content":{"rendered":"<p>A little while back, I wrote about an idea that <a href=\"http:\/\/www.twitter.com\/mike_conley\">a<\/a> <a href=\"http:\/\/www.twitter.com\/wolever\">few<\/a> <a href=\"http:\/\/www.twitter.com\/gvwilson\">of<\/a> <a href=\"http:\/\/www.twitter.com\/bwinton\">us<\/a> had been bouncing around:\u00a0 <a href=\"http:\/\/mikeconley.ca\/blog\/2010\/03\/02\/peer-code-review-achievements\/\">peer code review achievements<\/a>.<\/p>\n<p>It started out as a bit of Twitter fun &#8211; but now it has evolved, and actually become a contender for <a href=\"http:\/\/mikeconley.ca\/blog\/2010\/03\/04\/research-proposal-my-problem-space\/\">my Masters research<\/a>.<\/p>\n<p>So I&#8217;ve been reading up on <a href=\"http:\/\/buildingreputation.com\/writings\/2010\/02\/on_karma.html\">reputation<\/a> and <a href=\"http:\/\/www.gamasutra.com\/blogs\/GregMcClanahan\/20091202\/3709\/Achievement_Design_101.php\">achievement systems<\/a>, and it&#8217;s been keeping me up at night.\u00a0 I&#8217;ve been tossing and turning, trying to figure out a way of applying these concepts to something like ReviewBoard.\u00a0 Is there a model that will encourage users to post review requests early and often?\u00a0 Is there a model that will encourage more thorough reviews from other developers?<\/p>\n<p>An idea eventually sprung to mind&#8230;<\/p>\n<h3>Idea 1:\u00a0 2 Week Games<\/h3>\n<p>In <a href=\"http:\/\/lostgarden.com\/2008\/10\/princess-rescuing-application-slides.html\">a speech he gave<\/a> a few years ago, <a href=\"http:\/\/lostgarden.com\">Danc of The Last Garden<\/a> placed the following bet:<\/p>\n<ul>\n<li>If an activity can be learned\u2026<\/li>\n<li>If the player\u2019s  performance can be measured\u2026<\/li>\n<li>If the player can be rewarded or  punished in a timely fashion\u2026<\/li>\n<li><em>Then any activity that meets  these criteria can be turned into a game.<\/em><\/li>\n<\/ul>\n<p>Let&#8217;s work off of this premise.<\/p>\n<p>Modeled on the idea of a sprint or iteration, let&#8217;s say that ReviewBoard has &#8220;games&#8221; that last 2 weeks.<\/p>\n<p>In a game, users score points in the following way:<\/p>\n<ul>\n<li>Posting a review request that eventually gets committed gives the author 1 point<\/li>\n<li>A review request that is given a ship-it, without a single defect found, gives the author <strong>The 1.5 Multiplier<\/strong> on their total points.\u00a0 The 1.5 Multiplier can be stolen by another player if they post a review request that also gets a ship-it without any defects being found.<\/li>\n<li>Any user can find\/file defects on a review request<\/li>\n<li>A defect must be &#8220;confirmed&#8221; by the author, or &#8220;withdrawn&#8221; by the defect-finder.<\/li>\n<li>After a diff has been updated, &#8220;confirmed&#8221; defects can be &#8220;fixed&#8221;.\u00a0 <strong>Each fixed defect gives the defect-finder and author 1 point each.<\/strong><\/li>\n<\/ul>\n<p>After two weeks, a final tally is made, <a href=\"http:\/\/mikeconley.ca\/blog\/2010\/03\/02\/peer-code-review-achievements\/\">achievements \/ badges<\/a> are doled out, and the scores are reset.\u00a0 A new game begins.\u00a0 Users can view their point history and track their performance over time.<\/p>\n<p>Granted, this game is open to cheating.\u00a0 But so is Monopoly.\u00a0 I can reach into the Monopoly bank and grab $500 without anybody noticing.\u00a0 It&#8217;s up to me not to do that, because it invalidates the game.\u00a0 In this case, cheating would only result in bad morale and a poorer piece of software.\u00a0 And since scores are reset every two weeks, what&#8217;s the real incentive to cheat?<\/p>\n<h3>Idea 2:\u00a0 Track My Performance<\/h3>\n<p>I&#8217;ve never built a reputation system before &#8211; but Randy Farmer and Bryce Glass have.\u00a0 <a href=\"http:\/\/oreilly.com\/catalog\/9780596159801\">They&#8217;ve even written a book about it<\/a>.<\/p>\n<p>Just browsing through their site, I&#8217;m finding quotes that suggests that there are some potential problems with my two week game idea.\u00a0 In particular, I have not considered the potentially harmful effects of displaying &#8220;points&#8221; publicly on a leader-board.<\/p>\n<p>According to Farmer \/ Glass:<\/p>\n<blockquote>\n<div>\n<p>It&#8217;s still too early to speak in absolutes about the design of  social-media sites, but one fact is becoming abundantly clear: ranking  the members of your community-and pitting them one-against-the-other in a  competitive fashion-is typically a bad idea. Like the fabled <em>djinni<\/em> of yore, leaderboards on your site promise riches (comparisons!  incentives! user engagement!!) but often lead to undesired consequences.<\/p>\n<\/div>\n<\/blockquote>\n<p><a href=\"http:\/\/buildingreputation.com\/doku.php?id=Chapter_7#Chap_7-Leaderboards_Considered_Harmful\">They go into more detail here<\/a>.<\/p>\n<p>Ok, so let&#8217;s say that they&#8217;re right.\u00a0 Then how about instead of pitting the reviewers against one another, I have the reviewers compete against themselves?<\/p>\n<p>Ever played Wii Sports?\u00a0 It tracks player performance on various games and displays it on a chart.\u00a0 It&#8217;s really easy to see \/ track progress over time.\u00a0 It&#8217;s also an incentive to keep performance up &#8211; because nobody wants to go below the &#8220;Pro&#8221; line.<\/p>\n<p>So how about we just show users a report of their performance over fixed time intervals&#8230;with fancy jQuery charts, etc?<\/p>\n<h3>So what?<\/h3>\n<p>Are either of these ideas useful?\u00a0 Would they increase the number of defects found per review request?\u00a0 Would they increase the frequency and speed of reviews?\u00a0 Would they improve user perception of peer code review?\u00a0 Would it be ignored?\u00a0 Or could it harm a team of developers?\u00a0 What are the benefits and drawbacks?<\/p>\n<p>If anything, it&#8217;d give ReviewBoard some ability to record metrics, which is handy when you want to show the big boss how much money you&#8217;re saving with code review.<\/p>\n<p>Might be worth looking into.\u00a0 Thoughts?<\/p>\n","protected":false},"excerpt":{"rendered":"<p>A little while back, I wrote about an idea that a few of us had been bouncing around:\u00a0 peer code review achievements. It started out as a bit of Twitter fun &#8211; but now it has evolved, and actually become a contender for my Masters research. So I&#8217;ve been reading up on reputation and achievement [&hellip;]<\/p>\n","protected":false},"author":4,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[454,626],"tags":[632,501,479,631,633,630,503,504],"class_list":["post-1089","post","type-post","status-publish","format-standard","hentry","category-code-reviews","category-research-computer-science-technology","tag-achievement","tag-code-review","tag-game","tag-karma","tag-points","tag-reputation","tag-research","tag-reviewboard"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/prmTy-hz","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/posts\/1089","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/comments?post=1089"}],"version-history":[{"count":8,"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/posts\/1089\/revisions"}],"predecessor-version":[{"id":3185,"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/posts\/1089\/revisions\/3185"}],"wp:attachment":[{"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/media?parent=1089"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/categories?post=1089"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/tags?post=1089"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}