I have seen plays that have very clear stories, and very clear plots.\u00a0 I leave the theatre knowing what has happened, and I can be pretty confident that the people who sat around me in the theatre all got the same message as I did.<\/p>\n
I have also seen plays that are completely the opposite.\u00a0 There doesn’t appear to be a story.\u00a0 There doesn’t appear to be plot.\u00a0 There are no real characters.\u00a0 For these plays, all of a sudden, I have to do the work in order to make sense of it all.\u00a0 And you can be pretty sure that every single audience member got something different out of it.<\/p>\n
I want to talk about this second kind of play.\u00a0 For now, I’m going to call this kind of play a discussion play<\/em>, because for me, the best part about these kinds of plays is the discussion I have with my friends afterwards. We’ll all sit down in a restaurant or a cafe, order some food, and try to figure out what the hell we just saw.\u00a0 Theories are tossed around.\u00a0 Everybody brings their own unique impressions and observations to the table.\u00a0 A very rich ecosystem of ideas develops.<\/p>\n (trust me, this all ties together in the end)<\/p>\n When Jason Cohen<\/a> did his Peer Review at Cisco Study<\/a>, he noticed that code that had been prepared by the author for review seemed to have a lower defect density than code that had not been prepared.<\/p>\n What do I mean by prepared?\u00a0 I’ll let Jason Cohen explain:<\/p>\n The idea of “author preparation” is that authors should annotate their source code before the review begins.\u00a0 Annotations guide the reviewer through the changes, showing which files to look at first and defending the reason and methods behind each code modification.\u00a0 The theory is that because the author has to re-think all the changes during the annotation process, the author will himself uncover most of the defects before the review even begins, thus making the review itself more efficient.\u00a0 Reviewers will uncover problems the author truly would not have thought of otherwise.<\/p>\n (Best Kept Secrets of Peer Code Review, p80-81)<\/p><\/blockquote>\n Looking at the data, author preparation does seem to have a palpable effect.\u00a0 As Cohen notes, “for all reviews with at least one author preparation comment, defects density is never over 30; in fact the most common case is for there to be no defects at all!”.<\/p>\n The study has two explanations for this:<\/p>\n Cohen buys into the first explanation.\u00a0 He writes:<\/p>\n A survey of the reviews in question show the author is being conscientious, careful, and helpful, and not misleading the reviewer.\u00a0 Often the reviewer will respond to or ask a question or open a conversation on another line of code, demonstrating that he was not dulled by the author’s annotations.<\/p><\/blockquote>\n I have huge respect for this study.\u00a0 But I don’t entirely buy this explanation.\u00a0 As Cohen later mentioned in an email to me, this conclusion is not derived from a controlled experiment, and also suffers from selection bias.<\/p>\n One of the worst things that can happen to me before going into a discussion play is for someone who has already seen it to tell me their impressions of what they thought was going on.\u00a0 As soon as I hear their opinion, my own objectivity is compromised.\u00a0 Whether I want to or not, I’ll have their impressions in the back of my mind, and I’ll be using it as a measuring stick or reference point for my own opinions and critiques. They’ve carved a cognitive path through the work, and I’m doomed to notice that path, and react to it.<\/p>\n This is horrible.\u00a0 This limits me.\u00a0 This more or less hobbles my ability to contribute something unique to the pool of ideas and criticisms in the after-play discussion.\u00a0 Every impression I have is tainted by someone else’s first impression.<\/p>\n Don’t get me wrong – I love hearing about everyone’s impressions.\u00a0 But after I have formed my own.<\/em> This way, I believe we cover more ground.\u00a0 A group of us watching a discussion play will carve unique cognitive paths through the work without influencing one another.\u00a0 When we finally open up and present these paths and ideas to one another over food and drink, I believe we cover more ground.<\/p>\n I have no data to back this up.\u00a0 Only years of theatre-going experience.<\/p>\n I recently received an email from a colleague of mine.\u00a0 She wanted me to go over some of her Javascript to make sure it was up to snuff, since she was relatively new to the language.\u00a0 I noticed that she had also sent a copy of the email to another developer who has pretty sharp Javascript chops.<\/p>\n When I finally had some free time, I went back to her email to write up the review.\u00a0 I felt bad – it was late, and the other reviewer hadn’t made a peep on the email thread, and she was hoping to use the code relatively soon.\u00a0 So I dove in, wrote my review, and sent it off.<\/p>\n A little while later, the other developer sent me his review, saying:<\/p>\n And here was my answer, which I didn’t send to you so as not to influence your reply. \u00a0\ud83d\ude09<\/p><\/blockquote>\n So the author of the code received two unique reviews, and neither of them had influenced the other.\u00a0 When I read his review, I noticed that we covered some similar ground, but a lot of unique ground as well.\u00a0 I suspect this wouldn’t have been the case had he sent his review to me first.<\/p>\n I hypothesize that author preparation in code review sabotages reviewers abilities to objectively carve their own unique cognitive paths through the code.\u00a0 They see things from the author’s point of view, and this dulls their critical eye.\u00a0 Because of this, I believe fewer defects are detected.<\/p>\n I will take this hypothesis one step further.<\/p>\n I suspect any <\/em>review, by the author or otherwise, will taint future reviews.\u00a0 If someone has already reviewed some code, I suspect this review will impact and possibly limit the ability of other reviewers to look at the code objectively.\u00a0 Like author preparation, I suspect this prevents reviewers from getting their own unique, valuable first impressions of the code.\u00a0 And I suspect that this causes some defects to go undetected.<\/p>\n It’s a simple idea really.\u00a0 Take a chunk of code, and get some number of developers to review it.\u00a0 Take this same code, add some author preparation comments, and get more developers to review it.\u00a0 Do all of the normal balancing, etc.<\/p>\n The question:\u00a0 do the number of detected defects drop?\u00a0 If so, this looks like evidence that author preparation sabotages review ability.<\/p>\n Take the experiment one step further.\u00a0 Take some code, have someone else review it, and then have participants review this code, having seen the first review.\u00a0 What happens to the number and type of defects that they find?\u00a0 What happens if they don’t see that initial review?\u00a0 What yields high defect detection?<\/p>\n Sounds doable.\u00a0 Sounds interesting.\u00a0 Sounds like something that would answer a few questions.<\/p>\n So what if one or both of my hypotheses are true?\u00a0 What does this mean for peer code review?<\/p>\n Well, if author preparation alone sabotages review ability, then the answer is simple:\u00a0 don’t let the authors prepare the review.\u00a0 The code goes up, and they stay silent.<\/p>\n But what if both are true?<\/p>\n An idea:\u00a0 how about I tweak MarkUs’s ReviewBoard so that reviewers cannot see what other reviewers have said until they’ve given one review?\u00a0 What would happen to the defect detection numbers?\u00a0 Would reviewers react negatively to this?\u00a0 Would there be lots of repetition in the comments?\u00a0 Sounds like something worth looking into.<\/p>\n I’d love to hear some thoughts on this.\u00a0 Anyone?<\/p>\n","protected":false},"excerpt":{"rendered":" Discussion Plays I have seen plays that have very clear stories, and very clear plots.\u00a0 I leave the theatre knowing what has happened, and I can be pretty confident that the people who sat around me in the theatre all got the same message as I did. I have also seen plays that are completely […]<\/p>\n","protected":false},"author":4,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[454,9],"tags":[600,593,501,1218,594,113,599,460,550,601,596,595,503,504,598,459,597,134],"class_list":["post-1004","post","type-post","status-publish","format-standard","hentry","category-code-reviews","category-theater","tag-author-preparation","tag-cisco-study","tag-code-review","tag-code-reviews","tag-cognition","tag-criticism","tag-first-impressions","tag-jason-cohen","tag-markus","tag-objectivity","tag-plays","tag-plot","tag-research","tag-reviewboard","tag-sabotage","tag-smart-bear","tag-stories","tag-theatre"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/prmTy-gc","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/posts\/1004","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/comments?post=1004"}],"version-history":[{"count":11,"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/posts\/1004\/revisions"}],"predecessor-version":[{"id":3198,"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/posts\/1004\/revisions\/3198"}],"wp:attachment":[{"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/media?parent=1004"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/categories?post=1004"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/tags?post=1004"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Back to Peer Code Reviews<\/h3>\n
\n
Back to those Discussion Plays<\/h3>\n
A Code Review Anecdote<\/h3>\n
The Hypothesis<\/h3>\n
Testing This Hypothesis<\/h3>\n
Implications and Ideas<\/h3>\n