1. As you mentioned, the impact of ‘reviewer influence’ upon one another. I’ve found that in any creative endeavor, the feedback you get from a given person will often change depending on whether they know what another person thinks. It probably has a lot to do with our inate sense of ‘role’ or ‘hierarchy’ within a social group: If a person of strong standing has expressed a certain opinion, then many people are going to be hesitant to express an opposing viewpoint.

2. WRT ‘Reviewee preparation’ another interesting experiment might be to have reviews performed at random vs. reviews performed based on certain criteria. If a developer knows that any given changeset might be subject to review, as opposed to only changesets which meet some deterministic criteria, will that affect the overall code quality? I would suspect that if developers knew that some randomly selected subset of their commits were going to be subjected to a thorough review (I can’t help but think of airport screening as an analogous example), they might be more careful in reviewing all of their commits.

In fact, I think the (self-)selection bias you mention might even be the entire story — more thoughtful developers will probably want to discuss their code.

However there’s one thing I think you should add to your experiment. It’s not enough to measure whether there’s a “drop” in defects, because it’s likely with multiple reviewers to simply find *different* defects. So I think you want to measure specifically WHICH defects each group found rather than the NUMBER of defects, and furthermore if you do want to summarize you should include things like defect severity.

This suggestion comes from experience — in the Cisco study we didn’t measure either of these things and in retrospect it really hampered our ability to make conclusions!

Let me know when you do this study!

Thanks for the input!

RE 1: True – I may have over-simplified in my post. You’re right: I’ll have to pay attention to what exactly the authors are saying when they’re preparing their code, and *how exactly* the reviews are reacting to it.

RE 3: True, those tools can help. Inspection rate, defect density and defect rate are a decent start, but they only tell me so much. I want more. I want to know *where* my reviewers looked. I want fine granularity. I don’t just want to know how long they spent looking at a particular file – I want to know how long they spent looking at a particular method. I want to know how long they spent looking at an area of code that’s already been heavily reviewed. I want to know how long they spent looking at an area of code that nobody has said a word about. I want a heat map showing how much each area of the code has been looked at.

In short, I want a lot.

1. There are really two different things going on with respect to author preparation: a.) what did the author write and b.) how did the reviewers react? When I am the author of a code review I tend to provide the back story and *not* information about the actual implementation. Example: I’m not much of a PHP wizard, yet I maintain a web site written by someone else in PHP. When I have to go learn a PHP idiom in order to enhance the site, in the resulting code review I’ll annotate the idiom with a comment that indicates “Based on what I’ve seen in the PHP docs and at site this is the correct approach – let me know if I’m way off track.”

Does that prejudice the reviewer? Perhaps, but only in the direction of: “This guy is new to this part of PHP, we need to tread carefully here.”

Even if I were to annotate with comments such as: “The method I’m calling here expects a hash map of this type,” the reaction of the reviewer is what matters. If the reviewer thinks: “Oh, okay, no problem” then that’s a very different end result than: “I’m not gonna’ trust this guy – I’m going to go read the method that is being called.”

2. Your next logical step, that reviewers prejudice each other, is also a possibility, but again to me it depends on the people involved. An additional factor, I suppose, is also the time pressure. I can see how it would be likely for a reviewer to say: “Looks like Joe covered this pretty thoroughly, I’m short on time so I’m gonna’ skim instead of really review.”

3. Tools can help. If some of the reviewers are slackin’ off, a tool that tracks the amount of time they spent in the review will help point that out.

As Jason pointed out: he didn’t set out to study the impact of author preparation, so a controlled experiment would be awesome. Can’t wait to read your results!