Mike Conley's Blog

I’ve been studying for a midterm.  It’s been a little while since I’ve had to write a test like this, and I’d forgotten what it feels like.

As a reward for some long studying, and a rather hard week for both of us, my girlfriend Em and I decided to watch The Lord of the Rings:  The Twin Towers DVD last night.  Three things:

First, it’s better than I remember.  Second, the special effects still hold up.

Third, I’ve realized that studying for an exam might be a lot like how generals prepare for war.  I try to out-think and out-strategize my opponent (the instructor)…what angles are they most likely to attack from?  What are my best defenses?  What can I attack with?  How will they try to surprise me?  What are my weaknesses?  What ammunition do I have?  What might I have to sacrifice?  Will I lose?

In this case, my opponent hadn’t given me much to work with.  No past exams.  No exam outline.  Nothing.  Just a point in the textbook, and everything up to it was fair game.

And so I strapped on my armor this morning (a pair of jeans, a t-shirt, my boots, and my pea coat), sheathed my sword (a 0.5mm mechanical pencil into my pencil case), and prepared for my epic battle.

I would be facing the Dark Lord of Automata Theory this morning.

As I approached the battlefield, a disquieting and familiar cry entered my mind:

Thanks for the support, Gandalf.  I hoped he was wrong.

After surveying the landscape, I breathed a sigh of relief.  Luckily, my opponent hadn’t tried anything too tricky.

The battle began.

I think I did alright.

Be the first to like.

Like

Unlike

It’s easy to get on the code review band-wagon, and tout it as the “silver bullet” for bugs, or the key to developing awesome, elegant software, etc.  It’s easy to get carried away, and forget that code review should probably be accompanied by rigorous testing, static analysis, and security integration from day one.

While the purpose of this blog post by Shawn Hernan from Microsoft may be to attack or question the merits of open source software, I see it as an interesting discussion on the role of code review in software engineering and how it relates to writing secure code.

Insert your own joke about Microsoft security here.  I, personally, think their IE team should read Shawn’s post.

Particularly interesting is one of the comments to the post by “danclarke_2000″:

I think another point is diminishing returns of code review..  Each extra code review brings less value than the preeding; review comments can already be known and awaiting action, not important enough to change etc

having extra eyes reviewing code means generating extra code review output.  Here is the true cost, all the code review comments of the many eyes have to pass through the bottleneck of the few people who have authority to make changes.  As each extra review has less value, processing the extra reviews has a higher and higher opportunity cost.

Sound kind of familiar?

Anyhow, Hernan’s post is an interesting read.  Click here to check it out.

UPDATE:

Here’s a quote from Joshua Bloch of Google on a similar topic:

…We programmers need all the help we can get, and we should never assume otherwise. Careful design is great. Testing is great. Formal methods are great. Code reviews are great. Static analysis is great. But none of these things alone are sufficient to eliminate bugs: They will always be with us. A bug can exist for half a century despite our best efforts to exterminate it. We must program carefully, defensively, and remain ever vigilant.

Read the entire post here.

5 people like this post.

Like

Unlike

I was just munching on some cereal and reading one of my (many) Peanuts books.

Sporadically, throughout the late 1960′s strips, Snoopy can be seen working on a novel, and receiving input from Charlie Brown, Linus, Lucy, and the rest of the gang.

It turns out that, a while back, someone went to the trouble to compile the complete text.  Here is Snoopy’s novel, copypasta’d from here:

It Was A Dark And Stormy Night

by Snoopy

Part I

It was a dark and stormy night. Suddenly, a shot rang out!

A door slammed. The maid screamed.

Suddenly, a pirate ship appeared on the horizon!

While millions of people were starving, the king lived in

luxury. Meanwhile, on a small farm in Kansas, a boy was

growing up.

Part II

A light snow was falling, and the little girl with the

tattered shawl had not sold a violet all day.

At that very moment, a young intern at City Hospital

was making an important discovery. The mysterious patient

in Room 213 had finally awakened. She moaned softly.

Could it be that she was the sister of the boy in Kansas

who loved the girl with the tattered shawl who was the

daughter of the maid who had escaped from the pirates?

The intern frowned.

“Stampede!” the foreman shouted, and forty thousand

head of cattle thundered down on the tiny camp. The two

men rolled on the ground grappling beneath the murderous

hooves. A left and a right. A left. Another left and right.

An uppercut to the jaw. The fight was over. And so the

ranch was saved.

The young intern sat by himself in one corner of the

coffee shop. he had learned about medicine, but more

importantly, he had learned something about life.

THE END

And here’s a description of Snoopy’s desired cover art:

“How about a bunch of pirates and foreign legionnaires fighting some cowboys with some lions and tigers and elephants leaping through the air at this girl who is tied to a submarine?”

-Snoopy

Now that’s some damn fine writing.

          It Was A Dark And Stormy Night
          by Snoopy

          Part I

   It was a dark and stormy night. Suddenly, a shot rang out!
A door slammed. The maid screamed.
   Suddenly, a pirate ship appeared on the horizon!
   While millions of people were starving, the king lived in
luxury. Meanwhile, on a small farm in Kansas, a boy was
growing up.

          Part II

   A light snow was falling, and the little girl with the
tattered shawl had not sold a violet all day.
   At that very moment, a young intern at City Hospital
was making an important discovery. The mysterious patient
in Room 213 had finally awakened. She moaned softly.
   Could it be that she was the sister of the boy in Kansas
who loved the girl with the tattered shawl who was the
daughter of the maid who had escaped from the pirates?
The intern frowned.
   "Stampede!" the foreman shouted, and forty thousand
head of cattle thundered down on the tiny camp. The two
men rolled on the ground grappling beneath the murderous
hooves. A left and a right. A left. Another left and right.
An uppercut to the jaw. The fight was over. And so the
ranch was saved.
   The young intern sat by himself in one corner of the
coffee shop. he had learned about medicine, but more
importantly, he had learned something about life.

          THE END

2 people like this post.

Like

Unlike

Every time I meet a non-native English speaker, I invariably ask them the same question:

I make fun of other languages all the time.  I can spout out gibberish that sounds like Russian, Chinese, French, etc.  What happens when someone who speaks a different language tries to spout out English gibberish?  What does English gibberish sound like?

Well, I guess I’m not the only one who is curious about it.  Here’s what English possibly sounds like to people from other countries.

This one might be my favourite – 14 seconds in:

Be the first to like.

Like

Unlike

Remember how I wrote a while back that I wanted to write a script to let me do some quick and easy pre-commit continuous integration with the MarkUs project?

Well, I think I just wrote one.

Introducing TestDrive…

TestDrive will fetch a review request, grab the latest diff (yes, found an easy way past the lack of API there), check out a fresh copy of MarkUs, throw down the diff, set it up with some Sqlite3 databases, run your tests, and voila – go to localhost:3000, and you’re running the review request diff.

I’ve been using it myself for about a week or so, and so far, it’s helped me catch a number of bugs that I wouldn’t have caught just by looking at the code in ReviewBoard.  Nice.

Click here to check out TestDrive.

Be the first to like.

Like

Unlike

In my opinion, at least for The MarkUs Project, post-commit code review would probably be analogous to putting on your shoes before your pants.  And though I mentioned earlier that there is plenty of preference for post-commit, I forgot to include this juicy little tidbit.

Click here to read one of the developers of ReviewBoard state his case for pre-commit code review.

To each their own.  But I dig his points.

Be the first to like.

Like

Unlike

I’ve been trying to record myself performing code reviews for The MarkUs Project.

It’s a lot harder than I thought it’d be.  The screencasts are really only useful if I’m saying what I’m thinking, and I’m finding it difficult to maintain stream of consciousness and perform an effective/thorough review.  The last few times I’ve tried it, I find myself blurting an expletive, stopping the recording in frustration, and then starting the review over so that I can do a good, proper job.

I think this is going to take more practice.

Be the first to like.

Like

Unlike

Discussion Plays

I have seen plays that have very clear stories, and very clear plots.  I leave the theatre knowing what has happened, and I can be pretty confident that the people who sat around me in the theatre all got the same message as I did.

I have also seen plays that are completely the opposite.  There doesn’t appear to be a story.  There doesn’t appear to be plot.  There are no real characters.  For these plays, all of a sudden, I have to do the work in order to make sense of it all.  And you can be pretty sure that every single audience member got something different out of it.

I want to talk about this second kind of play.  For now, I’m going to call this kind of play a discussion play, because for me, the best part about these kinds of plays is the discussion I have with my friends afterwards. We’ll all sit down in a restaurant or a cafe, order some food, and try to figure out what the hell we just saw.  Theories are tossed around.  Everybody brings their own unique impressions and observations to the table.  A very rich ecosystem of ideas develops.

Back to Peer Code Reviews

(trust me, this all ties together in the end)

When Jason Cohen did his Peer Review at Cisco Study, he noticed that code that had been prepared by the author for review seemed to have a lower defect density than code that had not been prepared.

What do I mean by prepared?  I’ll let Jason Cohen explain:

The idea of “author preparation” is that authors should annotate their source code before the review begins.  Annotations guide the reviewer through the changes, showing which files to look at first and defending the reason and methods behind each code modification.  The theory is that because the author has to re-think all the changes during the annotation process, the author will himself uncover most of the defects before the review even begins, thus making the review itself more efficient.  Reviewers will uncover problems the author truly would not have thought of otherwise.

(Best Kept Secrets of Peer Code Review, p80-81)

Looking at the data, author preparation does seem to have a palpable effect.  As Cohen notes, “for all reviews with at least one author preparation comment, defects density is never over 30; in fact the most common case is for there to be no defects at all!”.

The study has two explanations for this:

1. Authors gave their code such a thorough look while annotating them, that most defects were eliminated right off the bat.
2. Since authors were actively explaining, or defending their code, this sabotaged the reviewers ability to do their job effectively.

Cohen buys into the first explanation.  He writes:

A survey of the reviews in question show the author is being conscientious, careful, and helpful, and not misleading the reviewer.  Often the reviewer will respond to or ask a question or open a conversation on another line of code, demonstrating that he was not dulled by the author’s annotations.

I have huge respect for this study.  But I don’t entirely buy this explanation.  As Cohen later mentioned in an email to me, this conclusion is not derived from a controlled experiment, and also suffers from selection bias.

Back to those Discussion Plays

One of the worst things that can happen to me before going into a discussion play is for someone who has already seen it to tell me their impressions of what they thought was going on.  As soon as I hear their opinion, my own objectivity is compromised.  Whether I want to or not, I’ll have their impressions in the back of my mind, and I’ll be using it as a measuring stick or reference point for my own opinions and critiques. They’ve carved a cognitive path through the work, and I’m doomed to notice that path, and react to it.

This is horrible.  This limits me.  This more or less hobbles my ability to contribute something unique to the pool of ideas and criticisms in the after-play discussion.  Every impression I have is tainted by someone else’s first impression.

Don’t get me wrong – I love hearing about everyone’s impressions.  But after I have formed my own. This way, I believe we cover more ground.  A group of us watching a discussion play will carve unique cognitive paths through the work without influencing one another.  When we finally open up and present these paths and ideas to one another over food and drink, I believe we cover more ground.

I have no data to back this up.  Only years of theatre-going experience.

A Code Review Anecdote

I recently received an email from a colleague of mine.  She wanted me to go over some of her Javascript to make sure it was up to snuff, since she was relatively new to the language.  I noticed that she had also sent a copy of the email to another developer who has pretty sharp Javascript chops.

When I finally had some free time, I went back to her email to write up the review.  I felt bad – it was late, and the other reviewer hadn’t made a peep on the email thread, and she was hoping to use the code relatively soon.  So I dove in, wrote my review, and sent it off.

A little while later, the other developer sent me his review, saying:

And here was my answer, which I didn’t send to you so as not to influence your reply.  ;)

So the author of the code received two unique reviews, and neither of them had influenced the other.  When I read his review, I noticed that we covered some similar ground, but a lot of unique ground as well.  I suspect this wouldn’t have been the case had he sent his review to me first.

The Hypothesis

I hypothesize that author preparation in code review sabotages reviewers abilities to objectively carve their own unique cognitive paths through the code.  They see things from the author’s point of view, and this dulls their critical eye.  Because of this, I believe fewer defects are detected.

I will take this hypothesis one step further.

I suspect any review, by the author or otherwise, will taint future reviews.  If someone has already reviewed some code, I suspect this review will impact and possibly limit the ability of other reviewers to look at the code objectively.  Like author preparation, I suspect this prevents reviewers from getting their own unique, valuable first impressions of the code.  And I suspect that this causes some defects to go undetected.

Testing This Hypothesis

It’s a simple idea really.  Take a chunk of code, and get some number of developers to review it.  Take this same code, add some author preparation comments, and get more developers to review it.  Do all of the normal balancing, etc.

The question:  do the number of detected defects drop?  If so, this looks like evidence that author preparation sabotages review ability.

Take the experiment one step further.  Take some code, have someone else review it, and then have participants review this code, having seen the first review.  What happens to the number and type of defects that they find?  What happens if they don’t see that initial review?  What yields high defect detection?

Sounds doable.  Sounds interesting.  Sounds like something that would answer a few questions.

Implications and Ideas

So what if one or both of my hypotheses are true?  What does this mean for peer code review?

Well, if author preparation alone sabotages review ability, then the answer is simple:  don’t let the authors prepare the review.  The code goes up, and they stay silent.

But what if both are true?

An idea:  how about I tweak MarkUs’s ReviewBoard so that reviewers cannot see what other reviewers have said until they’ve given one review?  What would happen to the defect detection numbers?  Would reviewers react negatively to this?  Would there be lots of repetition in the comments?  Sounds like something worth looking into.

I’d love to hear some thoughts on this.  Anyone?

Be the first to like.

Like

Unlike

I just found this on Slashdot – a guy named Fabien Sanglard reviews the code for the iPhone DOOM port.  This is cool – I wish I had time to read it thoroughly.  It’ll have to wait until after my assignments are done.

Looks like this guy has a thing for FPS’s – he’s also done a code review for “Wolfiphone” (Wolfenstein for iPhone) and the original Quake.

I like this idea.  Why aren’t there more of these on the web?  Or am I not looking hard enough?

Be the first to like.

Like

Unlike