Mike Conley's Blog

From my software development experience, there are a few different words for the generic notion of a “problem”.

A bug or a defect, for example, is defined as the following from Wikipedia:

A software bug is the common term used to describe an error flaw, mistake, failure, or fault in a computer program or system that produces an incorrect or unexpected result, or causes it to behave in unintended ways. Most bugs arise from mistakes and errors made by people in either a program’s source code or its design, and a few are caused by compilers producing incorrect code.

An issue goes a level higher – a bug is an issue, but an issue might not be a bug.  Wikipedia says:

In computing, the term issue is a unit of work to accomplish an improvement in a system. An issue could be a bug, a requested feature, task, missing documentation, and so forth. The word “issue” is popularly misused in lieu of “problem.” This usage is probably related.

Where am I going with all of this?

Well, remember when I said I was going to add defect reporting/tracking capabilities to Review Board?  I asked for some feedback on my UI mockups on the developer mailing list, and an interesting conversation on terminology erupted.

Anyhow, the long and the short of it is – we’re going to be calling “problems still existing within a review request revision” issues.  And this is distinct from the sort of thing that might show up in an issue tracker.

Maybe down the line, we’ll have a way for administrators to set their own word for it.  From the thread, it sounds like everybody and their brother has their own favourite terminology.  ”Issue” will have to do for now.

Thanks again to everyone on the list who contributed to the conversation.

In my last post, I talked about an extension for Review Board that would allow users to register “defects”, “TODOs” or “problems” with code that’s up for review.

After chatting with the lead RB devs for a bit, we’ve decided to scrap the extension.

[audible gasp, booing, hissing]

Instead, we’re just going to put it in the core of Review Board.

[thundering applause]

Defects

Why is this useful?  I’ve got a few reasons for you:

1. It’ll be easier for reviewees to keep track of things left to fix, and similarly, it’ll be harder for reviewees to accidentally skip over fixing a defect that a reviewer has found
2. My statistics extension will be able to calculate useful things like defect detection rate, and defect density
3. Maybe it’s just me, but checking things off as “fixed” or “completed” is really satisfying
4. Who knows, down the line, I might code up an extension that lets you turn finding/closing defects into a game

However, since we’re adding this to the core of Review Board, we have to keep it simple.  One of Review Board’s biggest strengths is in its total lack of clutter.  No bells.  No whistles.  Just the things you need to get the job done.  Let the extensions bring the bells and whistles.

So that means creating a bare-bones defect-tracking mechanism and UI, and leaving it open for extension.  Because who knows, maybe there are some people out there who want to customize what kind of defects they’re filing.

I’ve come up with a design that I think is pretty simple and clean.  And it doesn’t rock the boat – if you’re not interested in filing defects, your Review Board experience stays the same.

Filing a Defect

I propose adding a simple checkbox to the comment dialog to indicate that this comment files a defect, like so:

No bells. No whistles. Just a simple little checkbox.

While I’m in there, I’ll try to toss in some hooks so that extension developers can add more fields – for example, the classification or the priority of the defect.  By default, however, it’s just a bare-bones little checkbox.

So far, so good.  You’ve filed a defect.  Maybe this is how it’ll look like in the in-line comment viewer:

A defect has been reported!

Two Choices

A reviewer can file defects reports, and the reviewee is able to act on them.

Lets say I’m the reviewee.  I’ve just gotten a review, and I’ve got my editor / IDE with my patch waiting in the background.  I see a few defect reports have been filed.  For the ones I completely agree with, I fix them in my editor, and then go back to Review Board and mark them as Fixed.

All fixed!

It’s also possible that I might not agree with one or more of the defect reports.  In this case, I’ll reply to the comment to argue my case.  I might also mark the defect report as Pass, which means, “I’ve seen it, but I think I’ll pass on that”.

I think I'll pass on that, thanks.

These comments and defect reports are also visible in the review request details page:

A defect has been filed.

All fixed up.

It's all good - just pass this defect report.

Thoughts?

What do you think?  Am I on the right track?  Am I missing a case?  Does “pass” make sense?  Will this be useful?  I’d love to hear your thoughts.

I just spent the long weekend in Ottawa and Québec City with my parents and my girlfriend Em.

During the long drive back to Toronto from Québec City, I had plenty of time to think about my GSoC project, and where I want to go with it once GSoC is done.

Here’s what I came up with.

Detach Reviewing Time from Statistics

I think it’s a safe assumption that my reviewing-time extension isn’t going to be the only one to generate useful statistical data.

So why not give extension developers an easy mechanism to display statistical data for their extension?

First, I’m going to extract the reviewing-time recording portion of the extension. Then, RB-Stats (or whatever I end up calling it), will introduce it’s own set of hooks for other extensions to register with.  This way, if users want some stats, there will be one place to go to get them.  And if an extension developer wants to make some statistics available, a lot of the hard work will already be done for them.

And if an extension has the capability of combining its data with another extensions data to create a new statistic, we’ll let RB-Stats manage all of that business.

Stopwatch

The reviewing-time feature of RB-Stats will become an extension on its own, and register its data with RB-Stats.  Once RB-Stats and Stopwatch are done, we should be feature equivalent with my demo.

Review Karma

I kind of breezed past this in my demo, but I’m interested in displaying “review karma”.  Review karma is the reviews/review-requests ratio.

But I’m not sure karma is the right word.  It suggests that a low ratio (many review requests, few reviews) is a bad thing.  I’m not so sure that’s true.

Still, I wonder what the impact will be to display review karma?  Not just in the RB-Stats statistics view, but next to user names?  Will there be an impact on review activity when we display this “reputation” value?

FixIt

This is a big one.

Most code review tools allow reviewers to register “defects”, “todos” or “problems” with the code up for review.  This makes it easier for reviewees to keep track of things to fix, and things that have already been taken care of.  It’s also useful in that it helps generate interesting statistics like defect density and defect detection rate (assuming Stopwatch is installed and enabled).

I’m going to tackle this extension as soon as RB-Stats, Stopwatch and Karma are done.  At this point, I’m quite confident that the current extension framework can more or less handle this.

Got any more ideas for me?  Or maybe an extension wish-list?  Let  me know.

If I’ve learned anything from my supervisor, it’s to demo. Demo often. Step out of the lab and introduce what you’ve been working on to the world. Hit the pavement and show, rather than tell.

So here’s a video of me demoing my statistics extension for Review Board.  It’s still in the early phases, but a lot of the groundwork has been taken care of.

And sorry for the video quality.  Desktop capture on Ubuntu turned out to be surprisingly difficult for my laptop, and that’s the best I could do.

So, without further ado, here’s my demo (click here if you can’t see it):

Not bad!  And I haven’t even reached the midterm of GSoC yet.  Still plenty of time to enhance, document, test, and polish.

If you have any questions or comments, I’d love to hear them.

The Primary Goal

From the very beginning, my GSoC project has been mainly focused towards one primary goal:  I want to build an extension for Review Board that will allow me to collect information about how long reviewers actually spend reviewing code.

That’s easier said than done.  When I started, the Review Board extension framework wasn’t really in a state to allow such an extension to exist.

So I’ve been tooling around in the Review Board code for the past 2 months, preparing the framework, and getting it ready to handle my extension.

And last night, it started to work.  I can now give rough estimates on how long a reviewer has spent reviewing code.

How It Works

My extension adds a new table to the database which stores “reviewing sessions”.  Each reviewing session is associated to a particular review request and user, and also has a field to store the number of seconds that a user has spent in review.

I’ve created a TemplateHook that allows me to inject Javascript into key areas of Review Board (in particular, the diff viewer, and the screenshot viewer).  The Javascript does the following:  every 10 seconds, we check to see if the mouse has moved on the body of the HTML document.  If it has, we send an “activity” notification to the server.

The server receives this activity notification through the Web API, and checks to see if the time lapsed since the last session update was greater than 10 seconds.  If it is, we increment the working session by 10 seconds and return a 200 HTTP code.  If it isn’t, we don’t change anything and return a 304 HTTP code.

Next, my extension waits for a user to publish a review.  When it notices that a review is being published, it finds the working session for that user and review request, and then attaches it to the published review.  If the user then starts looking at the diff or screenshots again, a new working session is created.

The result?  A pretty decent estimate of how long a user has spent reviewing the code.  No time gets recorded if the user gets up and has a sandwich.  No time gets recorded if the user is on another tab reading Reddit.

Not bad.  For a first draft, anyhow.

I think I’m going to try to chart the data somehow, so that users can track their inspection rates.  I’ll let you know how that goes.

I can’t honestly say I’ve been doing much test-driven development on Review Board.  Django is a really cool web-framework, but I miss all of the nice testing tools from the Rails ecosystem.

Anyhow, today, I decided to run the tests, and BAM – Segmentation Fault:

Testing Perforce binary diff parsing … SKIP: perforce/p4python is not installed
Testing PerforceTool.get_changeset … SKIP: perforce/p4python is not installed
Testing Perforce empty and normal diff parsing … SKIP: perforce/p4python is not installed
Testing Perforce empty diff parsing … SKIP: perforce/p4python is not installed
Testing PerforceTool.get_file … SKIP: perforce/p4python is not installed
Testing parsing SVN diff with binary file … ok
Testing SVNTool.get_file … Segmentation Fault

Yikes.  Getting a segfault in Python is unusual, and a little jarring.  However, it did let me narrow down the problem a bit:  it must have to do with the pysvn bindings that Review Board uses to talk to Subversion, because those are written in C++ (which can certainly segfault).

The machine I was using had Ubuntu Hardy on it, and the Hardy packages have pysvn 1.5.2-1.  Taking a look at the pysvn homepage, the most recent version is actually 1.7.2.

So, I uninstalled pysvn, downloaded the source for 1.7.2, compiled it, and installed it manually.

Segfault gone.  Tests pass.  Awesome sauce.

According to the timeline, I’m still in the community-bonding period for GSoC.  Coding for my project is supposed to start sometime towards the end of May.

So I’m using the time to do the following:

• Close as many small, easy tickets as I can for the  upcoming Review Board release.  I’ve already posted some patches for review.  More forthcoming.
• Get to know the tools I’ll be using.  Review Board is hosted on Github, and I’m relatively new to the whole DVCS thing.  I’ve been figuring out how to use Git, how to post patches, merging, branching, etc.
• Get to know the area I’ll be working in.  I’ve been figuring out how Django apps organize themselves.  I’ve also drawn up a map of the current state of the extension framework to help me visualize it.
• Get to know the other developers working on Review Board.  I’ve been hanging out in the #reviewboard-soc FreeNode IRC channel.  Very nice, and helpful people to work with.
• Develop a plan of attack for my project

And this last point is the one I want to talk about.

The Review Board Extension Life-Cycle

An extension isn’t just some isolated piece of code that gets crammed into an application.  When you’ve got multiple extensions already installed and running, installing and activating a new extension is like introducing a new animal into an ecosystem.  You have to make sure that your new animal plays nice with the others, and that, in the morning, there won’t be a pile of rotting corpses where your application used to be.

I may have gotten carried away with my metaphor.

So let’s look at what I’m envisioning as the life-cycle for a Review Board extension.  I’ll start right from the top.

Getting the Extension

Ideally, this will work as nicely as WordPress’s implementation:  a Review Board administrator is given a catalog of extensions to choose from within the Administrator interface, and one click later, the desired extensions are downloaded and ready to be installed.

For all you system administrators out there, that last idea might make your toes curl.  A Review Board administrator is not necessarily a system administrator, and the system administrator knows what he/she likes on their machine.  An application that can go and download other applications can be dangerous.  We have to ensure that the application that we’re downloading is the one we’re trying to download.  The last thing we need is some man-in-the-middle to do something cute and bork the code review machine.  And we want to ensure that the extension functions as advertised.  No hidden features.  No self-destruct mechanisms.  No back doors.

Do I have a plan for this part?  Well….no, not really.  I don’t imagine I’ll get that far – I consider it a little out of my scope.  So, for my project, I think it’ll satisfy if the system admin (or Review Board admin) can manually download the extension, decompress it, and place it where Review Board can work with it.  That other stuff can come later (and I’ll try to design so that it can come later easily).

Installing the Extension

Ok, so at this point, we’ve got our extension downloaded in a place where Review Board can see it.

So now what?

Now we need to install the extension.  To me, that means letting the extension put its roots into the RB install by creating database tables, preparing initial data, and generally doing everything to make conditions suitable for the extension to function.

When an extension install begins, I imagine it will consider the following questions (in no particular order):

1. Do I (the extension) depend on other extensions to function?  If so, are those extensions present?  If not, let the user know so that they can go get them.
2. Are there some extensions already installed that will conflict with me, or make me behave badly?  If so, let the user know so that they can either remove that conflict, or find an alternative extension.
3. Is the user entirely aware of what I can do?  Make sure that my capabilities, limitations, and behavioural quirks are known to the user.

Once those 3 questions are answered, and everything is looking good for the install, the extension will create the database tables it needs (if any) in order to function.  If anything goes wrong during this process, the database changes will be rolled back, and the user will get a full read out about what went wrong.

If nothing goes wrong, the extension will be installed.  The user might then be asked to set some initial operating parameters for the extension.

Ok great – the extension is installed.  Now what?

Activating the Extension

For something that sounds so dramatic, the explanation about what happens is pretty short:  Review Board simply becomes aware that the extension is activated, and passes data through the necessary hooks in order for the extension to function properly.  Upon activation, the extension should do a quick double-check to ensure that all prerequisites for the extension have been met.  This is because we can’t trust users to activate an extension immediately after downloading them.

The Extension Runs

While it’s activated, the extension will probably react to various events that happen on Review Board.  Tables will be updated.  View methods will be run.  Templates will be rendered.

If anything, ever, goes horribly wrong with an extension, the following will happen:

1. A log entry will be written, dumping the error, and information about what the user was trying to do
2. An error message will be displayed to the user, trying to tell them what exactly happened
3. The extension (and its dependents) will be deactivated.  They will no longer react to events on Review Board.  Their tables will still be there, the settings will still, but the extension will be, in essence, asleep.  Dormant.  Non-reactive.

The administrator could try to reactivate the extension at this point.  They might try to contact the extension developer for support.

The Extension is Un-installed

If the user wants to rid themselves of an extension, they must first deactivate it.  This will put it (and its dependents) in the dormant state.  It just switches them off, nothing else.

Deactivated extensions can then be un-installed.  If the user chooses to un-install the extension, the extension database tables and settings will be wiped out.

The extension itself won’t be deleted though – at least, not within the scope of my project.  The extension files will need to be removed from Review Board manually.

At this point, any dependents that this uninstalled extension had will no longer be able to be activated.

Anyhow, that’s how I envision the life-cycle.  It’s my first go at it, so I’d love to hear some feedback if you have any.

If you didn’t already know, Review Board is an open-source web-based code review tool.  The MarkUs Team has been using Review Board for pre-commit code review for about a year now.  This has given the team a number of advantages:

1. For a team that usually has a 4 month turnover, this allows us to quickly get new team members up to speed with how to contribute to MarkUs.  We review every change that they propose, and give them tips/guidance on how to make it fit in well with the application.  They learn, and the applications code stays healthy.
2. We catch defects before they enter the code base.  Simple as that.
3. We get a good sense of what other people are working on, and what is going on in the code.  Review Board has become a central conversation and learning hub for the developers on the MarkUs team.

So, the long and the short of it:  I like Review Board.  Review Board helps us write better code.  I want to make Review Board better.

So what am I proposing?

How to Avoid A Bloated Software Monster

You can never make some people happy.

No matter how decent your software is, someone will eventually come up to you and say:

Wow!  Your software would be perfect if only it had feature XYZ!  Sadly, because you don’t have feature XYZ, I can’t use it.  Please implement, k thx!

And so you either have to politely say “no”, and lose that user, or say “yes”, and add feature XYZ to the application.  And for users out there who don’t need, or don’t care about feature XYZ, that new feature just becomes a distraction and adds no value.  Make this happen a bunch of times, and you’ve got yourself a bloated mutha for a piece of software.

And we don’t want a bloated piece of software.  But we do want to make our users happy, and provide feature XYZ for them if they want it.

So what’s the solution?  We provide an extension framework (which is also sometimes called a plug-in architecture).

An extension framework allows developers to easily expand a piece of software to do new things.  So, if a user wants feature XYZ, we (or someone else) just creates and make available an extension that implements the feature.  The user installs the extension, activates it, and bam – our user is happy as a clam with their new feature.

And if we make it super-easy to develop them, third-party developers can write new, wonderful, interesting extensions to do things that…well, we wouldn’t have considered in the first place. It’s a new place for innovation.  What’s that old cliché?

If you build it [the plug-in framework], they will come [the third-party developers who write awesome things]

And the developers do come.  Just look at Firefox add-ons or WordPress plugins.  Entire ecosystems of extensions, doing things that the original developers would probably have never dreamed of doing on their own.  Hell, I’ve even written a Firefox add-on. And users love customizing their Firefox / WordPress with those extensions.  It adds value.

So we get wins all over the place:

• Our user gets their feature
• The software gets more attractive because it’s flexible and customizable
• The original software developers get to focus on the core piece of software, and let the third-party developers focus on the fringe features

And this is where I think I can help Review Board.

(Before I go on, if you’re interested, here’s another article on the how and the why of plug-in architectures)

Review Board Extensions

So if you look at the Review Board Wiki, or glance at the mailing lists you see numerous requests from users for new features, for example:

It would be nice if the review board had a “next comment” button that is always available to click, or had a collapse/expand button. This would make it easier to see other people’s comments in cases like this.

…

It will be nice to have post-commit support. Instead of every post-commit review being a separate URL, if we could setup default rules for post-commit reviews to update an existing review providing the diff-between-diff features, it would be very useful.

The Review Board developers could smell the threat of bloated feature-creep from a mile away.  So, in a separate branch, they began working on integrating an extension framework into Review Board.

The extension branch, however, has been gathering dust, while the developers focus on more critical patches and releases.

My GSoC proposal is to finish off a draft of the extension framework, document it, and build a very simple extension for it.  My simple extension will allow me to record basic statistics about Review Board reviewers – for example, how long they spend on a particular review, their inspection rate, etc.

Having been a project lead MarkUs for so long, it’s going to be a good experience to be back on “the bottom” – to be the new developer who doesn’t entirely have a sense of the application code yet.  It’s going to be good to go code spelunking again.  I’ve done some preliminary explorations, and it’s reminding me of my first experiences with MarkUs.  Like a submarine using its sonar, I’m slowly getting a sense of the code terrain.

I’ll let you know what my first few sweeps find.

One of my potential research experiments involves augmenting ReviewBoard, and so, I’ve been studying the ReviewBoard code.

I want to give ReviewBoard the ability to record certain statistics – for example, the number of defects in a review request, number of defects found per reviewer, defect density, inspection rate, etc…

It turns out I’m not the only one who wants this to happen.  Check out this thread.

So it turns out that the devs building ReviewBoard are planning on integrating an extension framework.  Theoretically, users could then write their own extensions to customize ReviewBoard as they see fit.  Think WordPress plugins, but for a code review tool.  Neat.

According to the above link:

A lot of this [extension framework] already exists in a private development branch, and it will be one of our primary focuses as soon as 1.0 goes out.

However, I found the following quote from a relatively recent mailing list message:

[with regards to the extensions framework]…We have to finish building it, testing
it, getting other features we have planned into the release, and performing
the release. So, we’re looking at a year or more. But I don’t see how we’d
get the functionality you want short-term.

Hrmph.  My thinking:  maybe I can help them with it, and we can crank this puppy out sooner.

So where is this private branch?  It actually took a little bit of detective work…

Detective Work

I started by looking at the ReviewBoard repo on Github, which I assumed that the extensions branch would fork from.

I was right – check out the fork network.  There’s a branch helpfully labeled “extensions” forked by davidt, one of the ReviewBoard devs.

Two observations:

1. This fork hasn’t been updated in about a month
2. The extensions stuff references stuff from Djblets which doesn’t exist in the Djblets master branch

So I perform the same exercise – I glance at the Djblets fork network, and see the handily named “extensions” fork that is in davidt’s git repo.  Nice.

Ok, so those are the forks/branches that I’m going to grab.

A Fluke

So, funny story:  I was able to get this “extension” version of ReviewBoard working on my work machine.  But it was by a complete fluke – this extension version of Djblets is actually missing some pieces.  The only reason it worked for me was because I accidentally ran with the master checkout of Djblets first.  This errored out, and also generated a bunch of compiled Python.  I then switched to the extensions branch, which left the compiled Python behind.  The compiled Python filled all of the missing pieces that are in the extensions branch of Djblets, and so it ran.

For a guy who is pretty new to both Git and Django, I found that problem pretty frustrating to track down.  Especially when I started writing up the checkout instructions… big thanks to Zuzel for her invaluable (and patient) Django guidance.

Anyhow, if you’re interested in the steps to failure, here they are…

Setting Up (for failure)

I’m running Ubuntu Karmic.  I have Git installed.

1. First of all, this document is invaluable
2. Check out django_evolution from SVN to some local directory
3. Put path to django-evolution into PYTHONPATH (export PYTHONPATH=$PYTHONPATH:/…/django-evolution-read-only/)
4. Get Python Setuptools (sudo apt-get install python-setuptools)
5. With easy_install, get nose, paramiko, recaptcha-client, Sphinx (sudo easy_install nose paramiko recaptcha-client sphinx)
6. Clone original master Djblets repo (git clone git://github.com/djblets/djblets.git) and CD into djblets directory
7. Add davidt’s remote Djblets branch (git remote add davidt git://github.com/davidt/djblets.git)
8. Fetch davidt’s repo  (git fetch davidt)
9. Track davidt’s extension branch for djblets (git branch extensions davidt/extensions)
10. In djblets root dir, type:  sudo python setup.py develop
11. This should install Django too.  The ReviewBoard install doc says to install from Subversion, but this will do for now.
12. Clone ReviewBoard git repo (git clone git://github.com/reviewboard/reviewboard.git) and Cd into it
13. Add davidt’s remote Reviewboard branch (git remote add davidt git://github.com/davidt/reviewboard.git)
14. Fetch davidt’s repo  (git fetch davidt)
15. Track davidt’s extensions branch of reviewboard (git branch extensions davidt/extensions)
16. Switch to extensions branch (git checkout extensions)
17. So reviewboard/reviewboard/extensions should have stuff in it.
18. python ./contrib/internal/prepare-dev.py
19. Manually create the directory reviewboard/reviewboard/htdocs/media/ext.  ReviewBoard expects it and will complain if it isn’t there.
20. Run ./contrib/internal/devserver.sh
21. Go to localhost:8080

If you’re like me, you’ll see an error about “no module named urls”.  This is because the extension branch of Djblets is missing urls.py in its djblets/log directory.  As you can see, urls.py exists in the master Djblets branch.

So I’ve accidentally got it running.  But this version of ReviewBoard I’ve created is like Frankenstein’s monster – it’s a sad, stitched together monstrosity that should probably never see the light of day.

I tried to merge the master branch of Djblets and ReviewBoard into the extensions branch, and then realized that I had no idea what I was doing.  The last guy you want doing a merge is a guy who doesn’t know what he’s doing.

So that’s where I stand with it.  Kind of a sad affair, really.  I’ve posted on the ReviewBoard developer mailing list asking for some guidance, but all is quiet so far.

In the meantime, I’ll continue studying the code when I can.  The extensions stuff is actually pretty far along, in my opinion – but what do I know, I’ve never built one.  I’ll also take a peek at some other examples of extension frameworks to see what I can learn about implementation (that’s right – I’m talking about you, WordPress).

I’ll let you know what I find.

A little while back, I wrote about an idea that a few of us had been bouncing around:  peer code review achievements.

It started out as a bit of Twitter fun – but now it has evolved, and actually become a contender for my Masters research.

So I’ve been reading up on reputation and achievement systems, and it’s been keeping me up at night.  I’ve been tossing and turning, trying to figure out a way of applying these concepts to something like ReviewBoard.  Is there a model that will encourage users to post review requests early and often?  Is there a model that will encourage more thorough reviews from other developers?

An idea eventually sprung to mind…

Idea 1:  2 Week Games

In a speech he gave a few years ago, Danc of The Last Garden placed the following bet:

• If an activity can be learned…
• If the player’s performance can be measured…
• If the player can be rewarded or punished in a timely fashion…
• Then any activity that meets these criteria can be turned into a game.

Let’s work off of this premise.

Modeled on the idea of a sprint or iteration, let’s say that ReviewBoard has “games” that last 2 weeks.

In a game, users score points in the following way:

• Posting a review request that eventually gets committed gives the author 1 point
• A review request that is given a ship-it, without a single defect found, gives the author The 1.5 Multiplier on their total points.  The 1.5 Multiplier can be stolen by another player if they post a review request that also gets a ship-it without any defects being found.
• Any user can find/file defects on a review request
• A defect must be “confirmed” by the author, or “withdrawn” by the defect-finder.
• After a diff has been updated, “confirmed” defects can be “fixed”.  Each fixed defect gives the defect-finder and author 1 point each.

After two weeks, a final tally is made, achievements / badges are doled out, and the scores are reset.  A new game begins.  Users can view their point history and track their performance over time.

Granted, this game is open to cheating.  But so is Monopoly.  I can reach into the Monopoly bank and grab $500 without anybody noticing.  It’s up to me not to do that, because it invalidates the game.  In this case, cheating would only result in bad morale and a poorer piece of software.  And since scores are reset every two weeks, what’s the real incentive to cheat?

Idea 2:  Track My Performance

I’ve never built a reputation system before – but Randy Farmer and Bryce Glass have.  They’ve even written a book about it.

Just browsing through their site, I’m finding quotes that suggests that there are some potential problems with my two week game idea.  In particular, I have not considered the potentially harmful effects of displaying “points” publicly on a leader-board.

According to Farmer / Glass:

It’s still too early to speak in absolutes about the design of social-media sites, but one fact is becoming abundantly clear: ranking the members of your community-and pitting them one-against-the-other in a competitive fashion-is typically a bad idea. Like the fabled djinni of yore, leaderboards on your site promise riches (comparisons! incentives! user engagement!!) but often lead to undesired consequences.

They go into more detail here.

Ok, so let’s say that they’re right.  Then how about instead of pitting the reviewers against one another, I have the reviewers compete against themselves?

Ever played Wii Sports?  It tracks player performance on various games and displays it on a chart.  It’s really easy to see / track progress over time.  It’s also an incentive to keep performance up – because nobody wants to go below the “Pro” line.

So how about we just show users a report of their performance over fixed time intervals…with fancy jQuery charts, etc?

So what?

Are either of these ideas useful?  Would they increase the number of defects found per review request?  Would they increase the frequency and speed of reviews?  Would they improve user perception of peer code review?  Would it be ignored?  Or could it harm a team of developers?  What are the benefits and drawbacks?

If anything, it’d give ReviewBoard some ability to record metrics, which is handy when you want to show the big boss how much money you’re saving with code review.

Might be worth looking into.  Thoughts?