Mike Conley's Blog

The Primary Goal

From the very beginning, my GSoC project has been mainly focused towards one primary goal:  I want to build an extension for Review Board that will allow me to collect information about how long reviewers actually spend reviewing code.

That’s easier said than done.  When I started, the Review Board extension framework wasn’t really in a state to allow such an extension to exist.

So I’ve been tooling around in the Review Board code for the past 2 months, preparing the framework, and getting it ready to handle my extension.

And last night, it started to work.  I can now give rough estimates on how long a reviewer has spent reviewing code.

How It Works

My extension adds a new table to the database which stores “reviewing sessions”.  Each reviewing session is associated to a particular review request and user, and also has a field to store the number of seconds that a user has spent in review.

I’ve created a TemplateHook that allows me to inject Javascript into key areas of Review Board (in particular, the diff viewer, and the screenshot viewer).  The Javascript does the following:  every 10 seconds, we check to see if the mouse has moved on the body of the HTML document.  If it has, we send an “activity” notification to the server.

The server receives this activity notification through the Web API, and checks to see if the time lapsed since the last session update was greater than 10 seconds.  If it is, we increment the working session by 10 seconds and return a 200 HTTP code.  If it isn’t, we don’t change anything and return a 304 HTTP code.

Next, my extension waits for a user to publish a review.  When it notices that a review is being published, it finds the working session for that user and review request, and then attaches it to the published review.  If the user then starts looking at the diff or screenshots again, a new working session is created.

The result?  A pretty decent estimate of how long a user has spent reviewing the code.  No time gets recorded if the user gets up and has a sandwich.  No time gets recorded if the user is on another tab reading Reddit.

Not bad.  For a first draft, anyhow.

I think I’m going to try to chart the data somehow, so that users can track their inspection rates.  I’ll let you know how that goes.

Be the first to like.

Like

Unlike

Recently, I came to the realization that I’ve been writing computer programs in one form or another since I was about 6 or 7 years old.

Along the way, I’ve had plenty of people to influence the way I think about code, and how I write it.  Sure, there have been plenty of textbooks along the way too, but I want to give some thanks to the people who have directly affected my abilities to do what I do.

And what better way of doing that then by listing them?

A Chronological List of People Who Have Influenced My Coding

1. My parents, for bringing home our first family computer.  It was an 8088XT IBM Clone – no hard drive, 640K of RAM, dual 5 1/4 floppies…it was awesome.  This is the computer I started coding on – but I couldn’t have started without…
2. My Uncle Mark and my Aunt Soo.  Both have degrees in Computer Science from the University of Waterloo (that’s where they met).  My recollection is pretty vague, but I’m pretty sure that a lot of the programming texts in my house (a big blue QuickBasic manual comes to mind) surely didn’t come from my parents – must have been those two.  With the book in one hand, and the 8088 in the other, I cranked out stupid little programs, little text adventure games, quizzes, etc.
3. The online QB community from the late 1990′s to the early 2000′s.  When my family got online, I soon found myself hanging out at NeoZones, in the #quickbasic IRC channel on EFNet… actually, a lot of crazy stuff was being done with QuickBasic back then – I remember when DirectQB came out, and somebody was able to code a raytracer…in BASIC.  It was awesome.  I’d say these were my foundation years, when I learned all of my programming fundamentals.
4. My friends Nick Braun, Joel Beck, and Doug McQuiggan – these three guys and I used to come up with crazy ideas for games, and I’d try to program them.  I’d come home from school, and pound out code for a computer game for a few hours in the basement.  More often then not, these projects would simply be abandoned, but still, a lot was learned here.
   Joel, Doug, our friend Julian and myself were also members of a band in highschool.  It was my job to build and maintain the band website, and this is when I learned to write HTML, basic Perl, and simple JavaScript.
5. After highschool, I went into Electrical and Computer Engineering at the University of Toronto.  I didn’t do too well at the Electrical bits, but I could handle myself at the Computer bits.  I learned OOP, Java, and basic design patterns from Prof. James McLean.
6. I also learned a great deal from Prof. McLean’s course text – Introduction to Computer Science Using Java by Prof. John Carter.  I know I said I wasn’t going to mention textbooks, but I also got taught Discrete Mathematics from Prof. Carter, so I thought I’d toss him in too.
7. My second (and last) semester in ECE had me taking Programming Fundamentals with Prof. Tarak Abdelrahman.  I learned basic C++ from Prof. Abdelrahman, and how to deal with large systems of code.
8. After my move to the Arts & Science Faculty, I took my first Computer Science course with Dr. Jim Clarke. I learned about Unit Testing, and more design patterns.  I also eventually learned some basic Python from him, but I think it was in another course.
9. I took CSC258 with Prof. Eric Hehner, and learned about the structure of computer processors.  Physically, this was a low-level as I’d ever gotten to computers.  I was familiar with writing Assembly from my QB days, but Prof. Hehner’s Opcode exercises were really quite challenging – in a pleasant way.  Also, check out his concept of Quote Notation…
10. After that year, I spent the first of three summers working for the District School Board of Niagara.  Ken Pidgen was my manager, Mila Shostak was my supervisor.  Ken gave me incredible freedom to work, and soon I was developing web applications, as opposed to just fixing up department websites (as I originally thought I would be doing).  Mila gave me guidance, and showed me how to use CSS to style a website.  She also got me started using PHP and MySQL to create basic web applications.
11. While working at the Board, I had the pleasure of sitting across from Jong Lee.  Jong and I would bounce ideas off of one another when we’d get stuck on a programming problem.  He was very experienced, and I learned lots of practical programming techniques from him.
12. Michael Langlois and Ken Redekop acted as my clients at the Board, and always gave me interesting jobs and challenges to perform.Everyone at the Board was always very positive with me, and I’ll always be grateful that they took a newbie undergrad under their wing!  I was given a ridiculous amount of freedom at the Board, and was allowed to experiment with various technologies to get the job done.  Through my three summers there, I learned bits about Rails, CakePHP, MVC, network security, how to deploy an application remotely, how to run a local server, how to develop locally and post to remote, ORM, Flash, web security…so many things.  The list is huge.
13. Karen Reid and Greg Wilson have been the latest influences on me.  The MarkUs Project was the first project I’ve ever worked on with a team.  It was my first time seriously using version control, my first time using a project management portal (Dr. Project), my first time learning Ruby, and my first time working on an open source project.  I’ve also learned plenty about time management, people, the business of software, and how to get things done.  Again, I’ve been given lots of freedom to learn, experiment, and hone my craft.

Anyhow, these are the people who come to mind.  I might add to this list if I remember anyone else.

But in the mean time, for the people listed above:  thank you.

Be the first to like.

Like

Unlike

Remember alertCheck – that Firefox Extension I wrote a while back, that helped suppress annoying popups?

I finally decided to post it on Mozilla Addons.  Now compatible with Firefox 3.5.*.

Here you go.

Be the first to like.

Like

Unlike

For this summer, the Computer Science Department at UofT has hired me to continue my work on the OLM project.  Click on that link, or check out my other post about OLM to see what it’s all about.

I just finished my first week of work, and it finished with a long weekend.  Not bad.

And I’ve got a great team – I’m working with Severin Gehwolf and Nelle Varoquaux, both excellent thinkers, programmers, and collaborators.  Severin is a UofT student like myself, and Nelle has flown in specially from France (!) to work with us.  They’re great, and we’re going to get a lot done.

So what have we to show for our first week of work?

Well for starters, we’ve gutted the entire database schema of OLM.  We started right from the bottom, and worked our way through every component of the database, trying to figure out what we could cut, trim, expand, and refactor.

And there was plenty to do.  This version of OLM has been in the works for a while now, and there have been plenty of awesome people working on it – but there’s been a variety of Ruby/Rails/JavaScript experience, and the cracks show.

I, myself, came into this project with no Rails experience whatsoever, and while I think I now more or less get the drift, I’m still by no means an expert.  Anyhow,  I’m looking at my old code too, and kind of grimacing.

But the ideas are all there.  It’s like a big hunk of marble that a whole lot of people gnawed and chiseled at for a little bit, trying to make a sculpture.  After the big DB schema refactor, I think the whole team can sort of see the rough form of what this thing is trying to become, and now we just need to carve it out.  Luckily, instead of a few hours per week like the last few semesters, we get a full summer to focus on it.

So, with the DB refactor done, the first thing has been to redesign the models/controllers to play nice with our new database tables.  It was scary, because after the refactor, everything broke – but we’re working on it, and it’s slowly starting to come back.

We’ve also decided to switch the file storage back-end.  Up until now, we were using Ruby to organize a file system back-end to do simple versioning of submitted files.  One of our goals this summer, is to build an abstraction layer that will allow us to choose different options for this versioned storage back-end.  In particular, we aim to support Subversion.  That’s right – a web-based Subversion front-end that supports commits, and catches (but doesn’t resolve) conflicts.  It’s a fun thought.

I have a feeling this is going to be a very interesting part of our project, and I’ll probably report on it more as it develops – but as it stands, it’s still being conceived on wipe-boards and scrap paper.

Anyhow, I’ll try to keep this blog up to date with what we’re doing.  Or maybe I’ll keep this blog up to date.  I’m conflicted.

Who knows, maybe this will be my last blog post of the summer.  I won’t lie – after working 8 hours on a computer, the last thing I want to do is come home and write a blog post.  If anything, my posts will probably wait until the weekends.

But we’ll see.

Be the first to like.

Like

Unlike

So, if you recall, I was asked to write a Firefox extension that would do word counting on websites.

Originally, when I started this project, I set a goal for myself:  I copied the text from Project Gutenberg’s First Folio version of Shakespeare’s Hamlet into OpenOffice Writer, recorded the word/line/character count statistics, and set that as my projected goal for my first iteration of my extension.

But there’s a problem with this approach:  I’m supposed to be copying the behaviour of Unix’s wc, not OpenOffice Writer’s word count.  Normally, this wouldn’t be a problem – a word count is a word count, a line count is a line count, and Writer should pump out the same numbers as wc.

Not so.

In my last post, I wrote:

According to OpenOffice Writer, this text has 32230 words, 173543 characters, and 4257 lines.

However, upon passing the same text (saved in the textfile “count.txt”) through wc, I got the following output:

5302 32230 178845 count.txt

Writer and wc agree on the number of words, but disagree on the number of lines – 5302 (wc) vs 4257 (Writer).  It’s a disagreement of about a thousand lines.

Brutal.

Anyhow, I’m going to focus on wc’s approach to line counting – simply returning the number of newline characters in the file.

And guess what…it works.  For Hamlet, my extension pumps out:

Document statistics:

Word Count:  32230
Line Count:  5302
Character Count:  178845
Character Count (no spaces):  142368

Nice.

Hamlet’s just the simple case though.  There are plenty of other cases to consider, but this is a start.

Anyhow, download here.

In this version, I’m using Mozilla’s TreeWalker implementation to stitch together the page text.  So far it seems to be working alright, but if it somehow ends up falling through, I might end up using something like Andrew Trusty’s code with the jQuery library to do the text stitching.

So there it is.  Maybe I’ll keep working on this, pretty it up a bit, etc.  However, work starts on Monday, and that’ll probably take up most of my technical attention.

We’ll see though.

Be the first to like.

Like

Unlike

It didn’t take long for another Firefox extension idea to come along.

Prof. Greg Wilson recently sent me an email, saying the following:

I’d like a Firefox plugin that does ‘wc’, i.e., counts characters, words, and lines on the current web page, and displays the results in the status bar.

Cool, I thought.  No problem.  That doesn’t sound too hard.

But I’ve been mulling and chewing this around in my head, and it’s actually a harder problem than it first sounds.

“wc“, short for word-count, is a small, simple, yet extraordinarily useful Unix utility that reads in some file, and spits out the number of words, characters, and lines for that file.

So what’s the problem?  What’s so hard about coding something like this for web pages?

Well, for starters, users of this proposed extension are probably only interested in the visible, readable text on a web page.  That means filtering out all of the HTML tags, all of the JavaScript, etc.  Also, many modern web pages make use of IFRAME’s, hidden DIV’s, etc.  Not to mention, most browsers do automatic word-wrapping, which could throw off the “line” counting.  How should I treat these cases?

I certainly don’t think this is an impossible task, just harder than it first sounded.

So here’s what I’m going to do:

First, I’m going to take care of the base case.  I’m going to take care of the case where users are viewing a page of all text, with almost zero HTML.

My test page will be an “etext” copy of Shakespeare’s Hamlet (first folio), hosted by Project Gutenberg.

According to OpenOffice Writer, this text has 32230 words, 173543 characters, and 4257 lines.

So that’s my target.  I’m going to create an extension that sits as a button on the status bar.  When the button is clicked, an alert will pop up with the statistics.  If all goes well, the numbers will match.

Sure, it’s not the most elegant interface, but it’ll do for now.

I’ll post more as it comes.

Be the first to like.

Like

Unlike

So, I think I’m more or less done the extension.

Someday, when I’ve got more extension development experience under my belt, I’ll probably come back to this and fix it up.  Until then, this will have to do.

Click here to download.

If you’re interested in looking at the source, just change the file extension from “.xpi” to “.zip”, and decompress.  It’s all there.

There’s no license on this thing, no GPL, MIT, nothing.  Use it however you want.  If you find it useful though, I’d love to hear from you – send me email, post a comment, Facebook, Twitter, whichever.

Whew.  I think I’m going to reward myself with some orange sherbet.  Om nom nom…

Here’s a really annoying website to test the extension with.  I really don’t recommend that you visit it without my extension installed.

The window hops around a bit, so just double click on the location bar, and type in something like “http://www.google.ca”.  This will start up the flood of alerts, and (hopefully) you’ll be able to suppress them after the first one hits.

Here’s the site.  Visit at your own risk.

UPDATE:

I’ve moved the extension to Mozilla Addons, and added Firefox 3.5 compatibility.

I’ve updated alertCheck.xpi so that it’ll play nice with Firefox 3.0b5, and hopefully Firefox 3.1.*.  Let me know if there are any behaviour foulups, and I’ll do my best to fix them.

Be the first to like.

Like

Unlike

Wow.  I think I got it.  I’ve got a Firefox plugin that can suppress all alert() dialogs on a page if the user checks a “suppress” box on the second alert() dialog.

The trick, was not to rely on the DOMContentLoaded event to fire to do the override.  Instead, I used the DOMWillOpenModalDialog to detect the first alert().  After detection, I overrode with an alertCheck which asked the user whether or not to “suppress more dialogs”.  If the user answers in the affirmative, alert() is simply overwritten with an empty function.

Piece of cake.

A couple of issues though…

Security

In order to override the alert() function, I have to write to document.getElementById(‘content’).contentWindow.wrappedJSObject.alert.

Remember how I mentioned the distance between the Extension JavaScript, and the inline content JavaScript?  I said it felt like a security layer.

I was totally right.

Check this out. I’ll quote:

You should be aware of XPCNativeWrappers when working with untrusted content. With XPCNativeWrappers turned on (which is the default in Firefox 1.5+), your extension can safely access the DOM of the content document, but not the content JavaScript. Bypassing XPCNativeWrapper to work with content JavaScript directly can lead to security problems.

Hrmph.  So I seem to be violating some security rules here.  So maybe my approach isn’t the greatest idea.  “Mook” from irc.mozilla.net #extdev suggested looking into commonDialog.xul…but I can’t seem to wrap my head around that just yet.

Imperfections

Not sure why yet, but while I can suppress dialog floods like this:

for (i = 0; i < 10; ++i) {
  alert(i);
}

It seems to fail on this:

for (i = 0; i < 10; ++i) {
  alert(i);
  confirm(i);
}

For some reason, regardless of whether or not I choose to suppress the dialogs, they just keep coming.  It works fine when I swap out the confirm() for a second alert().  Not exactly sure why.  Yet.

Ok, so I’m going to clean the code up, and post it soon.  I’ll also post a link to a real, brutally annoying website where you can test the alertCheck extension.  Just give me a bit.

Be the first to like.

Like

Unlike

Ok, so there have been some developments.

Before I go into this though, I just want to make it clear that I have very little knowledge or experience working with XUL, or writing Firefox extensions in general.  I’ve dabbled, but I’m mostly ignorant.  So all you Firefox aficionados out there…go easy on me.

So, developments:

Using this site, I created a skeleton for my extension, calling it “alertCheck”.

And then I hit a brick wall right away, trying to find the right “alert” to override.

Let me explain.  Mozilla extension development is strange for me because of all of the various layers to Firefox, and the distance between the JavaScript in the extension, and any page that is loaded in the browser.  It feels like a security layer – and it makes sense:  you really don’t want the Javascript on a website to monkey around with the internals of your browser.

Thankfully, this forum and this forum provided some help.

Now, after the appcontent is loaded, I wait for the DOMContentLoaded event to fire, and then write my new alert function to here:

document.getElementById('content').contentWindow.wrappedJSObject.alert

The problem is, if I wait for the DOMContentLoaded event to fire, the alert override happens (obviously) after all of the DOM Content is loaded into the browser.  This is only useful if the alert’s that we want to capture are fired using <body onLoad> or some other function that detects when the DOM has been loaded.  It’s no good for inline Javascript alerts, since these commands are parsed and executed line by line as the HTML is processed.

Phew.  Big mouthful.

So, where does that leave me?  Well, I was just hanging out in irc.mozilla.net in the #extdev channel.  Here’s a chunk of what was said:

<mike_conley> Hey all - I'm trying to write an extension to override window.alert, with an alertCheck that allows users to disable future alerts.
<mike_conley> So far, I'm able to override alert by writing to document.getElementById('content').contentWindow.wrappedJSObject.alert
<mike_conley> however, now it's a matter of timing - I'm doing the override after the DOMContentLoaded event is fired
<mike_conley> But this means that it doesn't catch alerts that are fired using inline javascript.
<mike_conley> So my question is, when should I do the override?
<Mook> I suspect just overriding commonDialog.xul is easier
<Mook> trying to poke the content JS's definition is going to be full of pain
<mike_conley> Cool, I'll look into that - thanks!
<Mook> (the content JS starts around http://mxr.mozilla.org/mozilla/source/dom/src/base/nsGlobalWindow.cpp#4022 and goes the the prompt service which pokes that xul)
<mike_conley> Excellent - I appreciate it.

So maybe there’s another approach that I should be taking – I’m going to look into overriding commonDialog.xul… I’ll write more when I have it.

Be the first to like.

Like

Unlike

Window.alert is a native function built into Firefox – but that doesn’t mean it can’t be overridden.

Check this out:

Open Firebug, and get to the console.  Then, click that little red arrow at the end of the input line so that you get the large box input on the right side of the screen.

Type this into the input box:

var alert_count = 0;
var old_alert = window.alert;
var alert_max = 5;
window.alert = function(alert_text) {
  if (alert_count < alert_max) {
    ++alert_count;
    old_alert(alert_text);
  } else {
    console.log("Reached maximum alerts");
  }
}

Now, hit “Run” at the bottom of that input window.  We’ve just overridden the window.alert function during runtime.

Hit “Clear” at the bottom of the input window, and type in:

for (i = 0; i < 10; ++i) {
  alert(i);
}

Hit “Run”.  Click “OK” for the first 5 alert windows, and watch as the rest of them are spewed out to the console.  Nice.

So, I don’t think this helps me much in creating my plug-in, but it’s interesting to see how window.alert is malliable at run-time.

This seems to be a more relevant discovery – Mozilla’s Chrome lets me create an alert popup with a checkbox using alertCheck.  I think this is exactly what I’m looking for.

I’ll tinker with it over the next few days, and post some code.

Be the first to like.

Like

Unlike