Tag Archives: greg wilson

This is my last “goodbye school” post, I swear.

So a few days ago, my official grades for my M. Sc. rolled in.  That same day, I went to the Bahen Centre, turned in my desk keys, got my keycard authorization revoked, and scheduled my computer for erasure.

It felt like some pretty big steps.  There was a palpable sense of finality.  I was out.  It was over.

The University has played a big role in my development, and despite all of my moaning and complaining over the years, I’m glad I went, and I’d do it again.

But not right now.

Graduate school almost didn’t happen for me, and I have two very important people to thank for making that happen:  Karen Reid and Greg Wilson.

I still fondly remember when you cornered me during that codesprint in 2009, and convinced me to try graduate school.  I don’t regret it.  It was the right decision.  So thank you both so much for convincing me, and giving me the chance, and thank you Greg for supervising, and guiding me through.

I learned lots.  I had fun.  🙂

Shakespeare wrote that brevity is the soul of wit.  Well, I

Research Experiment: A Recap

Before I start diving into results, I’m just going to recap my experiment so we’re all up to speed.

I’ll try to keep it short, sweet, and punchy – but remember, this is a couple of months of work right here.

Ready?  Here we go.

What I was looking for

A quick refresher on what code review is

Code review is like the software industry equivalent of a taste test.  A developer makes a change to a piece of software, puts that change up for review, and a few reviewers take a look at that change to make sure it’s up to snuff.  If some issues are found during the course of the review, the developer can go back and make revisions.  Once the reviewers give it the thumbs up, the change is put into the software.

That’s an oversimplified description of code review,  but it’ll do for now.

So what?

What’s important is to know that it works. Jason Cohen showed that code review reduces the number of defects that enter the final software product. That’s great!

But there are some other cool advantages to doing code review as well.

  1. It helps to train up new hires.  They can lurk during reviews to see how more experienced developers look at the code.  They get to see what’s happening in other parts of the software.  They get their code reviewed, which means direct, applicable feedback.  All good things.
  2. It helps to clean and homogenize the code.  Since the code will be seen by their peers, developers are generally compelled to not put up “embarrassing” code (or, if they do, to at least try to explain why they did).  Code review is a great way to compel developers to keep their code readable and consistent.
  3. It helps to spread knowledge and good practices around the team.  New hires aren’t the only ones to benefit from code reviews.  There’s always something you can learn from another developer, and code review is where that will happen.  And I believe this is true not just for those who receive the reviews, but also for those who perform the reviews.

That last one is important.  Code review sounds like an excellent teaching tool.

So why isn’t code review part of the standard undergraduate computer science education?  Greg and I hypothesized that the reason that code review isn’t taught is because we don’t know how to teach it.

I’ll quote myself:

What if peer code review isn’t taught in undergraduate courses because we just don’t know how to teach it?  We don’t know how to fit it in to a curriculum that’s already packed to the brim.  We don’t know how to get students to take it seriously.  We don’t know if there’s pedagogical value, let alone how to show such value to the students.

The idea

Inspired by work by Joordens and Pare, Greg and I developed an approach to teaching code review that integrates itself nicely into the current curriculum.

Here’s the basic idea:

Suppose we have a computer programming class.  Also suppose that after each assignment, each student is randomly presented with anonymized assignment submissions from some of their peers.  Students will then be asked to anonymously peer grade these assignment submissions.

Now, before you go howling your head off about the inadequacy / incompetence of student markers, or the PeerScholar debacle, read this next paragraph, because there’s a twist.

The assignment submissions will still be marked by TA’s as usual.  The grades that a student receives from her peers will not directly affect her mark.  Instead, the student is graded based on how well they graded their peers. The peer reviews that a student completes will be compared with the grades that the TA’s delivered.  The closer a student is to the TA, the better the mark they get on their “peer grading” component (which is distinct from the mark they receive for their programming assignment).

Now, granted, the idea still needs some fleshing out, but already, we’ve got some questions that need answering:

  1. Joordens and Pare showed that for short written assignments, you need about 5 peer reviews to predict the mark that the TA will give.  Is this also true for computer programming assignments?
  2. Grading students based on how much their peer grading matches TA grading assumes that the TA is an infallible point of reference.  How often to TA’s disagree amongst themselves?
  3. Would peer grading like this actually make students better programmers?  Is there a significant difference in the quality of their programming after they perform the grading?
  4. What would students think of peer grading computer programming assignments?  How would they feel about it?

So those were my questions.

How I went about looking for the answers

Here’s the design of the experiment in a nutshell:

Writing phase

I have a treatment group, and a control group.  Both groups are composed of undergraduate students.  After writing a short pre-experiment questionnaire, participants in both groups will have half an hour to work on a short programming assignment.  The treatment group will then have another half an hour to peer grade some submissions for the assignment they just wrote.  The submissions that they mark will be mocked up by me, and will be the same for each participant in the treatment group.  The control group will not perform any grading – instead, they will do an unrelated vocabulary exercise for the same amount of time.  Then, participants in either group will have another half an hour to work on the second short programming assignment. Participants in my treatment group will write a short post-experiment questionnaire to get their impressions on their peer grading experience.  Then the participants are released.

Here’s a picture to help you visualize what you just read.

Tasks for each group in my experiment.

So now I’ve got two piles of submissions – one for each assignment, 60 submissions in total.  I add my mock-ups to each pile.  That means 35 submissions in each pile, and 70 submissions in total.

Marking phase

I assign ID numbers to each submission, shuffle them up, and hand them off to some graduate level TA’s that I hired.  The TA’s will grade each assignment using the same marking rubric that the treatment group used to peer grade.  They will not know if they are grading a treatment group submission, a control group submission, or a mock-up.

Choosing phase

After the grading is completed, I remove the mock-ups, and pair up submissions in both piles based on who wrote it.  So now I’ve got 30 pairs of submissions:  one for each student.  I then ask my graders to look at each pair, knowing that they’re both written by the same student, and to choose which one they think is better coded, and to rate and describe the difference (if any) between the two.  This is an attempt to catch possible improvements in the treatment group’s code that might not be captured in the marking rubric.

So that’s what I did

So everything you’ve just read is what I’ve just finished doing.

Once the submissions are marked, I’ll analyze the marks for the following:

  1. Comparing the two groups, is there any significant improvement in the marks from the first assignment to the second in the treatment group?
    1. If there was an improvement, on which criteria?  And how much of an improvement?
  2. How did the students do at grading my mock-ups?  How similar were their peer grades to what the TAs gave?
  3. How much did my two graders agree with one another?
  4. During the choosing phase, did my graders tend to choose the second assignment over the first assignment more often for the treatment group?

And I’ll also analyze the post-experiment questionnaire to get student feedback on their grading experience.

Ok, so that’s where I’m at.  Stay tuned for results.

The Shoulders of Tall, Smart People

Recently, I came to the realization that I’ve been writing computer programs in one form or another since I was about 6 or 7 years old.

Along the way, I’ve had plenty of people to influence the way I think about code, and how I write it.  Sure, there have been plenty of textbooks along the way too, but I want to give some thanks to the people who have directly affected my abilities to do what I do.

And what better way of doing that then by listing them?

A Chronological List of People Who Have Influenced My Coding

  1. My parents, for bringing home our first family computer.  It was an 8088XT IBM Clone – no hard drive, 640K of RAM, dual 5 1/4 floppies…it was awesome.  This is the computer I started coding on – but I couldn’t have started without…
  2. My Uncle Mark and my Aunt Soo.  Both have degrees in Computer Science from the University of Waterloo (that’s where they met).  My recollection is pretty vague, but I’m pretty sure that a lot of the programming texts in my house (a big blue QuickBasic manual comes to mind) surely didn’t come from my parents – must have been those two.  With the book in one hand, and the 8088 in the other, I cranked out stupid little programs, little text adventure games, quizzes, etc.
  3. The online QB community from the late 1990’s to the early 2000’s.  When my family got online, I soon found myself hanging out at NeoZones, in the #quickbasic IRC channel on EFNet… actually, a lot of crazy stuff was being done with QuickBasic back then – I remember when DirectQB came out, and somebody was able to code a raytracer…in BASIC.  It was awesome.  I’d say these were my foundation years, when I learned all of my programming fundamentals.
  4. My friends Nick Braun, Joel Beck, and Doug McQuiggan – these three guys and I used to come up with crazy ideas for games, and I’d try to program them.  I’d come home from school, and pound out code for a computer game for a few hours in the basement.  More often then not, these projects would simply be abandoned, but still, a lot was learned here.
    Joel, Doug, our friend Julian and myself were also members of a band in highschool.  It was my job to build and maintain the band website, and this is when I learned to write HTML, basic Perl, and simple JavaScript.
  5. After highschool, I went into Electrical and Computer Engineering at the University of Toronto.  I didn’t do too well at the Electrical bits, but I could handle myself at the Computer bits.  I learned OOP, Java, and basic design patterns from Prof. James McLean.
  6. I also learned a great deal from Prof. McLean’s course text – Introduction to Computer Science Using Java by Prof. John Carter.  I know I said I wasn’t going to mention textbooks, but I also got taught Discrete Mathematics from Prof. Carter, so I thought I’d toss him in too.
  7. My second (and last) semester in ECE had me taking Programming Fundamentals with Prof. Tarak Abdelrahman.  I learned basic C++ from Prof. Abdelrahman, and how to deal with large systems of code.
  8. After my move to the Arts & Science Faculty, I took my first Computer Science course with Dr. Jim Clarke. I learned about Unit Testing, and more design patterns.  I also eventually learned some basic Python from him, but I think it was in another course.
  9. I took CSC258 with Prof. Eric Hehner, and learned about the structure of computer processors.  Physically, this was a low-level as I’d ever gotten to computers.  I was familiar with writing Assembly from my QB days, but Prof. Hehner’s Opcode exercises were really quite challenging – in a pleasant way.  Also, check out his concept of Quote Notation…
  10. After that year, I spent the first of three summers working for the District School Board of Niagara.  Ken Pidgen was my manager, Mila Shostak was my supervisor.  Ken gave me incredible freedom to work, and soon I was developing web applications, as opposed to just fixing up department websites (as I originally thought I would be doing).  Mila gave me guidance, and showed me how to use CSS to style a website.  She also got me started using PHP and MySQL to create basic web applications.
  11. While working at the Board, I had the pleasure of sitting across from Jong Lee.  Jong and I would bounce ideas off of one another when we’d get stuck on a programming problem.  He was very experienced, and I learned lots of practical programming techniques from him.
  12. Michael Langlois and Ken Redekop acted as my clients at the Board, and always gave me interesting jobs and challenges to perform.Everyone at the Board was always very positive with me, and I’ll always be grateful that they took a newbie undergrad under their wing!  I was given a ridiculous amount of freedom at the Board, and was allowed to experiment with various technologies to get the job done.  Through my three summers there, I learned bits about Rails, CakePHP, MVC, network security, how to deploy an application remotely, how to run a local server, how to develop locally and post to remote, ORM, Flash, web security…so many things.  The list is huge.
  13. Karen Reid and Greg Wilson have been the latest influences on me.  The MarkUs Project was the first project I’ve ever worked on with a team.  It was my first time seriously using version control, my first time using a project management portal (Dr. Project), my first time learning Ruby, and my first time working on an open source project.  I’ve also learned plenty about time management, people, the business of software, and how to get things done.  Again, I’ve been given lots of freedom to learn, experiment, and hone my craft.

Anyhow, these are the people who come to mind.  I might add to this list if I remember anyone else.

But in the mean time, for the people listed above:  thank you.