Tag Archives: Computer Science

Exploring Peer Review in the Computer Science Classroom: Part 2 (Exciting Conclusion)

Now, where was I?

Oh yeah, I was reviewing this paper, and getting right to the good part – the experiment method, data, and results.

I hate to disappoint you, but this section starts as a bit of a downer:

…Unfortunately, the data collected was spotty, to say the least, and was not linked together well enough to support a reasonable, detailed analysis to meet our goals.

Yikes.  Oh well, let’s see what happened…

One issue we discovered was that our design was too broad to give definitive results.  We did, however, have enough data to allow us to narrow down the focus for a second study.  From the analysis of one class, we were able to identify two interesting areas for further research.  The type of review appears [sic] have a significant effect on the length and focus of the review.  We also found evidence that students reviewed some of the concepts differently than they did others.  Both of these findings should be explored in the future work.

Good.  At least there’s some groundwork for somebody to do a future study.

Reading on, I’m impressed with the scope with which the writers performed their experiment.  For example, instead of just experimenting on a single classroom, these people used experiments across 8 classrooms, and each classroom had a number of participants ranging from 10 to 60.  Ambitious.  Nice.

While their experiment may have been too broad to get the results they were looking for, it certainly has more authority than the studies that just used a single, small class.

However, maybe I spoke too soon:

The data collected for this study did not occur as smoothly as we would have wished. … As a result, we were able to collect a large amount of data but it is not as complete as we would have liked.”

Hm.  Doesn’t sound that great.

Apparently, data was supposed to be collected from surveys, review rubrics, and questionnaires, but it looks like the questionnaire data kind of fell through:

The number of responses to the questionnaires was low.  Three classes had no post-questionnaire responses at all.  Of those classes that did have responses for the second questionnaire, the number was too small to lend any confidence to a statistical analysis…

Review rubric data was a little more interesting – 996 completed rubrics were collected from 299 reviewers from the 8 classes over the course of the study.  Nice.  But, again, it wasn’t all flowers and hugs:

While the amount of data was large, it is also incomplete.  Of those classes which were intended to have both training and a peer reviews [sic], three of them were not able to complete the second review assignment and, so, have nothing to be compared to.  Two of the other classes have only a moderate number of participants (15-20) which is not as high as we would have liked for our statistical analysis.  One class provided no viable information at all…

Things just seem to get worse and worse for these guys.

And, not to kick them while they’re down, but their writing seems to get worse and worse too.  I’m noticing more typos and tense errors as I go along.  Maybe the stress was getting to them…

So, what did they find?  Drum roll, please…

Final Results

Surprise!  It’s inconclusive!

It sounds like they found more questions than answers…

Is it type or order (or both) that is causing the effects the [sic] training and peer review?

It took me a little while to figure out what they were asking here.  Apparently, before engaging in peer review exercises, students would critique material that was provided by the instructor.  It seems that the training review step, on average, generated more verbose comments (though, not necessarily relevant comments) than the peer review step.  But the peer review step tended to produce more relevant comments.  The experimenters have a couple of theories on why that is:

  1. Social pressure could cause students to be less verbose on their critiques on one another.
  2. Increased learning after the training exercise could cause the students to be more succinct and precise in their reviews
  3. Students were more engaged in the training exercise, and felt more inclined to be more verbose (even though they were less precise)

Unfortunately, the experimenters note, a lot rests on the motivation and attitude of the students, which wasn’t really considered or measured during the design of the experiment.  So, to break it down simple, the type of review (training vs peer) and the order (training first, then peer review), caused some numbers to change in their tables…but they don’t really know why.

They had questions on other things too…

Why are there differences in how concepts are reviewed?

  • Are there differences in conceptual difficulty?
  • Do the reviews improve student learning of these concepts?

The three CS topics that were focused on during these courses were the OOP concepts of Abstraction, Decomposition, and Encapsulation.  The experimenters also theorized that successful reviews go through the following steps:

  1. Analysis
  2. Evaluation
  3. Explanation
  4. Revision

(Unspecified) variations in how the students used these steps, and how verbose they were at each step, caught the experimenters’ attention.  They wonder if this has something to do with the conceptual difficulty of each topic, or if the reviews were effecting their understanding over time.

More questions they brought up…

Is reviewing an engaging and interesting task in computer science?

Very good question.  The experimenters noted that they had no measure of student’s interest, feeling, and engagement in the reviewing process.  They note that it is important to look at these attitudes over time for improvements or problems.

Are there significant learning benefits to reviewing in the early computer science curriculum as compared to other, common homework/lab exercises?

I’ll let them explain this one:

While we have identified a number of potential benefits from reviewing, we have not shown that it is better than or as good as what we currently do.  We require some sort of baseline to compare our efforts to.  We need a control group in our experiments in order to judge effectiveness.

And then the paper pretty much ends.

Where To Go From Here

The authors do a good job of lining up some interesting questions towards the end.  I guess this is how you salvage an experiment that didn’t go as planned – find the deeper questions, and see if somebody else can do a better job.

Or maybe, if you give the authors enough time, they’ll try to do the better job themselves. I think I’ve found the next paper to review.

A Few Things Drama Can Bring to Computer Science

So, yesterday I wrote:

[W]hat can Drama bring to Computer Science?

The easy one is presentation/communication skills.  A CS student might be brilliant, but that doesn’t mean they can present or communicate.  And if an idea can’t be communicated, it’s worthless.

But what else?  Any ideas?  I’m going to think about this for a bit, and I’ll see if I can come up with any more.

I posted the question on Twitter, and on my Facebook.  I was quite surprised by the amount of feedback I got back – apparently, quite a few people are interested in this topic.

Thanks for everybody who posted, or who came up to talk to me about this!  Let me summarize what I heard back:

  • Without a doubt, work in Drama hones movement/body senses.  It also trains us to use and take care of our body, and voice, like a musician would take care of a musical instrument.  Spending too much time hunkered over a keyboard can have detrimental effects on the body over time – I can personally admit to having absolutely awful shoulder tension, no doubt to my constant typing.  I only became aware of this tension, and how to deal with it, thanks to my work in Drama.  The dichotomy between body and mind is, in my humble opinion, a Western myth, and when you stop separating them, and get them to work together, amazing things can happen.  Just ask any contact improviser.
  • Drama is also emotional work.  No, this doesn’t mean we sit in a big circle and cry, and get credit for it.  Emotions are something that we study – how to mimic them, how to summon them out of ourselves, how to describe them, and abstractly represent them.  This is where Psychology, Drama, and Human-Computer Interaction might have some overlap.  In particular, it must be remembered that theatre is a communications medium between the actor(s) on stage, and the audience.  A webpage is also a communications medium.  Perhaps the theatre can teach a website a thing or two about communication.  I wonder what Marshall McLuhan would have to say on all of this…
  • Drama folk are creative, and are used to doing impossible, unreasonable things.  If you ask them to fly, they’ll figure out a way of doing it.  It’ll probably be abstract, and involve crazy lighting effects, but they’ll do it.  Production Managers are used to getting crazy, impossible requests from Directors all the time.  In my opinion, that’s what Directors are for!  Sometimes (usually due to time constraints), the Production Manager just says no to the Director – usually, though, they just go ahead and make impossible things happen – like building a triple layered reflection box.  This thing was a beast, and used a ton of computing power for live, context sensitive visual effects. I’m proud to have been a part of that.
  • In Drama, if the project is no fun, the end result suffers.  I’m pretty sure the same goes for software.  Drama students have a way of finding the “game”, the “jeu”, and the “play” (that’s why it’s called a “play”, people!) in what they’re doing.  The best actors are the ones who are clearly having a great time on stage, and are sharing this with the audience.  I believe this is applicable to software development…
  • If you want to think about complex systems, think about the stage.  At any given moment, n actors are on stage, interacting with various bits of set or props, interacting with each other – and each has their own motivation and personal story.  It can’t be a coincidence that the I* modeling language orients itself around terms like “actors” and “goals”.  It also can’t be a coincidence that many adventure game engines refer to in-game sprites as actors…

But now I want to hit the big one.  There is one thing that I really think Drama can bring to Computer Science.  Drama students are very good at it.  From what I can tell, Computer Science students rarely get exposed to it.

That thing is collaboration skills.

I already know that a few of my fellow Drama students will laugh at that – and say, “there are plenty of people in this department without collaboration skills”.  Yes, this is true.  But they tend not to do very well, or produce anything too interesting.

For me, the best, most exciting stuff comes when I’m with a group, and we’re not sure where we’re going with a project, but we just try things. We all throw a bunch of ideas in the middle, and try to put them on their feet.  The most unexpected things can happen.

Two years ago, I took a course in Experimental Theatre.  We were broken down into groups of 3 or 4 right at the beginning of the term, and given this challenge – show us what you like to see in theatre.  Show us what you think good theatre looks like.

That was it.  A blank canvas.  No script.  No “spec”.  Just each other.  It felt hopeless at first – we’d improv things, trying to get a feel for what our group wanted to do.  Nothing would happen, it’d fall flat.  We were lost.

But slowly, something started to piece itself together.  We found some material that we wanted to play with (The Wizard of Oz), and a subject that we liked – “home”.  What it means to be home, why people leave their homes, why we miss home, why we can’t stand home, what if we can’t get home, etc.  We divided the work up into 4 sections – 1 for each of us:  Dorothy, Cowardly Lion, Scarecrow, Tin Man.

It’s really hard to describe what we did.  The characters and structure from The Wizard of Oz was just a playground for a huge meditation on what “home” meant to different people.

And, wouldn’t you know it, the Robert Dziekanski Taser Incident happened just a week or so before we were to present.  It integrated perfectly into our piece.

When we finally presented it, some people were incredulous, others nauseous, others outraged.  Some were crying.  We had a huge class debate on whether or not it was appropriate to include the film clip of the Taser Incident in our piece.

But a lot of people really got something out of it.  And I believe a bunch of people from that class went to a protest rally about the incident that took place only a few days later.  I heard a lot of really positive things.  We were so excited by it that we almost took it to the Toronto Fringe Festival.

In my opinion, that was one of the most interesting, educational, horrifying, and rewarding art pieces I’d ever been involved in.  And it all started from nothing.

When are Computer Science students grouped up, and told to make whatever they want?  When are they given total freedom to just go crazy, and come up with something beautiful?  Something unique?  When are they given the frightening prospect of a blank canvas?  Maybe I’m being naive – but where are the collaborative creativity assignments in computer science education?

Now, I can imagine someone shouting – “but what about those group assignments!  What about CSC318, or CSC301?  Those were collaborative!”.

My friend, thanks for trying, but there’s a distinct difference between group problem solving, and collaborative creation.  In my mind, for collaborative creation at its best, the ensemble starts with nothing and must create something from it.  It’s the difference between having a script to toy with, and not having a script at all.

And don’t just tell me that an independent study fits the bill.  The word “independent” sabotages the whole idea – the key word is collaborate.

Oh, and did I mention that Artful Making sounds like an excellent book? Why don’t you go to their website, and read the forward by Google’s own Dr. Eric Schmidt.  I found it very illuminating.  I think this is going to the top of my to-read list.

Thanks to Blake Winton, Veronica Wong, Cam Gorrie, Jorge Aranda, Neil Ernst, Peter Freund, Jennifer Dowding, and Yev Falkovich for their input on this.  Yes, those little conversations made an impact!

What Can Drama Bring to Computer Science?

Yesterday, a bunch of Greg Wilson’s grad students had dinner at his place.  We got to meet his wife, his daughter, and eat some pretty amazing food.  It also gave his new grad students an opportunity to say an official “hello”, and introduce themselves to everybody else.

After introducing myself as having had an undergraduate degree in Computer Science and Drama, somebody made some remark about what an interesting combination that is. Greg replied by saying something like “That’s why I chose him”, and told a story about how one of the best programmers he ever knew was originally training to become a Rabbi, and got into Computer Science because he was working on some translations of ancient texts.

This got me thinking.  When I started focusing on both Drama and Computer Science, I remember always finding ways where Computer Science could help Drama.  I can easily rattle off a bunch of examples:

  • Better, more flexible sound cueing software (QLab is nice, but I think we can go deeper)
  • Communication tools for production teams, to help coordinate stage managers, directors, production managers, etc
  • Interfaces for movement artists to communicate with computers with their bodies in real-time, which in turn can drive things like sound/lighting cues, or other stage effects
  • Tools for doing cool, advanced projections – check out Lighttwist for example
  • Programming environments / domain specific languages for production crews who have to program lighting, sound, and video cues.  We used Isadora at the UCDP, which is like PureData with more of a GUI.  But…again…maybe we could do better.

So, while I was at the UCDP, all of these ideas rattled around in my head. I’ve now come to the realization that this has been completely one-sided.

So let’s switch it around – what can Drama bring to Computer Science?

The easy one is presentation/communication skills.  A CS student might be brilliant, but that doesn’t mean they can present or communicate.  And if an idea can’t be communicated, it’s worthless.

But what else?  Any ideas?  I’m going to think about this for a bit, and I’ll see if I can come up with any more.

UPDATE: So here’s what I found…

The Shoulders of Tall, Smart People

Recently, I came to the realization that I’ve been writing computer programs in one form or another since I was about 6 or 7 years old.

Along the way, I’ve had plenty of people to influence the way I think about code, and how I write it.  Sure, there have been plenty of textbooks along the way too, but I want to give some thanks to the people who have directly affected my abilities to do what I do.

And what better way of doing that then by listing them?

A Chronological List of People Who Have Influenced My Coding

  1. My parents, for bringing home our first family computer.  It was an 8088XT IBM Clone – no hard drive, 640K of RAM, dual 5 1/4 floppies…it was awesome.  This is the computer I started coding on – but I couldn’t have started without…
  2. My Uncle Mark and my Aunt Soo.  Both have degrees in Computer Science from the University of Waterloo (that’s where they met).  My recollection is pretty vague, but I’m pretty sure that a lot of the programming texts in my house (a big blue QuickBasic manual comes to mind) surely didn’t come from my parents – must have been those two.  With the book in one hand, and the 8088 in the other, I cranked out stupid little programs, little text adventure games, quizzes, etc.
  3. The online QB community from the late 1990’s to the early 2000’s.  When my family got online, I soon found myself hanging out at NeoZones, in the #quickbasic IRC channel on EFNet… actually, a lot of crazy stuff was being done with QuickBasic back then – I remember when DirectQB came out, and somebody was able to code a raytracer…in BASIC.  It was awesome.  I’d say these were my foundation years, when I learned all of my programming fundamentals.
  4. My friends Nick Braun, Joel Beck, and Doug McQuiggan – these three guys and I used to come up with crazy ideas for games, and I’d try to program them.  I’d come home from school, and pound out code for a computer game for a few hours in the basement.  More often then not, these projects would simply be abandoned, but still, a lot was learned here.
    Joel, Doug, our friend Julian and myself were also members of a band in highschool.  It was my job to build and maintain the band website, and this is when I learned to write HTML, basic Perl, and simple JavaScript.
  5. After highschool, I went into Electrical and Computer Engineering at the University of Toronto.  I didn’t do too well at the Electrical bits, but I could handle myself at the Computer bits.  I learned OOP, Java, and basic design patterns from Prof. James McLean.
  6. I also learned a great deal from Prof. McLean’s course text – Introduction to Computer Science Using Java by Prof. John Carter.  I know I said I wasn’t going to mention textbooks, but I also got taught Discrete Mathematics from Prof. Carter, so I thought I’d toss him in too.
  7. My second (and last) semester in ECE had me taking Programming Fundamentals with Prof. Tarak Abdelrahman.  I learned basic C++ from Prof. Abdelrahman, and how to deal with large systems of code.
  8. After my move to the Arts & Science Faculty, I took my first Computer Science course with Dr. Jim Clarke. I learned about Unit Testing, and more design patterns.  I also eventually learned some basic Python from him, but I think it was in another course.
  9. I took CSC258 with Prof. Eric Hehner, and learned about the structure of computer processors.  Physically, this was a low-level as I’d ever gotten to computers.  I was familiar with writing Assembly from my QB days, but Prof. Hehner’s Opcode exercises were really quite challenging – in a pleasant way.  Also, check out his concept of Quote Notation
  10. After that year, I spent the first of three summers working for the District School Board of Niagara.  Ken Pidgen was my manager, Mila Shostak was my supervisor.  Ken gave me incredible freedom to work, and soon I was developing web applications, as opposed to just fixing up department websites (as I originally thought I would be doing).  Mila gave me guidance, and showed me how to use CSS to style a website.  She also got me started using PHP and MySQL to create basic web applications.
  11. While working at the Board, I had the pleasure of sitting across from Jong Lee.  Jong and I would bounce ideas off of one another when we’d get stuck on a programming problem.  He was very experienced, and I learned lots of practical programming techniques from him.
  12. Michael Langlois and Ken Redekop acted as my clients at the Board, and always gave me interesting jobs and challenges to perform.Everyone at the Board was always very positive with me, and I’ll always be grateful that they took a newbie undergrad under their wing!  I was given a ridiculous amount of freedom at the Board, and was allowed to experiment with various technologies to get the job done.  Through my three summers there, I learned bits about Rails, CakePHP, MVC, network security, how to deploy an application remotely, how to run a local server, how to develop locally and post to remote, ORM, Flash, web security…so many things.  The list is huge.
  13. Karen Reid and Greg Wilson have been the latest influences on me.  The MarkUs Project was the first project I’ve ever worked on with a team.  It was my first time seriously using version control, my first time using a project management portal (Dr. Project), my first time learning Ruby, and my first time working on an open source project.  I’ve also learned plenty about time management, people, the business of software, and how to get things done.  Again, I’ve been given lots of freedom to learn, experiment, and hone my craft.

Anyhow, these are the people who come to mind.  I might add to this list if I remember anyone else.

But in the mean time, for the people listed above:  thank you.

Courses for Graduate School

With summer coming to a close, my upcoming school year is starting to piece itself together.  I’ve been assigned my research supervisor, and I just selected my courses:

Fall:

  • CSC2125F:  Topics in Software Engineering: Government 2.0
  • CSC2526F:  HCI: Topics in Ubiquitous Computing

Winter:

  • CSC2431S:  Topics in Computational Molecular Biology
  • CSC2511S:  Natural Language Computing

Psyched.