I didn’t know that a half hour ago.\u00a0 Emily told me, and she’s a titan when it comes to the English language.<\/p>\n
The graphs below were created a few days ago, before I knew this rule.\u00a0 So they use peer’s<\/em> instead of peers’<\/em>.\u00a0 I dun goofed.\u00a0 And I’m too lazy to change them (and I don’t want to use OpenOffice Draw more than I have to).<\/p>\n I just wanted to let you Internet people know that I’ve realized this, since their are so many lot of grammer nazi’s<\/a> out they’re on the webz.<\/p>\n Now, with that out of the way, where were we?<\/p>\n If you read my experiment recap<\/a>, then you know that my treatment group wrote a questionnaire after they were done all of their assignment writing.<\/p>\n The questionnaire was used to get an impression of how participants felt about their peer reviewing experience.<\/p>\n Just to remind you, my participants were marking mock-ups that I created for an assignment that they had just written.\u00a0 There were 5 mock-ups per assignment, so 10 mock-ups in total.\u00a0 Some of my mock-ups were very concise.\u00a0 Others were intentionally horrible and hard to read.\u00a0 Some were extremely vigilant in their documentation.\u00a0 Others were laconic.\u00a0 I tried to capture a nice spectrum of first year work. None of my participants knew that I had mocked the assignments up.<\/p>\n The questionnaire made the following statements, and asked students to agree on a scale from 1 to 5, where 1 was Strongly Disagree and 5 was Strongly Agree:<\/p>\n For questions 2, 5, 7, 8, and 10, participants were asked to expand with a written comment if they answered 3 or above.<\/p>\n Of the 30 participants in my study, 15 were in my treatment group, and therefore only 15 people filled out this questionnaire.<\/p>\n The graphs are histograms<\/strong> – that means that the higher the bar is, the more participants answered the question that way.<\/p>\n So, without further ado, here are the results…<\/p>\n <\/a> While there’s more weight on the positive side, opinion seems pretty split on this one.\u00a0 It might really depend on what kind of social \/ working group you have in your programming classes.<\/p>\n It might also depend on how adherent students are to the rules, since sharing code with your peers is a bit of a no-no according to the UofT Computer Science rules of conduct.\u00a0 Most programming courses have something like the following on their syllabus:<\/p>\n Never look at another student\u2019s assignment solution, whether it is on paper or on the computer Of course, this only applies before an assignment is due.\u00a0 Once the due date has passed, it’s OK to look at one another’s code…but how many students do that?<\/p>\n Anyhow, looking at the graph, I don’t think we got too much out of that one.\u00a0 Let’s move on.<\/p>\n Well, that’s a nice strong signal.\u00a0 Clearly, there’s more weight on the positive side.\u00a0 So my participants seem to understand that grading the code is teaching them something.\u00a0 That’s good.<\/p>\n And now for an interesting question:\u00a0 is there any relationship between the amount of programming experience of the participant, and how they answered this question?\u00a0 Good question.\u00a0 Before the experiment began, all participants filled out a brief questionnaire.\u00a0 The questionnaire asked them to provide, in months, how much time they’ve spent in either a programming intensive course, or a programming job.\u00a0 So that’s my fuzzy measure for programming experience.<\/p>\n The result was surprising.<\/p>\n Number of participants:\u00a0 7 Number of participants:\u00a0 1 Number of participants:\u00a0 4 Number of participants:\u00a0 1 Number of participants:\u00a0 1 So there’s no evidence here that participants with more experience felt they learned less from the peer grading.<\/p>\n This was one of those questions where participants were asked to expand if they answered 3 or above.\u00a0 Here are some juicy morsels:<\/p>\n I learned some tricks and shortcuts of coding that make the solution more elegant and sometimes shorter.<\/p><\/blockquote>\n …it showed me how hard some code are to read since I do not know what is in the programmer’s head.<\/p><\/blockquote>\n I learned how different their coding style are compared to mine, as well as their reasoning to the assignment.<\/p><\/blockquote>\n l learned about how other people think differently on same question and their programming styles can be different very much.<\/p><\/blockquote>\n one of the codes I marked is very elegant and clear. It uses very different path from others. I really enjoyed that code. I think good codes from peers help us learn more.<\/p><\/blockquote>\n I didn’t know about the random.shuffle method.\u00a0 I also didn’t know that it would have been better to use Exceptions which I don’t really know.<\/p><\/blockquote>\n The different design or thinking towards the same question’s solution, and other ways to interpret a matter.<\/p><\/blockquote>\n Other people can have very convoluted solutions…<\/p><\/blockquote>\n Different ways of solving a problem<\/p><\/blockquote>\n A few Python shortcuts, especially involving string manipulation. As well, I learned how to efficiently shuffle a list.<\/p><\/blockquote>\n algorithm (ways of thinking), different ways of doing the same thing<\/p><\/blockquote>\n Sometimes a few little tricks or styles that I had forgotten about.\u00a0 Also just a few different ways to go about solving the problem.<\/p><\/blockquote>\n So what conclusions can I draw from this?<\/p>\n It looks like, regardless of experience, students seem to think peer grading teaches them something – even if it’s just a different design, or an approach to a problem.<\/p>\n Another clear signal in the “strongly agree” camp.\u00a0 This one is kind of a no-brainer though – seeing work by others certainly gives us a sense of how our own work rates in comparison.\u00a0 We do this kind of comparison all the time.<\/p>\n Anyhow, my participants seem to agree with that.<\/p>\n Again, a lot of agreement there.\u00a0 Students are curious to know what their peers think of their work.\u00a0 They care what their peers think.\u00a0 This is good.\u00a0 This is important.<\/p>\n Hm.\u00a0 More of a mixed reaction here.\u00a0 There’s more weight on the “strongly agree” side, but not a whole lot more.<\/p>\n This is interesting though.\u00a0 If I find that my treatment group does perform better on their second assignment, is it possible that their improvement isn’t from the grading, but rather from their intense study of the rubric?<\/p>\n So, depending on whether or not there’s an improvement, my critics could say I might have a wee case of confounding factor<\/a> syndrome, here.<\/p>\n And I would agree with them.\u00a0 However, I would also point out that if there was <\/em>an improvement in the treatment group, it wouldn’t matter what the actual source<\/em> of the learning was – the peer grading (along with the rubric) caused an improvement.\u00a0 And that’s fine.\u00a0 That’s an OK result.<\/p>\n Of course, this is all theoretical until I find out if there was an improvement in the treatment group grades.\u00a0 Stay tuned for that.<\/p>\n Anyhow, this was another one of those questions where I asked for elaboration for answers 3 and up.\u00a0 Here’s what the participants had to say:<\/p>\n I would have checked for exceptions (and know what exceptions to check). I would have put more comments and docstrings into my code. I would have named my variables more reasonably.<\/p>\n I would’ve wrote out documentation. (ie. docstrings) Though I found that internal commenting wasn’t necessary.<\/p>\n i’ll add more comments to my code and maybe some more exceptions.<\/p>\n Added comments and docstrings.<\/p>\n Code’s design, style, clearness, readability and docstrings.<\/p>\n Made more effort to write useful docstrings and comments<\/p>\n I would’ve included things that I wouldn’t have included if I was coding for myself (such as comments and docstrings).<\/p>\n Added more documentation (I forget what it’s called but it’s when you surround the comments with “” ”’ “”)<\/p>\n Written more docstrings and comments (even though I think the code was simple enough and the method names self-explanatory enough that the code didn’t need more than one or two terse docstrings).<\/p>\n I forgot about docstrings and commenting my code<\/p><\/blockquote>\n So it sounds like evaluation on documentation wasn’t clear enough in my assignment specification.\u00a0 There’s also some indication that participants thought that documentation wasn’t necessary if the code is simple enough.\u00a0 With respect to Docstrings<\/a>, I’d have to disagree, since Docstrings are overwhelmingly useful for generating and compiling documentation.\u00a0 That’s just my own personal feelings on the matter, though.<\/p>\n Note:<\/strong> this is not to be confused with “I enjoyed grading my peers’ work”, which is the next question.<\/p>\n Mostly agreement here.\u00a0 So that’s interesting – participants enjoyed the simple act of seeing and reading<\/em> <\/em>code written by their peers.<\/p>\n It looks like, in general, students don’t really enjoy grading their peers’ code.<\/strong> Clearly, it’s not a universal opinion – you can see there’s some disagreement in the graph.\u00a0 Still, the trend seems to go towards the “strongly disagree” camp.<\/p>\n That’s a very useful finding.\u00a0 There’s nothing worse than sweating your butt off to design and construct a new task for students, only to find out that they hate doing it.\u00a0 We may have caught this early.<\/p>\n And I don’t actually find this that surprising:\u00a0 code review isn’t exactly a pleasurable experience.\u00a0 The benefits are certainly nice, but code review is a bit like flossing… it just seems to slow the morning routine down, regardless of the benefits.<\/p>\n Here’s what some participants had to say about their answers:<\/p>\n Because I like to compare my thoughts and other people’s thoughts.<\/p>\n well, some of the codes are really hard to read. But I did learn something from the grading. And letting students grade the codes is more fair.<\/p>\n I got to see where I went wrong and saw more creative\/efficient solutions which will give me ideas for future assignments. But otherwise it was really boring.<\/p>\n So that I can learn from my peer’s thinking which gives me more diversity of coding and problem-solving.<\/p>\n Sometimes you see other student’s styles of coding\/commenting\/documenting and it helps you write better code. Sometimes you learn things that you didn’t know before. Sometimes it’s funny to see how other people code.<\/p>\n It was interesting to see their ideas, although sometimes painful to see their style.<\/p>\n not so much the grading part, but analyzing\/looking at the different ways of coding the same thing<\/p>\n It gave me a rare prospective to see how other people with a similar educational background write their code.<\/p>\n Makes you think more critically about the overall presentation of your code.\u00a0 You ask yourself : “What would someone think of my code if they were doing this?\u00a0 Would I get a good mark?”<\/p><\/blockquote>\n This one is more or less split right down the middle, with a little more weight on the agree side.<\/p>\n Again, participants who answered 3 or above were asked to elaborate.\u00a0 Here are some comments:<\/p>\n The hardest part was trying to trace through messy code in order to figure out if it actually works.<\/p>\n Emotionally, I know what the student is doing but I have to give bad marks for comments or style which makes me feel bad. Sometimes it is hard to distinguish the mark whether it is 3 or 4. The time was critical (did not have time to finish all papers) which might result in giving the wrong mark. I kept comparing marks and papers so I could get almost the fairest result between all students. It is hard to mark visually, i.e. not testing the code. Some codes are hard to read which make it hard for marking and I can assume it is wrong but it actually works.<\/p>\n Giving bad marks are hard!\u00a0 Reading bad code is painful!\u00a0 It wasn’t fun! \ud83d\ude41<\/p>\n It just became really tedious trying to understand people’s code.<\/p>\n To test and verify their code is hard sometimes as their method of solving a problem might be complicated. I need to think very carefully and test their code progressively.<\/p>\n The rubric felt a little too strict. Sometimes a peer’s code had small difficulties that could easily be overcome, but would be labeled as very poor. Also, the rubric wasn’t clear enough, especially on the error handling portions and style. There could be many ways of coding for example the __str__ functions (using concatenation versus using format eg. ‘ %s’ % string as opposed to using + str(string) +)<\/p>\n I just found it hard to read other’s code because I already have a set idea of how to solve the problems. I did not see how the solutions of my peers would’ve improved my own solutions, so I did not find value in this.<\/p>\n Reading through each line of code and trying to figure out what it does<\/p>\n Reading through convoluted, circuitous code to determine correctness.<\/p>\n Not every case is clear-cut, and sometimes it’s hard to decide which score to give.<\/p>\n Being harsh and honest.\u00a0 I guess it’s good not to ever meet the people who wrote the codes (unlike TAs) because they aren’t there to defend themselves.\u00a0 Saves some headaches \ud83d\ude42<\/p><\/blockquote>\n Ok, more or less full agreement here.\u00a0 At least, no disagreement.\u00a0 But also no full agreement.\u00a0 It’s sort of a lethargic “meh” with a flaccid thumbs up.<\/p>\n The conclusion?\u00a0 My participants felt that, more or less, their grading was probably fair.\u00a0 I guess.<\/p>\nThe Post-Experiment Questionnaire<\/h3>\n
A note on the peer reviewing experience<\/h4>\n
Anyhow, back to the questionnaire…<\/h4>\n
\n
<\/a><\/h3>\n
![\"It](\"http:\/\/mikeconley.ca\/blog\/wp-content\/uploads\/2010\/07\/unusual_to_see_peers_code.jpg\" "\\"It")
<\/a><\/p>\n
\nscreen. Never show another student your assignment solution. This applies to all drafts of a solution
\nand to incomplete solutions.<\/p><\/blockquote>\n<\/a><\/h3>\n
![\"Seeing](\"http:\/\/mikeconley.ca\/blog\/wp-content\/uploads\/2010\/07\/seeing_code_taught_me_things_didnt_know.jpg\" "\\"Seeing")
<\/a><\/p>\nFor participants who answered 5 (strongly agreed that they learned things they didn’t already know):<\/h4>\n
\nMaximum number of months:\u00a0 36
\nMinimum number of months:\u00a0 4
\nAverage number of months:\u00a0 16<\/p>\nFor participants who answered 4:<\/h4>\n
\nNumber of months:\u00a0 16<\/p>\nFor participants who answered 3:<\/h4>\n
\nMaximum number of months:\u00a0 16
\nMinimum number of months:\u00a0 8
\nAverage number of months:\u00a0 13<\/p>\nFor participants who answered 2:<\/h4>\n
\nAverage number of months:\u00a0 5<\/p>\nFor participants who answered 1 (strongly disagreed that they learned things they didn’t already know):<\/h4>\n
\nAverage number of months: 16<\/p>\nIf you answered 3 or greater to the question above, what did you learn?<\/h4>\n
<\/a><\/h3>\n
![\"Because](\"http:\/\/mikeconley.ca\/blog\/wp-content\/uploads\/2010\/07\/quality_of_my_work.jpg\" "\\"Because")
<\/a><\/p>\n<\/a><\/h3>\n
![\"I](\"http:\/\/mikeconley.ca\/blog\/wp-content\/uploads\/2010\/07\/know_how_peers_graded_me.jpg\" "\\"I")
<\/a><\/p>\n<\/a><\/h3>\n
![\"I](\"http:\/\/mikeconley.ca\/blog\/wp-content\/uploads\/2010\/07\/seeing_rubric_would_have_changed_things.jpg\" "\\"I")
<\/a><\/p>\nIf you answered 3 or greater to the question above, what would you have done differently?<\/h4>\n
<\/a><\/h3>\n
![\"During](\"http:\/\/mikeconley.ca\/blog\/wp-content\/uploads\/2010\/07\/enjoyed_seeing_others_students_assignments.jpg\" "\\"During")
<\/a><\/p>\n<\/a><\/h3>\n
![\"I](\"http:\/\/mikeconley.ca\/blog\/wp-content\/uploads\/2010\/07\/enjoyed_grading_peers_work.jpg\" "\\"I")
<\/a><\/p>\nIf you answered 3 or greater to the question above, why did you enjoy grading your peer’s work?<\/h4>\n
<\/a><\/h3>\n
![\"I](\"http:\/\/mikeconley.ca\/blog\/wp-content\/uploads\/2010\/07\/found_grading_peers_difficult.jpg\" "\\"I")
<\/a><\/p>\nIf you answered 3 or greater to the question above, what about grading your peer’s work was difficult?<\/h4>\n
<\/a><\/h3>\n
![\"I'm](\"http:\/\/mikeconley.ca\/blog\/wp-content\/uploads\/2010\/07\/confident_my_grading_was_fair.jpg\" "\\"I'm")
<\/a><\/p>\n<\/a><\/h3>\n
![\"Because](\"http:\/\/mikeconley.ca\/blog\/wp-content\/uploads\/2010\/07\/because_peers_coded_different.jpg\" "\\"Because")
<\/a><\/p>\n