{"id":1676,"date":"2010-08-24T23:38:45","date_gmt":"2010-08-25T04:38:45","guid":{"rendered":"http:\/\/mikeconley.ca\/blog\/?p=1676"},"modified":"2023-12-20T16:25:15","modified_gmt":"2023-12-20T21:25:15","slug":"some-more-results-did-the-graders-agree","status":"publish","type":"post","link":"https:\/\/mikeconley.ca\/blog\/2010\/08\/24\/some-more-results-did-the-graders-agree\/","title":{"rendered":"Some More Results: Did the Graders Agree?"},"content":{"rendered":"

My experiment<\/a> makes a little bit of an assumption – and it’s the same assumption most teachers probably make before they hand back work.\u00a0 We assume that the work has been graded correctly and objectively.<\/em><\/p>\n

The rubric that I provided to my graders was supposed to help sort out all of this objectivity business.\u00a0 It was supposed to boil down all of the subjectivity into a nice, discrete, quantitative value.<\/p>\n

But I’m a careful guy, and I like back-ups.\u00a0 That’s why I had 2 graders do my grading.\u00a0 Both graders worked in isolation on the same submissions, with the same rubric.<\/p>\n

So, did it work?\u00a0 How did the grades match up?\u00a0 Did my graders tend to agree?<\/p>\n

Sounds like it’s time for some data analysis!<\/p>\n

About these tables…<\/h3>\n

I’m about to show you two tables of data – one table for each assignment<\/a>.\u00a0 The rows of the tables map to a single criterion on that assignments rubric<\/a>.<\/p>\n

The columns are concerned with the graders marks for each criterion.\u00a0 The first columns, Grader 1 – Average<\/strong> and Grader 2 – Average<\/strong>, simply show the average mark given for each criteria for each grader.<\/p>\n

Number of Agreements<\/strong> shows the number of times the marks between both graders matched for that criterion.\u00a0 Similarly, Number of Disagreements<\/strong> shows how many times they didn’t match.\u00a0 Agreement Percentage<\/strong> just converts those two values into a single percentage for agreement.<\/p>\n

Average Disagreement Magnitude<\/strong> takes every instance where there was a disagreement, and averages the magnitude of the disagreement (a reminder:\u00a0 the magnitude here is the absolute value of the difference).<\/p>\n

Finally, I should point out that these tables can be sorted by clicking on the headers.\u00a0 This will probably make your interpretation of the data a bit easier.<\/p>\n

So, if we’re clear on that, then let’s take a look at those tables…<\/p>\n

Flights and Passengers Grader Comparison<\/h3>\n

[table id=6 \/]<\/p>\n

Decks and Cards Grader Comparison<\/h3>\n

[table id=7 \/]<\/p>\n

Findings and Analysis<\/h3>\n

It is very rare for the graders to fully agree<\/h4>\n

It only happened once, on the “add_passenger” correctness criterion of the Flights and Passengers assignments. \u00a0If you sort the tables by “Number of Agreements” (or Number of Disagreements), you’ll see what I mean.<\/p>\n

Grader 2 tended to give higher marks than Grader 1<\/h4>\n

In fact, there are only a handful of cases (4, by my count), where this isn’t true:<\/p>\n

    \n
  1. The add_passenger correctness criterion on Flights and Passengers<\/li>\n
  2. The internal comments criterion on Flights and Passengers<\/li>\n
  3. The error checking criterion on Decks and Cards<\/li>\n
  4. The internal comments criterion on Decks and Cards<\/li>\n<\/ol>\n

    The graders tended to disagree more often on design and style<\/h4>\n

    Sort the tables by Number of Disagreements descending, and take a look down the left-hand side.<\/p>\n

    There are 14 criteria in total for each assignment. \u00a0If you’ve sorted the tables like I’ve asked, the top 7 criteria of each assignment are:<\/p>\n

    Flights and Passengers<\/h5>\n
      \n
    1. Style<\/li>\n
    2. Design of __str__ method<\/li>\n
    3. Design of heaviest_passenger method<\/li>\n
    4. Design of lightest_passenger method<\/li>\n
    5. Docstrings<\/li>\n
    6. Correctness of __str__ method<\/li>\n
    7. Design of Flight constructor<\/li>\n<\/ol>\n
      Decks and Cards<\/h5>\n
        \n
      1. Correctness of deal method<\/li>\n
      2. Style<\/li>\n
      3. Design of cut method<\/li>\n
      4. Design of __str__ method<\/li>\n
      5. Docstrings<\/li>\n
      6. Design of deal method<\/li>\n
      7. __str__<\/li>\n<\/ol>\n

        Of those 14, 9<\/strong> have to do with design or style. \u00a0It’s also worth noting that Doctrings and the correctness of the __str__ methods are in there too.<\/p>\n

        There were slightly more disagreement in Decks and Cards than in Flights and Passengers<\/h4>\n

        Total number of disagreements for Flights and Passengers: \u00a0136 (avg: \u00a09.71 per criterion)<\/p>\n

        Total number of disagreements for Decks and Cards: \u00a0161 (avg: \u00a011.5 per criterion)<\/p>\n

        Discussion<\/h3>\n

        Being Hands-off<\/h4>\n

        From the very beginning, when I contacted \/ hired my Graders, I was very hands-off.\u00a0 Each Grader was given the assignment specifications and rubrics ahead of time to look over, and then a single meeting to ask questions.\u00a0 After that, I just handed them manila envelopes filled with submissions for them to mark.<\/p>\n

        Having spoken with some of the undergraduate instructors here in the department, I know that this isn’t usually how grading is done.<\/p>\n

        Usually, the instructor will have a big grading meeting with their TAs.\u00a0 They’ll all work through a few submissions, and the TAs will be free to ask for a marking opinion from the instructor.<\/p>\n

        By being hands-off, I didn’t give my Graders the same level of guidance that they may have been used to.\u00a0 I did, however, tell them that they were free to e-mail me or come up to me if they had any questions during their marking.<\/p>\n

        The hands-off thing was a conscious choice by Greg and myself.\u00a0 We didn’t want me to bias the marking results, since I would know which submissions would be from the treatment group, and which ones would be from control.<\/p>\n

        Anyhow, the results from above have driven me to conclude that if you just hand your graders the assignments and the rubrics, and say “go”, you run the risk of seeing dramatic differences in grading from each Grader.\u00a0 From a student’s perspective, this means that it’s possible to be marked by “the good Grader”, or “the bad Grader”.<\/p>\n

        I’m not sure if a marking-meeting like I described would mitigate this difference in grading.\u00a0 I hypothesize that it would, but that’s an experiment for another day.<\/p>\n

        Questionable Calls<\/h4>\n

        If you sort the Decks and Cards table by Number of Disagreements, you’ll find that the criterion that my Graders disagreed most on was the correctness of the “deal” method.\u00a0 Out of 30 submissions, both Graders disagreed on that particular criterion 21 times (70%).<\/p>\n

        It’s a little strange to see that criterion all the way at the top there.\u00a0 As I mentioned earlier, most of the disagreements tended to be concerning design and style.<\/p>\n

        So what happened?<\/p>\n

        Well, let’s take a look at some examples.<\/p>\n

        Example #1<\/h5>\n

        The following is the deal method from participant #013:<\/p>\n

        def deal(self, num_to_deal):\r\n  i = 0\r\n  while i < num_to_deal:\r\n    print self.deck.pop(0)\r\n    i += 1\r\n<\/pre>\n
        Grader 1 gave this method a 1 for correctness, where Grader 2 gave this method a 4.<\/p>\n
        That’s a big disagreement.\u00a0 And remember, a 1 on this criterion means:<\/p>\n
        Barely meets assignment specifications. Severe problems throughout.<\/p><\/blockquote>\n
        I think I might have to go with Grader 2 on this one.\u00a0 Personally, I wouldn’t use a while-loop here – but that falls under the design criterion, and shouldn’t impact the correctness of the method.\u00a0 I’ve tried the code out.\u00a0 It works to spec.\u00a0 It deals from the top of the deck, just like it’s supposed to.\u00a0 Sure, there are some edge cases missed here (what is the Deck is empty?\u00a0 What if we’re asked to deal more than the number of cards left?\u00a0 What if we’re asked to deal a negative number of cards?\u00a0 etc)… but the method seems to deliver the basics.<\/p>\n
        Not sure what Grader 1 saw here.\u00a0 Hmph.<\/p>\n
        Example #2<\/h5>\n
        The following is the deal method from participant #023:<\/p>\n
        def deal(self, num_to_deal):\r\n res = []\r\n for i in range(0, num_to_deal):\r\n   res.append(self.cards.pop(0))\r\n<\/pre>\n
        Grader 1 gave this method a 0 for correctness.\u00a0 Grader 2 gave it a 3.<\/p>\n
        I see two major problems with this method.\u00a0 The first one is that it doesn’t print out the cards that are being dealt off:\u00a0 instead, it stores them in a list.\u00a0 Secondly, that list is just tossed out once the method exits, and nothing is returned.<\/p>\n
        A “0” for correctness simply means Unimplemented, which isn’t exactly true:\u00a0 this method has been implemented, and has the right interface.<\/p>\n
        But it doesn’t conform to the specification whatsoever.\u00a0 I would give this a 1.<\/p>\n
        So, in this case, I’d side more (but not agree) with Grader 1.<\/p>\n
        Example #3<\/h5>\n
        This is the deal method from participant #025:<\/p>\n
        def deal(self, num_to_deal):\r\n    num_cards_in_deck = len(self.cards)\r\n    try:\r\n        num_to_deal = int(num_to_deal)\r\n        if num_to_deal > num_cards_in_deck:\r\n            print \"Cannot deal more than \" + num_cards_in_deck + \" cards\\n\"\r\n        i = 0\r\n        while i < num_to_deal:\r\n            print str(self.cards[i])\r\n            i += 1\r\n        self.cards = self.cards[num_to_deal:]\r\n    except:\r\n        print \"Error using deal\\n\"\r\n<\/pre>\n
        Grader 1 also gave this method a 1 for correctness, where Grader 2 gave a 4.<\/p>\n
        The method is pretty awkward from a design perspective, but it seems to behave as it should – it deals the provided number of cards off of the top of the deck and prints them out.<\/p>\n
        It also catches some edge-cases:\u00a0 num_to_deal is converted to an int, and we check to ensure that num_to_deal is less than or equal to the number of cards left in the deck.<\/p>\n
        Again, I’ll have to side more with Grader 2 here.<\/p>\n
        Example #4<\/h5>\n
        This is the deal method from participant #030:<\/p>\n
        def deal(self, num_to_deal):\r\n  ''''''\r\n  i = 0\r\n  while i <= num_to_deal:\r\n    print self.cards[0]\r\n    del self.cards[0]\r\n<\/pre>\n
        Grader 1 gave this a 1.\u00a0 Grader 2 gave this a 4.<\/p>\n
        Well, right off the bat, there’s a major problem:\u00a0 this while-loop never exists.\u00a0 The while-loop is waiting for the value i to become greater than num_to_deal…but it never can, because i is defined as 0, and never incremented.<\/p>\n
        So this method doesn’t even come close to satisfying the spec.\u00a0 The description for a “1” on this criterion is:<\/p>\n
        Barely meets assignment specifications. Severe problems throughout.<\/p><\/blockquote>\n
        I’d have to side with Grader 1 on this one.\u00a0 The only thing this method delivers in accordance with the spec is the right interface.\u00a0 That’s about it.<\/p>\n
        Dealing from the Bottom of the Deck<\/h4>\n
        I received an e-mail from Grader 2 about the deal method.\u00a0 I’ve paraphrased it here:<\/p>\n
        If the students create the list of cards in a typical way, for suit in  CARD_SUITS; for rank in CARD_RANKS, and then print using something like:
        \nfor card in self.cards
        \nprint str(card) +\u00a0 “\\n”
        \nThen for deal, if they pick the cards to deal using pop() somehow, like:
        \nfor i in range(num_to_deal):
        \nprint str(self.cards.pop())<\/p>\n
        Aren’t they dealing from the bottom<\/p><\/blockquote>\n
        My answer was “yes, they are, and that’s a correctness problem”.\u00a0 In my assignment specification, I was intentionally vague about the internal collection of the cards – I let the participant figure that all out.\u00a0 All that mattered was that the model made sense, and followed the rules.<\/p>\n
        So if I print my deck, and it prints:<\/p>\n
        Q of Hearts\r\nA of Spades\r\n7 of Clubs<\/pre>\n
        Then deal(1) should print:<\/p>\n
        Q of Hearts\r\n<\/pre>\n
        regardless of the internal organization.<\/p>\n
        Anyhow, only Grader 2 asked for clarification on this, and I thought this might be the reason for all of the disagreement on the deal method.<\/p>\n
        Looking at all of the disagreements on the deal methods, it looks like 7 out of the 20 can be accounted for because students were unintentionally dealing from the bottom of the deck, and only Grader 2 caught it.<\/p>\n
        Subtracting the “dealing from the bottom” disagreements from the total leaves us with 13, which puts it more in line with some of the other correctness criteria.<\/p>\n
        So I’d have to say that, yes, the “dealing from the bottom” problem is what made the Graders disagree so much on this criterion:\u00a0 only 1 Grader realized that it was a problem while they were marking.\u00a0 Again, I think this was symptomatic of my hands-off approach to this part of the experiment.<\/p>\n
        In Summary<\/h3>\n
        My graders disagreed.\u00a0 A lot.\u00a0 And a good chunk of those disagreements were about style and design.\u00a0 Some of these disagreements might be attributable to my hands-off approach to the grading portion of the experiment.\u00a0 Some of them seem to be questionable calls from the Graders themselves.<\/p>\n
        Part of my experiment was interested in determining how closely peer grades from students can approximate grades from TAs.\u00a0 Since my TAs have trouble agreeing amongst themselves, I’m not sure how that part of the analysis is going to play out.<\/p>\n
        I hope the rest of my experiment is unaffected by their disagreement.<\/p>\n
        Stay tuned.<\/p>\n
        See anything?<\/h3>\n
        Do my numbers make no sense?\u00a0 Have I contradicted myself?\u00a0 Have I missed something critical?\u00a0 Are there unanswered questions here that I might be able to answer?\u00a0 I’d love to know.\u00a0 Please comment!<\/p>\n","protected":false},"excerpt":{"rendered":"
        My experiment makes a little bit of an assumption – and it’s the same assumption most teachers probably make before they hand back work.\u00a0 We assume that the work has been graded correctly and objectively. The rubric that I provided to my graders was supposed to help sort out all of this objectivity business.\u00a0 It […]<\/p>\n","protected":false},"author":4,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[454,626],"tags":[831,828,829,830,824,827,823,802,803,816,825,826],"class_list":["post-1676","post","type-post","status-publish","format-standard","hentry","category-code-reviews","category-research-computer-science-technology","tag-comparison","tag-criteria","tag-criterion","tag-data-analysis","tag-disagreement","tag-fairness-in-grading","tag-graders","tag-grading","tag-marking","tag-rubrics","tag-tas","tag-teaching-assistants"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/prmTy-r2","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/posts\/1676","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/comments?post=1676"}],"version-history":[{"count":45,"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/posts\/1676\/revisions"}],"predecessor-version":[{"id":3157,"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/posts\/1676\/revisions\/3157"}],"wp:attachment":[{"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/media?parent=1676"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/categories?post=1676"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/tags?post=1676"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}