{"id":1750,"date":"2010-08-30T21:16:33","date_gmt":"2010-08-31T02:16:33","guid":{"rendered":"http:\/\/mikeconley.ca\/blog\/?p=1750"},"modified":"2023-12-20T16:25:15","modified_gmt":"2023-12-20T21:25:15","slug":"some-more-results-did-the-graders-agree-part-2","status":"publish","type":"post","link":"https:\/\/mikeconley.ca\/blog\/2010\/08\/30\/some-more-results-did-the-graders-agree-part-2\/","title":{"rendered":"Some More Results: Did the Graders Agree? – Part 2"},"content":{"rendered":"

(Click here to read the first part of the story)<\/a><\/p>\n

I’m just going to come right out and say it: \u00a0I’m no stats buff.<\/p>\n

Actually, maybe that’s giving myself too much credit. \u00a0I barely scraped through my\u00a0compulsory statistics course. \u00a0In my defense, the teaching was abysmal, and the class average was in the sewer the entire time.<\/p>\n

So, unfortunately, I don’t have the statistical chops that a real scientist should.<\/p>\n

But, today, I learned a new trick.<\/p>\n

Pearson’s Correlation Co-efficient<\/h3>\n

Joorden’s and Pare gave me the idea while I was reviewing their paper<\/a> for the Related Work section of my thesis. \u00a0They used it in order to inspect mark agreement between their expert markers.<\/p>\n

In my last post on Grader agreement<\/a>, I was looking at mark agreement at the equivalence level. \u00a0Pearson’s Correlation Co-efficient should (I think) let me inspect mark agreement at the “shape” level.<\/p>\n

And by shape level, I mean this: \u00a0if Grader 1 gives a high mark for a participant, then Grader 2 gives a high mark. \u00a0If Grader 1 gives a low mark for the next participant, then Grader 2 gives a low mark. \u00a0These high and low marks might not be equal, but the basic shape of the thing is there.<\/p>\n

And this page<\/a>, with it’s useful table<\/a>, tell me how I can tell if the correlation co-efficient that I find is significant. \u00a0Awesome.<\/p>\n

At least, that’s my interpretation of Pearson’s Correlation Co-efficient. \u00a0Maybe I’ve got it wrong. \u00a0Please let me know if I do.<\/p>\n

Anyhow, it can’t hurt to look at some more tables. \u00a0Let’s do that.<\/p>\n

About these tables…<\/h3>\n

Like my previous post on graders, I’ve organized my data into two tables – one for each assignment.<\/p>\n

Each table has a row for that assignments criteria.<\/p>\n

Each table has two columns – the first is strictly to list the assignment criteria. \u00a0The second column gives the Pearson Correlation Co-efficient for each criterion. \u00a0The correlation measurement is between the marks that my two Graders gave on that criterion across all 30 submissions for that assignment.<\/p>\n

I hope that makes sense.<\/p>\n

Anyways, here goes…<\/p>\n

Da-ta!<\/h3>\n

Decks and Cards Grader Correlation Table<\/h4>\n

[table id=8 \/]<\/p>\n

Flights and Passengers Grader Correlation Table<\/h4>\n

[table id=9 \/]<\/p>\n

What does this tell us?<\/h3>\n

Well, first off, remember that for each assignment, for each criterion, there were 30 submissions.<\/p>\n

So N = 30.<\/p>\n

In order to determine if the correlation co-efficients are significant, we look at this table<\/a>, and find N – 2 down the left hand side:<\/p>\n

28 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 .306 \u00a0 \u00a0.361 \u00a0 \u00a0.423 \u00a0 \u00a0.463<\/p><\/blockquote>\n

Those 4 values on the right are the critical values that we want to pass for significance.<\/p>\n

Good news! \u00a0All<\/strong> of the correlation co-efficients fall within the range of [.306, .463]. \u00a0So now, I’ll show you their significance by level:<\/p>\n

p < 0.10<\/h4>\n
    \n
  • Design of __str__ in Decks and Cards assignment<\/li>\n<\/ul>\n

    p < 0.05<\/h4>\n
      \n
    • Design of deal method in Decks and Cards assignment<\/li>\n<\/ul>\n

      p < 0.02<\/h4>\n
        \n
      • Design of heaviest_passenger method in Flights and Passengers<\/li>\n<\/ul>\n

        p < 0.01<\/h4>\n

        Decks and Cards<\/h5>\n
          \n
        • Design of Deck constructor<\/li>\n
        • Style<\/li>\n
        • Internal Comments<\/li>\n
        • __str__ method correctness<\/li>\n
        • deal method correctness<\/li>\n
        • Deck constructor correctness<\/li>\n
        • Docstrings<\/li>\n
        • shuffle method correctness<\/li>\n
        • Design of shuffle method<\/li>\n
        • Design of cut method<\/li>\n
        • cut method correctness<\/li>\n
        • Error checking<\/li>\n<\/ul>\n
          Flights and Passengers<\/h5>\n
            \n
          • Design of __str__ method<\/li>\n
          • Design of lightest_passenger method<\/li>\n
          • Style<\/li>\n
          • Design of Flight constructor<\/li>\n
          • Internal comments<\/li>\n
          • Design of add_passenger method<\/li>\n
          • __str__ method correctness<\/li>\n
          • Error checking<\/li>\n
          • heaviest_passenger method correctness<\/li>\n
          • Docstrings<\/li>\n
          • lightest_passenger method correctness<\/li>\n
          • Flight constructor correctness<\/li>\n
          • add_passenger method correctness<\/li>\n<\/ul>\n

            Wow!<\/strong><\/p>\n

            Correlation of Mark Totals<\/h3>\n

            Joorden’s and Pare ran their correlation statistics on assignments that were marked on a scale from 1 to 10. \u00a0I can do the same type of analysis by simply running Pearson’s on the totals for each participant by each Grader.<\/p>\n

            Drum roll, please…<\/p>\n

            Decks and Cards<\/h4>\n

            p(28) = 0.89, p < 0.01<\/p>\n

            Flights and Passengers<\/h4>\n

            p(28) = 0.92, p < 0.01<\/p>\n

            Awesome!<\/strong><\/p>\n

            Summary \/ Conclusion<\/h3>\n

            I already showed before that my two Graders rarely agreed mark for mark, and that one Grader tended to give higher marks than the other.<\/p>\n

            The analysis with Pearson’s correlation co-efficient seems to suggest that, while there isn’t one-to-one agreement, there is certainly a significant correlation – with the majority of the criteria having a correlation with p < 0.01!<\/strong><\/p>\n

            The total marks also show a very strong, significant, positive correlation.<\/p>\n

            Ok, so that’s the conclusion here: \u00a0the Graders marks do not match, but show moderate to high positive correlation to a significant degree.<\/strong><\/p>\n

            How’s My Stats?<\/h3>\n

            Did I screw up somewhere? \u00a0Am I making fallacious claims? \u00a0Let me know – post a comment!<\/p>\n","protected":false},"excerpt":{"rendered":"

            (Click here to read the first part of the story) I’m just going to come right out and say it: \u00a0I’m no stats buff. Actually, maybe that’s giving myself too much credit. \u00a0I barely scraped through my\u00a0compulsory statistics course. \u00a0In my defense, the teaching was abysmal, and the class average was in the sewer the […]<\/p>\n","protected":false},"author":4,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[626],"tags":[469,840,802,841,803,836,837,838,839,635],"class_list":["post-1750","post","type-post","status-publish","format-standard","hentry","category-research-computer-science-technology","tag-data","tag-grader-agreement","tag-grading","tag-marker-agreement","tag-marking","tag-pearsons-correlation-co-efficient","tag-r","tag-significance","tag-statistical-significance","tag-statistics"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/prmTy-se","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/posts\/1750","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/comments?post=1750"}],"version-history":[{"count":14,"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/posts\/1750\/revisions"}],"predecessor-version":[{"id":1759,"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/posts\/1750\/revisions\/1759"}],"wp:attachment":[{"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/media?parent=1750"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/categories?post=1750"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mikeconley.ca\/blog\/wp-json\/wp\/v2\/tags?post=1750"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}