[R] Cohen's kappa, unequal score ranges

[Julia Myatt]

Hi,

Thank you so much for your help with this, I can get it to work now! 

Just one more question, my actual data will involve hundreds of rows in my two columns, is there anyway to import my data frame via read.table, and then convert this into the matrix ready for ckappa, rather than typing the data into R directly?

Sorry, I really am a beginner!

Thanks again,

Julia. 

________________________________________
From: Scot W. McNary [smcnary at charm.net]
Sent: 23 December 2009 15:41
To: Julia Myatt
Cc: r-help at r-project.org
Subject: Re: [R] Cohen's kappa, unequal score ranges

Julia,

I was able to get your toy data to work, using the n * 2 data frame,
which is what ckappa takes as input.

 > chk <- data.frame(matrix(c(
+    3,   4,
+    4,   3,
+    2,   1,
+    2,   1,
+    5,   1,
+    2,   4), byrow = TRUE, ncol = 2))
 > chk
  X1 X2
1  3  4
2  4  3
3  2  1
4  2  1
5  5  1
6  2  4

 > sk <- ckappa(chk)
 > sk
$table
  1 2 3 4 5
1 0 0 0 0 0
2 2 0 0 1 0
3 0 0 0 1 0
4 0 0 1 0 0
5 1 0 0 0 0

$kappa
[1] -0.0909091

 > str(sk)
List of 2
 $ table: num [1:5, 1:5] 0 2 0 0 1 0 0 0 0 0 ...
  ..- attr(*, "dimnames")=List of 2
  .. ..$ : chr [1:5] "1" "2" "3" "4" ...
  .. ..$ : chr [1:5] "1" "2" "3" "4" ...
 $ kappa: num -0.091

Hope this helps,

Scot

Julia Myatt wrote:
> Hi Jim,
>
> Thanks for your help, I tried that function which did result in an output. The only issue now is my summary looks like this:
>
>       Length Class  Mode
> table 16     -none- numeric
> kappa  1     -none- numeric
>
> Not what I was expecting, this was the case when my data was in the following format:
>
>   ob1 ob2
> 1   3   4
> 2   4   3
> 3   2   1
> 4   2   1
> 5   5   1
> 6   2   4
>
> I then tried entering my data as a matrix like the one below (all this data is made up until I get it to work!):
>
>  p1 p2 p3 p4 p5
> p1  3  1  0  4  2
> p2  3  5  0  7  1
> p3  2  3  4  6  3
> p4  4  1  0  4  0
> p5  8  2  0  2  5
>
> and this resulted in the same type of output:
>
>  Length Class  Mode
> table 36     -none- numeric
> kappa  1     -none- numeric
>
> If that is the kappa output (1) something isn't right as I purposely made the data without good interrater reliability! I was wondering if you have anymore tips as to what I am doing wrong? I'm afraid I'm very new to R so don't even know the basics!!
>
> Thanks for all your help,
>
> Julia.
>
>
> ________________________________________
> From: Jim Lemon [jim at bitwrit.com.au]
> Sent: 23 December 2009 02:15
> To: Julia Myatt
> Cc: r-help at r-project.org
> Subject: Re: [R] Cohen's kappa, unequal score ranges
>
> On 12/23/2009 01:58 AM, Julia Myatt wrote:
>
>> Hi,
>>
>> I am having problems getting cohen's kappa to work. I have been using the function:
>>
>>
>>
>>> <-ckappa(x,y)
>>>
>>>
>> from the psy package.
>>
>> I am trying to test for inter-observer reliability, I have 2 observers and 26 categories, however, the two observers might not necessarily have the same range of categories (I have unequal score ranges). However, I thought R could cope with this.
>>
>> Each time I try the function above I get the following output:
>>
>>
>>
>>> Error in ckappa(ob1, ob2) : unused argument(s) (ob2)
>>>
>>>
>> I got this even when I altered the data to have equal score ranges, therefore I am not sure what this means or what I should do!!
>>
>> Any help in this area would be much appreciated, or anything about the best way to deal with inter-observer reliability (my data is all categorical),
>>
>>
>>
> Hi Julia,
> The ckappa function only takes one argument, a rater by object matrix or
> dataframe of scores. Try:
>
> ckappa(cbind(ob1,ob2))
>
> Jim
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>

--
Scot McNary
smcnary at charm dot net