Cohen's Kappa Calculator — 2×2 Table
Enter the four cell counts from your contingency table. The calculator finds κ, Pₒ, and Pₑ instantly.
Cohen's Kappa Interpretation Table
The table below follows the Landis and Koch (1977) framework — the most widely cited set of qualitative labels for kappa ranges. These categories are guidelines adopted across many fields, not mathematical thresholds with objective authority. Different researchers and disciplines use different boundaries.
| Cohen's κ | Agreement Level | Practical Meaning | Badge |
|---|---|---|---|
| < 0 | Less than chance | Agreement is lower than the chance model predicts | Poor |
| 0.00 – 0.20 | Slight | Very limited agreement beyond chance | Slight |
| 0.21 – 0.40 | Fair | Some agreement beyond chance; often insufficient for research | Fair |
| 0.41 – 0.60 | Moderate | Moderate agreement; acceptable in some disciplines | Moderate |
| 0.61 – 0.80 | Substantial | Strong agreement; generally accepted for research | Substantial |
| 0.81 – 1.00 | Almost perfect | Very high agreement; near-ideal inter-rater reliability | Near-Perfect |
Context matters: These thresholds come from Landis JR & Koch GG (1977), Biometrics, 33(1), 159–174. They are not universal statistical laws. Interpretation should account for the study design, number of categories, prevalence of each category, and the consequences of disagreement in your field.
What Is Cohen's Kappa?
Cohen's kappa (κ) measures how much two raters agree on categorical classifications beyond the level of agreement attributable to chance. It was introduced by Jacob Cohen in 1960 as a correction to simple percent agreement, which can be misleading when one category is far more common than others.
The formula:
κ = (Pₒ − Pₑ) / (1 − Pₑ)where Pₒ is the observed proportion of agreement and Pₑ is the proportion of agreement expected from the marginal distributions of both raters.
Key distinction: A percent agreement of 90% sounds strong. But if 85% agreement would be expected by chance given how often each rater uses each category, then κ ≈ 0.33 — only fair agreement. Kappa surfaces this difference; raw percent agreement conceals it.
What Values Can Cohen's Kappa Take?
Kappa ranges from negative values to +1. The exact lower bound depends on the marginal distributions, so claiming a fixed minimum of −1 for every dataset is not accurate.
Is There One Universal Cohen's Kappa Interpretation Table?
No. Different researchers propose different threshold boundaries, and no single framework has been adopted universally. The table below compares the two most commonly cited systems.
| Framework | κ Range | Label Used | Important Note |
|---|---|---|---|
| Landis & Koch (1977) | 0.00 – 0.20 | Slight | Most widely cited; not universally accepted |
| Landis & Koch (1977) | 0.21 – 0.40 | Fair | Same source, different label from McHugh |
| McHugh (2012) | 0.21 – 0.39 | Minimal | Uses different boundary and terminology |
| McHugh (2012) | 0.40 – 0.59 | Weak | Labels differ from Landis & Koch for same range |
| Discipline-specific | Varies | Varies | Clinical fields often require κ ≥ 0.80 |
Always state which interpretation framework you used when reporting kappa. A κ of 0.55 might be labelled "moderate" (Landis & Koch) or "weak" (McHugh) — the number is the same, but the label differs. Readers need to know your reference.
Observed Agreement vs Expected Agreement
Kappa rests on the difference between two quantities. Getting both right is the foundation of the calculation.
| Component | Symbol | Meaning | How Computed |
|---|---|---|---|
| Observed agreement | Pₒ | Proportion of items where both raters agree | (a + d) / n |
| Expected agreement | Pₑ | Agreement predicted from each rater's marginal distributions | [(a+b)(a+c) + (c+d)(b+d)] / n² |
| Cohen's kappa | κ | Agreement beyond what chance predicts | (Pₒ − Pₑ) / (1 − Pₑ) |
Two raters might agree on 90% of cases. But if Rater A marks 95% of items as positive, and Rater B marks 93% as positive, then both raters choosing positive by chance alone would agree on roughly (0.95 × 0.93) + (0.05 × 0.07) ≈ 89% of items. Kappa would then reflect only the 1% extra agreement beyond that baseline — far less impressive than the raw 90% suggests.
Cohen's Kappa Formula
κ = (Pₒ − Pₑ) / (1 − Pₑ) where: Pₒ = observed proportion agreement = (a + d) / n Pₑ = expected proportion agreement = [(a+b)(a+c) + (c+d)(b+d)] / n² n = total number of observations (a + b + c + d)The 2×2 Contingency Table
| Rater B: Positive | Rater B: Negative | Row Total | |
|---|---|---|---|
| Rater A: Positive | a Both positive |
b A pos, B neg |
a + b |
| Rater A: Negative | c A neg, B pos |
d Both negative |
c + d |
| Column Total | a + c | b + d | n |
Cells a and d are the agreement cells (diagonal). Cells b and c are disagreements.
Worked Example — Step by Step
Scenario: Two radiologists independently assess 100 CT scans as either indicating a pulmonary nodule (Positive) or not (Negative).
| Radiologist B: Positive | Radiologist B: Negative | Row Total | |
|---|---|---|---|
| Radiologist A: Positive | 45 | 5 | 50 |
| Radiologist A: Negative | 10 | 40 | 50 |
| Column Total | 55 | 45 | 100 |
Calculation Steps
| Step | Operation | Result |
|---|---|---|
| 1 | Total n = 45 + 5 + 10 + 40 | n = 100 |
| 2 | Pₒ = (a + d) / n = (45 + 40) / 100 | Pₒ = 0.85 |
| 3 | Row totals: (a+b) = 50, (c+d) = 50 Col totals: (a+c) = 55, (b+d) = 45 |
— |
| 4 | Pₑ = [(50×55) + (50×45)] / 100² = [2750 + 2250] / 10000 |
Pₑ = 0.50 |
| 5 | κ = (0.85 − 0.50) / (1 − 0.50) = 0.35 / 0.50 |
κ = 0.70 |
Interpretation (Landis & Koch, 1977)
κ = 0.70 falls in the substantial agreement range (0.61–0.80). While raw agreement was 85%, kappa adjusts for the 50% expected by chance. The two radiologists agree considerably more than chance predicts, though not perfectly. For clinical diagnostic work, many guidelines would recommend further rater training or protocol clarification to push kappa above 0.80.
APA report: "Inter-rater agreement was assessed using Cohen's kappa. The estimated kappa was κ = .70, indicating substantial agreement (Landis & Koch, 1977)."
Why High Agreement Can Mean Low Kappa: The Prevalence Effect
Scenario: Two screeners independently review 200 records for a rare condition. Only 10 cases are positive.
| Screener B: Positive | Screener B: Negative | Row Total | |
|---|---|---|---|
| Screener A: Positive | 7 | 3 | 10 |
| Screener A: Negative | 3 | 184 | 187 |
| Column Total | 10 | 190 | 200 |
The Numbers Reveal the Gap
Percent agreement is 95.5% — which sounds excellent. But the vast majority of cases are negative, so two screeners who independently classify nearly everything as negative will agree by chance most of the time. Kappa accounts for this: κ = 0.53 (moderate agreement) is a truer reflection of how well the screeners are identifying the positive cases. Reporting only percent agreement would overstate the reliability of this screening process.
Cohen's Kappa vs Percent Agreement
| Measure | Accounts for Chance? | Best Used When | Main Limitation |
|---|---|---|---|
| Percent agreement | ✗ No | Quick communication; balanced categories | Inflated by unequal category frequencies |
| Cohen's κ (unweighted) | ✓ Yes | Nominal or ordinal categories; two raters | Sensitive to prevalence and marginal distributions |
| Weighted κ | ✓ Yes | Ordinal categories where partial credit for near-agreement makes sense | Weight choice (linear vs quadratic) affects result |
Many journals and reporting guidelines recommend presenting both percent agreement and kappa together. Percent agreement is more intuitive; kappa is more rigorous. Neither measure alone gives the complete picture.
Unweighted vs Weighted Cohen's Kappa
Unweighted kappa treats every disagreement as equal — a rater marking "Poor" when the other marks "Excellent" is the same error as marking "Good" vs "Very Good." Weighted kappa allows different penalties for different types of disagreement, which matters for ordinal scales.
Linear Weights
Penalty increases proportionally with the distance between categories. A two-step disagreement earns twice the penalty of a one-step disagreement. Common in Likert-scale assessments.
Quadratic Weights
Penalty increases with the square of the distance. Large disagreements are penalized more heavily than small ones. Often preferred in clinical severity ratings and pain scales.
When to use weighted kappa
Use weighted kappa when your categories are ordered and a near-disagreement should carry less cost than a distant one. Example: two clinicians rating disease severity on a 4-point scale (1=Mild, 4=Severe). If one rates "Mild" and the other rates "Moderate," that is a smaller problem than one rating "Mild" and the other "Severe." Note: quadratic weighted kappa is mathematically equivalent to the intraclass correlation coefficient (ICC) under certain conditions.
Is Cohen's Kappa Statistically Significant?
Two separate questions apply to any kappa estimate:
Is kappa statistically significant?
H₀: κ = 0 (no agreement beyond chance). Large enough samples will make even a small kappa statistically significant. A significance test alone says little about practical reliability.
How large is kappa?
The magnitude of kappa, referenced against an interpretation framework (e.g., Landis & Koch), determines practical importance. A κ of 0.10 can be statistically significant but practically irrelevant.
No universal critical-value table exists for kappa in the same way one exists for t, F, or chi-square. Significance depends on the sample size, observed agreement, expected agreement, marginal distributions, and the standard error method used. Report a p-value and confidence interval from your statistical software rather than consulting a generic kappa critical-value table.
Confidence Interval for Kappa
A confidence interval around κ describes uncertainty in the estimate. Example: κ = 0.72, 95% CI [0.62, 0.82]. This means that if the study were repeated many times, 95% of such intervals would contain the true population kappa. Wider intervals indicate smaller samples or more variable agreement patterns. Statistical software (R, SPSS, Python, Stata) calculates both the standard error and confidence interval directly from your data.
Common Kappa Values — Quick Reference
What does each kappa value mean in practice? The table below uses the Landis and Koch (1977) framework. Remember: thresholds should match your discipline's conventions.
| κ Value | Interpretation (L&K) | Practical Meaning |
|---|---|---|
| 0.10 | Slight | Raters agree only marginally more than chance; rater training likely needed |
| 0.25 | Fair | Some agreement beyond chance; insufficient for most research |
| 0.40 | Moderate (lower boundary) | Acceptable in some exploratory research; often needs improvement |
| 0.50 | Moderate | Moderate reliability; acceptable in some contexts with caveats |
| 0.60 | Moderate (upper) / Substantial (lower) | Borderline; many researchers treat this as the minimum for publishable agreement |
| 0.70 | Substantial | Good agreement; widely considered acceptable for research reporting |
| 0.80 | Almost perfect | Strong agreement; often set as the target in clinical and diagnostic research |
| 0.90 | Almost perfect | Very high reliability; approaching the standard for instrument validation |
| 1.00 | Perfect | Every item classified identically by both raters; rarely achieved in practice |
Special Cases: Negative Kappa and κ = 0
Negative Kappa (κ < 0)
Observed agreement is lower than what the chance model predicts given the marginal distributions. Possible causes include systematic disagreement between raters, misaligned coding instructions, or category distributions so skewed that the expected agreement is very high. A negative kappa does not automatically mean raters are always opposite — it means their pattern of agreement falls below the baseline the formula establishes.
Kappa = 0
Observed agreement equals what would be expected if both raters chose categories according to their individual frequencies without regard for each other. This does not mean the raters never agree — it means their agreement is entirely explained by the chance model. Raters with κ = 0 may still share 70% or 80% raw agreement if one category dominates.
Cohen's Kappa vs Fleiss' Kappa
| Feature | Cohen's Kappa | Fleiss' Kappa |
|---|---|---|
| Number of raters | Designed for 2 | 2 or more raters |
| Each rater's contribution | Pair compared directly | Pooled marginals used |
| Chance agreement adjustment | ✓ Yes | ✓ Yes |
| Weighted version | ✓ Yes | Standard version unweighted |
| Typical application | Two-rater reliability studies | Multi-rater reliability studies |
When you have exactly two raters, Cohen's kappa is the standard choice. When three or more raters assess the same items, Fleiss' kappa or the intraclass correlation coefficient (ICC) is more appropriate, depending on whether your data are categorical or continuous.
Cohen's Kappa vs Intraclass Correlation Coefficient (ICC)
The measurement scale — not the number of raters alone — determines which reliability statistic to use.
Use Cohen's Kappa
Your categories are nominal or ordinal. Raters assign items to discrete, labelled classes (e.g., yes/no, diagnosis A/B/C, severity 1–5). Kappa is appropriate for agreement on categorical labels.
Use ICC
Your measurements are continuous or interval-scaled. Raters record numeric scores (e.g., blood pressure measurements, pain scale readings, test scores). ICC captures both systematic bias and random variability, which kappa does not.
Common Cohen's Kappa Mistakes
Treating kappa as percent agreement
A κ of 0.70 does not mean 70% agreement. Kappa is not a proportion; it is the ratio of observed-beyond-chance agreement to maximum possible agreement beyond chance.
Assuming 0.80 is universally "good"
Some fields require κ ≥ 0.80 for acceptable reliability; others accept κ ≥ 0.60. Always state the threshold and framework relevant to your discipline.
Ignoring the prevalence effect
Reporting only percent agreement when one category is rare inflates the apparent reliability of the rating process. Always report kappa alongside percent agreement.
Using Cohen's kappa for more than two raters
With three or more raters, Fleiss' kappa or another multi-rater agreement index is appropriate. Cohen's kappa is a pairwise measure.
Treating statistical significance as practical importance
A large sample can make a κ of 0.15 statistically significant. That does not mean the raters agree well enough for research or clinical use.
Not reporting the interpretation framework
Different frameworks assign different labels to the same kappa value. Always cite which thresholds you used so readers can evaluate your claims.
How to Report Cohen's Kappa
Reporting should include the kappa value, confidence interval where available, and the interpretation framework used.
General Template
APA-Style Example
Software output: In R (use the irr or psych package), Python (sklearn.metrics.cohen_kappa_score), SPSS (Crosstabs → Statistics → Kappa), and Stata (kappa command). Always check whether your software reports unweighted or weighted kappa, and which standard error method it uses for the confidence interval.
Sources and Academic References
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46. doi:10.1177/001316446002000104 — Original introduction of kappa for two-rater nominal agreement.
Landis JR, Koch GG. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174. doi:10.2307/2529310 — Source of the widely cited kappa interpretation categories.
McHugh ML. (2012). Interrater reliability: The kappa statistic. Biochemia Medica, 22(3), 276–282. doi:10.11613/BM.2012.031 — Alternative interpretation thresholds and a thorough clinical discussion.
Fleiss JL. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378–382. doi:10.1037/h0031619 — Original source for the multi-rater extension of kappa.
Gwet KL. (2014). Handbook of Inter-Rater Reliability (4th ed.). Advanced Analytics. — Comprehensive treatment of kappa, its variants, and alternatives. Discusses the prevalence paradox and proposes alternative agreement statistics.
Related Statistical Tables and Resources
Frequently Asked Questions
Cohen's kappa (κ) is a statistic that measures the agreement between two raters classifying items into categories, correcting for the level of agreement expected purely by chance. Introduced by Jacob Cohen in 1960, it ranges from negative values (worse than chance) through 0 (chance-level) to 1 (perfect agreement).
There is no universal answer. Using the Landis and Koch (1977) framework, many researchers treat κ ≥ 0.61 as substantial and acceptable for research. Clinical and diagnostic settings often require κ ≥ 0.80. The appropriate standard depends on your discipline, the consequences of disagreement, and whether the categories are balanced.
κ = 0.80 falls in the "almost perfect" range under Landis and Koch (1977). It is generally considered strong inter-rater reliability in most research contexts. In some clinical applications, κ = 0.80 is the minimum acceptable threshold. In psychometric instrument development, targets can be even higher. Always compare against the conventions of your specific field.
Under Landis and Koch (1977), κ = 0.60 sits at the boundary between moderate and substantial agreement. Many researchers treat it as the minimum threshold for publishable reliability data, though this varies. Under McHugh (2012), 0.60 falls in the "moderate" range with a different label. Report the framework you used.
A negative kappa means the observed agreement between raters is less than the agreement predicted by chance given the marginal distributions. Possible causes include systematic disagreement (one rater tends to classify items that the other does not), misaligned coding protocols, or unusual category distributions. It does not automatically mean the raters always disagree — it means their pattern of agreement is below the chance baseline the formula establishes.
This is the prevalence effect. When one category dominates — say 90% of items are negative — two raters who independently classify most items as negative will share high raw agreement by chance. Kappa removes that chance-based agreement. The result can be a kappa of 0.30 or 0.40 despite 90% percent agreement. This is not a flaw in kappa; it is a correct reflection of what the raters' classifications actually contribute beyond the baseline.
There is no universal critical-value table for kappa equivalent to the t or chi-square tables. Significance depends on sample size, observed agreement, expected agreement, marginal distributions, and the standard error method used. Use your statistical software's p-value and confidence interval output rather than looking up a generic kappa critical value.
Cohen's kappa is designed for exactly two raters rating the same set of items. Fleiss' kappa extends the approach to three or more raters and does not require every item to be rated by every rater. The formulas differ in how expected agreement is calculated. When you have two raters, Cohen's kappa is standard. With more than two, consider Fleiss' kappa or the ICC depending on scale type.
Use weighted kappa when your categories are ordered (ordinal) and disagreements at different distances should not carry equal weight. For example, if your rating scale runs from 1 (No symptoms) to 5 (Severe), disagreeing by 1 point should count less than disagreeing by 4 points. Unweighted kappa treats both equally. Quadratic weighting is commonly recommended for clinical ordinal scales; linear weighting is more conservative.
Use the psych package: library(psych); cohen.kappa(x = your_matrix) where your_matrix is a 2-column matrix or data frame of rater classifications. The irr package also provides kappa2() for two raters. For weighted kappa, use the weight argument in cohen.kappa().
Use sklearn.metrics.cohen_kappa_score(y1, y2) from scikit-learn, where y1 and y2 are arrays of rater classifications. For weighted kappa, pass the weights parameter with 'linear' or 'quadratic'. The statsmodels library also includes kappa through its contingency table functions.
Yes. Many researchers and journal guidelines recommend reporting both. Percent agreement is easily understood by all readers, including those unfamiliar with kappa's chance-correction. Kappa is more rigorous. Together, they give a complete and transparent picture of rater reliability — especially useful when category prevalence may affect interpretation.
Unweighted kappa treats ordinal categories as nominal — it ignores the ordering. For ordinal data where the distance between disagreements matters, weighted kappa (linear or quadratic) is preferred. It assigns lower penalties to near-disagreements and higher penalties to distant ones, which better reflects the ordinal structure of the scale.
Sample size affects the precision of the kappa estimate and the width of the confidence interval, not the kappa value itself (assuming the same underlying agreement pattern). Small samples produce wide confidence intervals, meaning κ = 0.65 from 30 observations could plausibly range from 0.40 to 0.90. A large sample produces a narrower interval. For planning purposes, larger samples are needed when you expect the true kappa to be moderate and categories to be imbalanced.
Key Takeaways
Kappa corrects for chance
Raw agreement can be misleading when categories are unequal. Kappa removes the agreement expected by chance, giving a more honest measure of rater reliability.
No single universal threshold
Different frameworks (Landis & Koch, McHugh, discipline-specific) use different labels for the same kappa value. Always cite which system you used and why.
Prevalence changes kappa
When one category is far more common, expected chance agreement rises, and kappa falls even if raw agreement is high. Understanding this is essential for rare-condition screening studies.
Report kappa with context
A complete report includes the kappa value, confidence interval, percent agreement, the interpretation framework used, and the number of raters and items rated.