This optional lab uses synthetic practice records. Read the lab introduction and intent rules first. No paid platform or AI account is needed.
Learn
A disagreement is useful diagnostic information. It may reveal a missed rule, a vague guideline or an example that genuinely lacks context. Adjudication records the decision and the reason; it is not simply choosing the most confident person.
Keep the original labels when documenting a disagreement. If a rule changes, record a new version and identify older records that need review. Do not quietly change the key to improve your score.
Try it
Write an adjudication note for each pair. Use the text, guideline and two labels below; do not invent extra context.
- C05: Annotator A = STATUS; B = CANCEL.
- C08: A = CANCEL; B = STATUS.
- C10: A = CANCEL; B = REVIEW.
- N08: A extracts gel pens as PRODUCT; B leaves it unlabelled.
Template: Record | A decision | B decision | Applicable rule | Final decision | Reason | Guideline clarification needed?
Check your answer — open after your attempt
- C05 is CANCEL. The request includes both a status question and an explicit cancellation; precedence settles it. No rule change is needed.
- C08 is STATUS. “Do not cancel” negates cancellation. The arrival question remains. No rule change is needed.
- C10 is REVIEW. Cancellation depends on “if it has not shipped,” and shipping status is unknown. This is the guideline's conditional case. Do not guess.
- N08 contains no PRODUCT under this guideline because “gel pens” is plural and excluded. The exclusion is artificial but explicit. A real project might broaden coverage; that would require a documented revision and consistent re-annotation, not a one-off exception.
Measure agreement carefully
Suppose two annotators agree on 9 of 12 intent labels. Simple agreement is 9 ÷ 12 = 75%. That describes matching labels, not whether either person is correct. Two people can agree on a mistaken answer. A score from twelve synthetic examples also cannot establish real-world performance.
For your own work, report exact matches against the lab key separately from agreement with another annotator. Include counts, denominator and error types. Do not report a percentage without saying what was counted.