Read the rules before choosing labels

Download the blank learner CSV. Complete your first pass before opening the answer key.

Practise applying a written rule, extracting evidence and resolving disagreements. This hands-on extension belongs alongside the existing AI Annotation Jobs course. It does not repeat its introductions to pay, work platforms or scams. You will create a small annotated dataset and a quality report using invented, harmless customer-support text.

Prerequisite: understand that annotation means applying a defined label or mark to an example. No coding, paid platform or AI account is needed. Use paper, a text editor or the supplied CSV files. Suggested study time is 90–120 minutes, an editorial estimate rather than measured learner performance.

All records, organisations and people in this lab are synthetic. Use only the supplied examples. Do not paste confidential work, employer datasets or platform assessments into an AI tool. Follow the actual assignment's rules about tools and assistance; this lab is independent practice, not an answer service for hiring tests.

What you will make

• Twelve single-label intent annotations with evidence.
• Eight sets of exact-text entity spans.
• Four adjudication notes explaining a rule-based decision.
• Six ratings of synthetic AI replies.
• A final quality report that states both its score and its limits.

This lab uses deliberately small datasets so you can inspect every answer. It is not a representative benchmark, employment assessment or proof that a model or annotator is ready for production.

Learn

A label is correct relative to the task's written rules, not merely because it sounds plausible. Two projects may classify the same sentence differently. Work from the current guideline version, preserve the original text and keep a record of unresolved ambiguity.

Our fictional task routes support requests for a stationery shop. It assigns exactly one intent label to each message. These are the lab's rules, not universal industry categories.

Guideline version 1.0

• CANCEL: the customer explicitly asks to stop or cancel an existing order. This wins over other intents in a multi-intent message.
• RETURN: the customer explicitly asks to return an item or obtain a refund. Use only when CANCEL does not apply. Questions about whether a refund is possible count as RETURN.
• STATUS: asks where an order is, when it will arrive or whether it has shipped. Use only when neither CANCEL nor RETURN applies.
• PRODUCT: asks about product properties, stock or buying a product. Use only when none of the above applies.
• OTHER: the text is understandable but has no covered intent, such as thanks or feedback without a request.
• REVIEW: missing or unclear text prevents choosing a label. REVIEW is not a substitute for reading the precedence rule.

Negation matters: “do not cancel” is not a cancellation request. Conditional cancellation is REVIEW if it depends on an unknown fact and the message does not establish whether the condition holds. A question “can I cancel?” counts as an explicit cancellation request for this routing task, even though it does not authorise an actual cancellation.

The precedence is CANCEL > RETURN > STATUS > PRODUCT. OTHER and REVIEW are fallback labels, not extra tags. Annotation never performs the requested customer action.

Worked example

Text: “Where is order 420? Actually, please cancel it.” Label: CANCEL. Evidence: “please cancel it.” Reason: explicit cancellation outranks the status question.

Try it

Before opening the answer key, label C01–C12 in classification.csv. Fill the blank label, evidence and review_note fields. Copy the shortest useful evidence phrase; keep a note when a rule resolves a competing interpretation.

Quick check

“Thanks, the notebook arrived” is OTHER. “Can I return the notebook?” is RETURN. “The blue one” is REVIEW because the intended request cannot be recovered from this fragment. Do not infer intent from imaginary previous messages.

Lab sources.