Test your AI grading before any student sees a score.

What the kit covers

1. Start with a robust rubric for every assignment. Everything else in this kit depends on it. A vague rubric gives you confident, inconsistent grades, from people and from AI alike.
2. Build the exemplar first. Write the actual 100% piece of work, not a description of one. Grade it on your own rubric before any student work. If your best example does not earn top marks, the rubric is wrong, not the example.
3. Write rubric bands against the exemplar. Describe each score level by what the work actually looks like, using the exemplar as the top band. Vague terms get applied literally by AI, so define them.
4. Write exit conditions as tests that can come back false. "The student can defend the main number without notes" can fail. "The student understands the topic" cannot. If a condition can't fail, it can't tell you anything.
5. Split completion-checked work from graded work. Most of a course only needs to be confirmed done. Keep a small core graded against the rubric. You will look at less, and you will know which things to look at.
6. Have two different AI models grade each piece. Use models from different providers. When they disagree, you decide using the rubric, not a fixed rule.
7. Run the same paper three times. The same paper can come back with a different score. Run one paper three times before you trust any single score, and see how far it swings.
8. Grade five papers by hand first. Score five papers yourself, then compare with the AI. Where you differ, fix the rubric wording before you grade the rest.
9. Send disagreements and low scores to a person. Every disagreement between the models and every score below your passing line gets a human review. Spot-check the rest.
10. Never post grades automatically. A person stays between the AI's score and the student, every time.
11. Stop the AI from doing the student's work. Decide what the AI may and may not do while students produce the work, and say so plainly to students.
12. Get the logistics right. Make sure the right file reaches the grader and that scores land in the right place in your gradebook. This is where the first problems show up, before any AI judgment happens.

What went wrong for us, and what we learned

• The same paper gets a different score each time. Identical work, run again, can come back noticeably higher or lower. Lesson: run the same paper three times before you trust the grader on an assignment, and if any criterion moves more than one level, fix the rubric wording first. If students can revise and the best version counts, don't regrade a resubmission that hasn't changed. Run your exemplar again whenever you change AI model or tool, because a new model is a new grader.
• The AI can't be trusted to count. Even with explicit instructions, the AI miscounted things a person could check in seconds. Lesson: anything that can be counted or checked by hand, such as length or whether a required section is there, gets checked by you or a simple checklist before the AI reads the work. Only judgment goes to the AI.
• A fixed tie-break rule just picks one model. When two models disagree, a fixed rule like "take the lower score" or "take the higher score" doesn't judge quality. It quietly picks whichever model runs stingier or more generous. Lesson: when the models disagree, you decide, using the rubric. Keep a tally of which model runs higher so you know its lean. A gap of two levels or more on one criterion usually means the rubric wording is unclear, so fix the wording.
• Most grading errors are rubric-wording errors. Several of our worst errors came from the rubric itself: feedback that contradicted the level descriptions, and evidence the AI didn't count because the rubric never said it counted. Lesson: write the exemplar first, then the rubric levels. When the AI scores your exemplar below full marks, ask whether the exemplar or the wording is wrong, and never lower the expected score to match the AI. For each criterion, write one fixed sentence that names what's missing. Say plainly what counts, including tables, frameworks and headers.
• Open-ended improvement advice makes students give up. When the AI kept suggesting how to get closer to full marks, round after round, students stopped revising. Lesson: limit revisions (three worked for us), make feedback name what's missing rather than suggest new content, and treat a student who meets every criterion as done.
• Your own test papers will fool you. An early check made the grader look too generous, but most of the sample was our own test submissions. Lesson: take your own test papers out of the comparison and judge the grader only on student work.
• Scoring papers by hand shows which way the AI leans. Our hand comparison found the AI giving credit for claims the work didn't support. Lesson: the hand comparison isn't a formality. Note, criterion by criterion, whether the AI ran high or low, and tighten that wording.
• If nearly everyone gets full marks, check the rubric. A cluster of top scores is a reason to check that your rubric separates strong work from adequate work. It isn't proof the class is excellent.
• The AI grades whatever it's given. Some students uploaded the wrong file, and the AI graded it anyway. Lesson: confirm it's the right file before grading.

Protect student privacy

This kit runs in your own AI account, so your students' work never comes to Crucibl. It goes wherever your AI account sends it, under that provider's terms. In the U.S., student work is usually part of the education record protected by FERPA, and many schools and districts allow only AI tools they have approved. Before you paste any student work, check your school's rules and whether it has an agreement with that AI provider. If it doesn't, use the kit on your exemplar and do the student scoring yourself. Remove names, ID numbers and anything else that identifies a student, including personal details in the work itself. Where you can, use a school or business account that doesn't train on your content.

What it doesn't include

• Ready-made grading prompts, rule sets or code. Those depend on the course and the checks around them, and without those they produce confident wrong grades.
• Ready-made rubrics. Yours has to be written against your own exemplar.
• Any promise that grading runs without you.

Price

• $49
• $29 for AI-Proof Pro subscribers

Purchases open Friday, November 6. The free tools work now. Want a free start? Every assignment redesign now ends with a short grading plan.

Planning this for a whole department? Contact us.

AI-Proof shares what I have learned so far. It is also the groundwork for the Crucibl platform I am building for AI-augmented courses.