# Author judgment standard — simplified

Author-only · 2026-10-24 · Not a replacement rubric

## Purpose

Check the drafter's answer and the AI judge's assessment against the materials actually supplied and the versioned grading references. Separate four things: established error, supported alternative, reference conflict, uncertainty. This procedure adds no points, penalties, flags, or severity rules.

## Three layers — check separately

1. **Drafter:** what did the answer state, produce, omit, or refer for human review?
2. **Judge:** is each finding, evidence citation, score, and severity supported? Did it miss a supported problem?
3. **Reference:** did the principles, rubric, or key omit an issue or conflict with the request/evidence? A reference defect is not automatically a model error.

## Sequence

1. Record package, request, versions, and whether you have already seen the answer or the judge's finding.
2. Read the exact task, teaching material, RFP, case information, evidence, and the applicable criterion. Name the basis: legal authority, document evidence, practice preference, or scoring policy.
3. Where feasible, assess the complete original answer **before** seeing the judge and save this first pass. Otherwise, disclose prior exposure; do not describe the review as blind.
4. Compare the first pass with the judge — including content the judge passed, not just flagged errors. Disagreement alone does not establish an error. Check meaningful cross-references and audience boundaries before claiming an omission.
5. For each finding, link: requirement → exact answer text (or precisely described absence) → supporting evidence/rule → disposition. Always consider the strongest supported alternative reading.
6. Apply quality and severity rules separately. Preserve the original judge result; record **confirm / reject / modify / unresolved** with reasons. Never silently overwrite a score or invent a new policy.
7. Only then add diagnostic interpretations.

## Coverage

All 13 base requests, plus each twin's affected requests, plus package-wide safety issues. Reviewing only judge-flagged errors cannot reveal misses. Anything unreviewed is reported as unreviewed — never claim a full audit from a sample. If you sample instead, document selection and exposure first.

## Evidence standard

Quotes must match the answer exactly. An omission is described as absent — never invent a quotation.

## Two lenses — analysis only, never grading rules

- **LSAT-style lens:** checks argument structure — assumptions, flaws, inference gaps [1]. It explains an established finding; it cannot create a new deduction.
- **Performance-Test-informed lens:** helps describe how the drafter and judge handle relevant facts, uncertainty, applicable principles, and professional communication [2]. It adds no criteria or scores and does not replace the rubric.

Retain the rubric, supporting guide, case-specific key, and judge reporting requirements used for each run. Any later substantive grading change must be separately versioned and disclosed.

## AI assistant

An AI assistant may provide optional diagnostic comments **after** the author's first pass. Record the displayed model identifier, interface, date, and materials supplied; disclose when the exact model version is unavailable. Its output is diagnostic, never the adjudication. This does not require an additional paid judge.

Prior research documents biases and reasoning limitations in LLM judging [3]. These motivate evidence-based review; they do not establish the presence or magnitude of those problems in this experiment.

## Reporting

Report supported misses, unsupported findings, incomplete explanations, severity disagreements, and reference limits — each with its unit and denominator. Selective review of Serious findings is not an unbiased accuracy estimate. Your final decision is the study's adjudication record, not infallible legal truth. Material uncertainty stays visible.

## References

[1] Law School Admission Council. “Logical Reasoning.” LSAT test format. The described skills include examining argument structure, detecting assumptions and flaws, and applying principles. Project diagnostic labels are adaptations, not official LSAT scores. https://www.lsac.org/lsat/taking-lsat/test-format/logical-reasoning

[2] State Bar of California. “California Bar Exam Grading,” Performance test section. Primary reference for professional-task assessment: https://www.calbar.ca.gov/admissions/examinations/california-bar-examination/california-bar-exam-grading . Supplementary explanation: UCLA School of Law Library, “California Bar Exam and MPRE: 3. The Performance Tests,” https://libguides.law.ucla.edu/c.php?g=183367&p=1209050 . These sources describe tasks involving supplied factual and legal materials, assessment of incomplete or unreliable facts, ethical issues, and professional communication. Their use here is an analytical comparison, not adoption of bar-exam scoring.

[3] Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., & Stoica, I. (2023). “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.” NeurIPS 2023 Datasets and Benchmarks Track. https://arxiv.org/abs/2306.05685v4 . The paper examines position, verbosity, and self-enhancement biases and reasoning limitations in its tested settings. Its reported agreement above 80% concerns human preferences in chat evaluation, not legal correctness or omission detection in this study.