Two teachers score the same paragraph with the same rubric. One gives it a 3. The other gives it a 2. Neither is careless. The rubric says "organized" at level 3 and "somewhat organized" at level 2, and each teacher has a different idea of the line between them.
A rubric that two people read differently cannot track progress. This guide covers how to build one that two people can repeat, how to turn its scores into a goal, and when a task analysis is a better choice.
When a rubric fits
Utah's goals guide says a team should decide ahead of time how progress on the target skill will be measured, using regularly used tools such as progress monitoring, checklists, curriculum-based measurements, rubrics, and behavior charts. For behavior, it lists frequency, duration, percentage of successful opportunities, or a rating scale.
A rubric fits when the skill is about quality and cannot be counted: the organization of a paragraph, the clarity of a spoken explanation, the quality of a conversation with a partner. If it can be counted, count it, as in percent accuracy vs trials vs duration.
Reliable and valid, in NCII's words
NCII's behavior guide defines two terms that matter here. Reliability is the extent to which scores are accurate and consistent. Validity is the extent to which scores represent the underlying construct, that is, the extent to which a score measures what it purports to measure.
For a rubric, reliability means two scorers agree. Validity means the levels describe the skill in the goal. A rubric can be reliable and still measure the wrong thing, such as neatness when the goal is organization.
Build it in five steps
- Name one skill. A rubric for "writing" is too broad. A rubric for "topic sentence and supporting details" is workable.
- Write levels in observable words. Describe what you can see in the work, not how it feels.
- Make adjacent levels different in one clear way. If level 3 and level 4 differ only in adjectives, scorers will drift.
- Add anchors. NCII says to create anchored examples and nonexamples when you define a target behavior. For a rubric, keep one real sample for each level.
- Test agreement. See below.
| Level | Description | Example sign |
|---|---|---|
| 4 | States the topic in a first sentence, adds three or more details that fit it, ends with a closing sentence | All three parts present |
| 3 | States the topic and adds two details that fit | Topic and two details |
| 2 | Has a topic sentence but details are off topic or only one is given | Topic with one on-topic detail |
| 1 | No clear topic sentence | Details without a topic |
Notice that each level can be checked by counting parts. That is what makes the rubric repeatable.
Check agreement
Have two people score the same five pieces independently, then compare. Count the pieces where both gave the same level and divide by five. If you agree on four of five, that is 80 percent. If you agree on two of five, look at where you differ. The cause is almost always a phrase in the descriptors.
Here is what a small check might look like, with invented scores.
| Paragraph | Scorer A | Scorer B | Agree? |
|---|---|---|---|
| 1 | 3 | 3 | Yes |
| 2 | 2 | 3 | No |
| 3 | 4 | 4 | Yes |
| 4 | 3 | 3 | Yes |
| 5 | 2 | 2 | Yes |
Agreement is 4 of 5, or 80 percent. The one disagreement, paragraph 2, is the useful part. The scorers sit down with it and find that one counted a detail as "on topic" and the other did not. They rewrite level 2 to say what counts, and keep paragraph 2 as an anchor.
Fix the wording, add an anchor for the disputed level, and try again with new pieces. This is the same kind of check described in interval recording and time sampling. Do it at the start, and again if a new person takes over scoring.
NCII notes a related warning about variability: highly variable data may mean the tool is not accurately measuring the student's skills, that there are implementation fidelity issues, or that the behavior chosen is not appropriate. If rubric scores jump around without a reason, look at the rubric first.
From rubric score to goal
A goal needs a criterion in numbers. A rubric score can supply it.
For example: "When given a grade 3 prompt and 15 minutes, Leila will write a paragraph that scores 3 or higher on the topic-and-details rubric on 4 of 5 consecutive samples by spring." That states the condition, the skill, the criterion, and the check. The baseline is the median score on three samples taken the same way.
Task analysis as an alternative
Project ACCESS describes a task analysis as a step-by-step breakdown of a task, such as washing dishes, using a vending machine, or solving a math problem, and gives an example scored as steps completed independently, such as steps 1 to 7 out of 10, or 70 percent.
This works for skills with a clear sequence, and it is easier to score reliably than a quality scale because each step is either done or not. It also shows which step the student is stuck on, which a single rubric score does not.
Permanent products
The same fact sheet lists permanent product as a method: a work sample or physical change that lasts long enough for you to take data on. A completed worksheet is one. A rubric is often applied to a permanent product, which has a practical benefit: you can rescore it later, or have a second person score it, without the student present.
Rubric mistakes
- Adjectives in the descriptors. "Good," "strong," and "weak" do not separate levels.
- Changing the rubric mid-year. A new rubric needs a new baseline.
- Scoring from memory. Score while the work is in front of you.
- One scorer, always. Add a second scorer on a sample.
- Averaging levels. A 3 and a 2 are not a 2.5 of anything. Report the median or the count at each level.
Levels and agreement rates
None of the sources gives a required number of levels or a minimum agreement rate. NCII cautions that many teacher-made measures have not been validated for individual progress monitoring. Treat a rubric as a good tool you have checked yourself, and pair it with a countable measure where you can. Terms are in the glossary.