# Assessments that make a certificate mean something

Web page: https://academy.memberpress.ai/courses/certifications/l1/

Lesson 1 of 3 in [Certifications That Hold Up](https://academy.memberpress.ai/courses/certifications/), 10 min.

A certificate is only worth what it took to earn it. "Completed the course" only says someone pressed play. A certificate that means something claims things a person can do, and the assessment checks each one.

**Name the skills**

Write the claim as three to five things a learner can do. Mark each one with the evidence that suits it: a quiz for knowing, a task for doing. Most certificates promise doing. Here's the list this course uses, for a short prompt-writing course:

```markdown
# Prompt writing: what the certificate claims

A certified learner can:

1. Write a prompt that gets a usable first draft.        [task]
2. Give the context and the audience a draft needs.       [task]
3. Tell when an answer needs checking, and check it.      [task]
4. Name the parts of a prompt and what each one does.     [quiz]

## The task

Pick a real piece of writing from your own work: an email, a lesson
intro, a product description. Write the prompt you would give Claude
for it, run it, and hand in three things: the prompt, the first draft
it produced, and two lines on what you checked before you'd use it.
```

**Let Claude draft the rubric**

A rubric is what makes judging a task fair: for each skill, what "not yet", "meets" and "strong" look like, in words two reviewers would read the same way. This script gives Claude the skills and the task and asks for exactly that. It uses the Anthropic Python SDK:

```bash
pip install anthropic
export ANTHROPIC_API_KEY="..."   # your key, from console.anthropic.com
```

```python
"""Draft a rubric for a certificate's task.

Takes the skills the certificate claims and the task brief,
and asks Claude for three levels per task skill, with criteria
a reviewer can see in the work, plus a short passing example.

    python3 rubric.py skills.md
"""
import sys

import anthropic

BRIEF = """Here are the skills a certificate claims and the task learners hand in.
Write a rubric for the skills marked [task].

For each one, give three levels: Not yet, Meets, Strong. One line each,
describing what a reviewer can see in the submission, worded so two
reviewers would score it the same way.
Then give a three-line example of a passing submission.
Plain text, no markdown, at most 26 lines, each under 80 characters."""

if __name__ == "__main__":
    skills = open(sys.argv[1]).read()
    print("Asking Claude for a rubric...\n")
    response = anthropic.Anthropic().messages.create(
        model="claude-sonnet-5-5",
        max_tokens=4000,
        messages=[{"role": "user", "content": f"{BRIEF}\n\n<skills>\n{skills}\n</skills>"}],
    )
    print("".join(b.text for b in response.content if b.type == "text"))
```

Here's what came back for the prompt-writing course, exactly as it ran:

```text
RUBRIC (rate each skill separately)

1. Usable first draft
Not yet: Draft is the wrong kind of piece or ignores a constraint in the prompt.
Meets: Draft is the right kind of piece and follows every constraint stated.
Strong: Meets, and the prompt's stated needs (length, tone, format) all show.

2. Context and audience
Not yet: Prompt names no reader and gives no background beyond the task.
Meets: Prompt names who will read it and states the purpose or situation.
Strong: Meets, plus specific facts the draft uses without inventing any.

3. Checking the answer
Not yet: Checks line is missing or says only "read it" or "looked fine".
Meets: Two lines name specific items checked (a date, fact, name, claim).
Strong: Meets, and says what was found or changed, or why those items.

EXAMPLE OF A PASSING SUBMISSION

Prompt: 100-word friendly email to Year 5 parents: trip moved to Fri 9 May.
Draft: Dear parents, the museum trip is now Fri 9 May. Please pack lunch...
Checks: Date matches my calendar; cut invented 8:30 bus time, not in plan.
```

Notice "Checking the answer": "not yet" is a line that only says "read it" or "looked fine", and "meets" names the specific things checked. The passing example even cuts a bus time the draft made up.

**Tighten it, then set the pass line**

Cut any line that grades effort or style instead of the skill. This rubric had none. Then write the pass line down before you grade anyone, and decide what happens when someone falls short:

```bash
grep -n -i -E "effort|neat|tidy|polish|creativ|style" rubric.txt \
  || echo "no line grades effort or style"
echo "PASS: Meets on every skill. Short of that: feedback, then one resubmission." \
  >> rubric.txt
```

**Test the test**

Score two submissions twice, apart, ideally with a second reviewer. This script asks Claude to score each submission in two separate calls and marks any line where the runs disagree:

```python
"""Score submissions against a rubric, twice each.

Each submission is scored in two separate calls to Claude, run 1 and
run 2. Where the two disagree, the line is marked: that's the rubric
wording to fix.

    python3 score.py rubric.txt sub-a.txt sub-b.txt
"""
import re
import sys

import anthropic

BRIEF = """Score this submission against the rubric. For each numbered skill,
answer with exactly one of: Not yet, Meets, Strong.
Reply with one line per skill, like "1: Meets", and nothing else."""


def score(rubric, submission):
    response = anthropic.Anthropic().messages.create(
        model="claude-sonnet-5-5",
        max_tokens=4000,
        messages=[{"role": "user", "content": f"{BRIEF}\n\n<rubric>\n{rubric}\n</rubric>\n\n"
                                              f"<submission>\n{submission}\n</submission>"}],
    )
    text = "".join(b.text for b in response.content if b.type == "text")
    return dict(re.findall(r"^\D*(\d)\D*?(Not yet|Meets|Strong)", text, re.M))


if __name__ == "__main__":
    rubric = open(sys.argv[1]).read()
    print(f"{'':12}{'skill':8}{'run 1':10}{'run 2':10}")
    for name in sys.argv[2:]:
        a, b = score(rubric, open(name).read()), score(rubric, open(name).read())
        for skill in sorted(a):
            mark = "" if a[skill] == b.get(skill) else "  <- disagree"
            print(f"{name:12}{skill:8}{a[skill]:10}{b.get(skill, '?'):10}{mark}")
```

The two sample submissions are a pottery-class welcome email that names what it checked, and a sourdough lesson intro whose checks say only "it looked accurate". Both runs agreed on every line:

```text
             skill   run 1     run 2
sub-a.txt   1       Meets     Meets
sub-a.txt   2       Meets     Meets
sub-a.txt   3       Strong    Strong
sub-b.txt   1       Meets     Meets
sub-b.txt   2       Not yet   Not yet
sub-b.txt   3       Not yet   Not yet
```

Where two runs or two reviewers disagree, the wording is the problem, not the learner. Fix the line and score again.

**The common mistake**

Writing the rubric after you've seen the submissions. You'll end up grading whoever you're reading, not the skill. Write the levels and the pass line first, then grade.

## Try these prompts

- Here are the skills my certificate claims and the task learners hand in. For each task skill, write Not yet, Meets and Strong in one line each, describing what a reviewer can see in the work, worded so two reviewers would score it the same way.
- Read this rubric and point out any line that grades effort, neatness or style instead of the skill, and any wording two reviewers might read differently.
- Here is my rubric and two learner submissions. Score each submission on each skill, then tell me which lines were hardest to decide and why.

Next: [A certification versus a mini course](https://academy.memberpress.ai/courses/certifications/l2/)
