AI, Quiz Me!

I’ve noticed my desire to write about programming in this age of AI deteriorating lately. Not because interesting things aren’t happening, certainly not, but because it seems that the insights I have are being had simultaneously by everybody else.

I’m going to ignore that sentiment and write something anyway. Partly because I want to be a write, not a write-not, and partly because my brain rewards me when I publish something, and I miss that feeling.

So let me tell you about this Claude skill that I’ve been enjoying lately. Ever since my programming process shifted to writing primarily-English-occasionally-curly-brackets rather than the other way around, I’ve had this sinking feeling that I am losing touch with whatever codebase I am working on. I’ve experienced jumps in abstraction before. I wrote x86 assembly for a good while, then C++ and then the languages that carry so much machinery around with them that it would have made assembly-me squirm. Embracing a higher level of abstraction means giving up some control but in return opening up a world of new opportunities. I get that.

But something feels different this time.

I guess first and foremost because an agent is not deterministic the way a compiler is. But also because it feels like this time, I get to decide how much control I am willing to give up. The agent is perfectly happy to accept vague and poorly thought out instructions and quite often do a reasonable job all the same. On the other hand, retaining control is fatiguing! Which means that I am now in yet another battle of will power, like I am in so many other aspects of life, like my sweet tooth. Dangerous! I need make an effort, I think, to keep an understanding of how every codebase I work with works.

So I tried to get Claude to quiz me. This has worked so well that I have put a notice in my calendar to run it every Friday, and I think it’s really valuable.

Here is the skill file:

---
name: codebase-quiz
description: Quiz the user on the current repo to find where their mental model of the code is wrong. Use when the user asks to be quizzed or tested on the codebase (”quiz me”, “codebase quiz”, “Friday quiz”).
---

# Codebase quiz

Goal: find where my mental model of this codebase is wrong, not teach me. Don’t explain
anything before I answer.

## Storage

`~/.claude/codebase-quiz/<repo>/`, where `<repo>` is the repository name from
`git remote get-url origin` (not the local path, so multiple clones share one history).

- `profile.md` — tenure, areas (name, paths, my 1–3 rating, current difficulty, questions asked
  so far), session length, onboarding date, top-level structure snapshot.
- `log.md` — one dated section per session.

## Setup (don’t narrate it)

### If `profile.md` is missing → onboarding

1. Survey the repo: top-level apps/packages, CLAUDE.md files, rules, docs. Propose 8–12 areas,
   each with the paths it covers.
2. For each area, count my commits: `git log --author=”$(git config user.email)” --oneline -- <paths> | wc -l`.
   Find my first commit date as proposed tenure.
3. Show the table (area, paths, my commits) and ask me to: correct tenure, edit the area list,
   rate each area 1–3, set session length (default 8 questions).
4. Write `profile.md`, including a snapshot of the top-level directory listing.
5. This session is a calibration round: one medium question per area. At the end, record each
   area’s starting difficulty from the results, and note where my self-rating and my result
   disagree.

### If `profile.md` exists

1. Read `profile.md` and `log.md`. Note days since the last session.
2. If the top-level structure differs from the snapshot, or onboarding is older than 6 months,
   propose area-list changes before starting.
3. Run `git log --merges --oneline --since=<last session date>`.
4. Plan <session length> questions:
   - re-test past misconceptions (more of these the longer since the last session),
   - ~2 on recent merges,
   - the rest favouring areas with a low rating or few questions asked so far.

### Every session

For each planned question, read the code that settles it before asking. Docs, CLAUDE.md and
rules files are sources for questions, not answer keys. If they disagree with the code, the
code is the answer, and note the doc as stale.

## Questions

- Kinds: **predict** (given this input/state, what happens), **trace** (what calls what, from
  UI action to network request), **locate** (where is this decided), **why / what breaks**,
  **spot the bug** (show a small plausible diff; what does it break).
- No trivia: names, spellings, line numbers. Ask about behaviour and boundaries.
- Prefer cases where the plausible default assumption is wrong.
- Free-text answers, no multiple choice.
- One question at a time. Ask my confidence (1–3) with it. Wait for my answer.

## After each answer

- Verdict: correct / partly / wrong. Then the actual behaviour in ≤5 lines with `file:line` refs.
- If partly or wrong: state the wrong assumption in one sentence.
- One follow-up only if my answer points to a deeper misunderstanding.
- Difficulty per area: 2 correct in a row → harder (edge cases, cross-package). Wrong → same
  topic, simpler.
- “disagree”: re-check the code, then concede or show the evidence.
- “skip” and “explain more” are allowed. “stop” ends the session early; still do the End steps.

## End

1. Table: #, area, kind, confidence, result.
2. Misconceptions found, each as “I believed X; actually Y (`file`)”.
3. Docs/rules that contradicted the code, with the specific claim.
4. Append all of the above, dated, to `log.md`.
5. Update `profile.md`: per-area difficulty and questions-asked counts.

Obviously, this is Claude-specific but you should be able to have your agent of choice adjust it as necessary quite easily.