Mizan
Docs

Methodology — the core skill (Modes 1–2)

Mirrored from the skill package. This is the canonical English text that Claude actually loads, published here so it can be read next to its Turkish mirror instead of only on GitHub. Edit the sources above, not this page; the Turkish mirror is docs/tr/metodoloji.md.

Mizan — Evidence-Tiered Auditing & Preregistration Registry

Mizan (Turkish/Arabic: “the scale”) turns a rigorous experimental-science discipline into a portable tool for evaluating any claim set and for maintaining living hypothesis registries. Its core commitments:

  1. Every claim gets an evidence tier. No untagged assertions.
  2. Thresholds are locked before results. If that’s impossible (retrospective analysis), the HARKing risk is stated explicitly — never silently absorbed.
  3. Every hypothesis carries a refutation condition. A claim that cannot fail is not audited, it is decorated.
  4. Refuted entries are never deleted. They are marked [R] and archived in place. Negative results are first-class results.
  5. Surprising positives get a symmetric control before a headline. A result that flatters the hypothesis is the one most in need of a confound check.
  6. Hit rates over curated examples. Three confirming anecdotes are selection bias; a scored prediction record is evidence.

Evidence tiers (use these exact labels, bilingual)

Tag TR EN Meaning
[K] Kanıtlanmış Proven Direct evidence supports it; source cited; threshold met
[H] Makul Hipotez Plausible hypothesis Theoretical grounding exists; empirical support missing or below threshold
[S] Spekülatif Speculative Interesting; not currently testable or no test designed
[R] Reddedildi Refuted Tested and failed its own threshold — kept on record, never deleted
[KKE] Kritik Kontrol Eksik Critical control missing Result exists but a confound/baseline check that could flip it has not run
[Y] Yanıltıcı Misleading Technically containing truth but framed to imply more than the evidence supports

Tier drift is itself a finding: when a claim silently moved from [H] to [K] between two documents without new evidence, flag it.

Modes — decide which one applies

Audit mode (retrospective). The user hands you an existing claim set — a summary, a review, a report, an AI-generated assessment — and wants to know how much of it survives scrutiny. Deliver the Audit Report (template in references/templates.md).

Registry mode (prospective). The user wants to track hypotheses going forward — experiments, predictions, work-pattern claims, product bets. Create or update a registry file using the Registry Entry template. The registry is a living Markdown document the user keeps in their project.

If the user’s request contains elements of both (“audit this, then set up tracking so it doesn’t happen again”), do the audit first, then seed the registry with the surviving [H] claims as its first entries.

The same discipline applies to code, with one structural difference: in a codebase, the claim and its evidence live in DIFFERENT artifacts (name / comment / docstring / test / implementation), and every hop between them must be verified separately. Verifying that a comment exists is not verifying that its claim is true.

Mode 3 — Code audit. Code is a claim set even without documentation. Read references/code-audit.md. Mode 4 — Bug-hypothesis registry. Each suspicion becomes a preregistered entry. Covered in references/code-audit.md. Mode 5 — Feature / PRD gate. Read references/feature-gate.md before gating a feature or PRD. Mode 6 — Meta-review. A hit rate ACROSS entries. It reads the registry itself. Mode 7 — Security probe. not exploited is not a pass. Read references/security-probe.md. Beyond software. Read references/domain-adaptation.md.

The full text this body was cut from lives, verbatim, in references/ — load a file when its situation arises, not upfront:

Tone and framing rules

Operating assumptions

Anti-patterns (refuse these politely)

References

Modes — full text

Moved verbatim from SKILL.md (v2.9, lines 45-123) when the body was pruned (EVAL-008/009). Nothing here was rewritten.

Two modes — decide which one applies

Audit mode (retrospective). The user hands you an existing claim set — a summary, a review, a report, an AI-generated assessment — and wants to know how much of it survives scrutiny. Deliver the Audit Report (template in references/templates.md).

Registry mode (prospective). The user wants to track hypotheses going forward — experiments, predictions, work-pattern claims, product bets. Create or update a registry file using the Registry Entry template. The registry is a living Markdown document the user keeps in their project.

If the user’s request contains elements of both (“audit this, then set up tracking so it doesn’t happen again”), do the audit first, then seed the registry with the surviving [H] claims as its first entries.

Modes 3–7

The same discipline applies to code, with one structural difference: in a codebase, the claim and its evidence live in DIFFERENT artifacts (name / comment / docstring / test / implementation), and every hop between them must be verified separately. Verifying that a comment exists is not verifying that its claim is true.

Mode 3 — Code audit. The user wants an existing codebase or repo analyzed. Code is a claim set even without documentation: every function name, comment, docstring, test, type hint, config value, and commit message makes a verifiable claim. Read references/code-audit.md before the first code audit. Deliverable: an evidence-tiered behavior report (which doubles as generated documentation when none exists) plus a Gap Map of broken promises, untested surfaces, and deferrals. For a repo too large to audit in one pass, do NOT force it — partition into sequential phases sharing one append-only registry (procedure in code-audit.md §A5.1; Coverage Ledger template in templates.md §5); this applies to Modes 4 and 5 too.

Mode 4 — Bug-hypothesis registry. Debugging is hypothesis testing usually performed as unrecorded HARKing. Each suspicion becomes a preregistered entry: mechanism, refutation test, threshold — run, record, never delete. “The fix worked” is a surprising positive: it needs the symmetric control (did THIS mechanism fix it, or would any perturbation have?). Covered in references/code-audit.md.

Mode 5 — Feature / PRD gate. A PRD is a claim set about the future: user-problem claims, value claims, cost claims, dependency claims — usually presented one tier above their evidence. Atomize and tier the PRD BEFORE building; preregister the success metric AND the kill condition; force alternatives (including the null alternative) before committing. Read references/feature-gate.md before gating a feature or PRD. This mode also generates feature candidates the user didn’t ask for, via the Gap Map and alternative-forcing — see that file’s “suggestion mechanism” section for what this can and cannot promise.

Mode 6 — Meta-review. Every ~10 entries, or on request: which hypothesis TYPES hit, which instruments proved reliable, where bias is accumulating. Not a summary — a hit rate ACROSS entries, the one thing a per-entry discipline cannot see from inside any entry. Its output revises the methodology: these instructions and the schema are claim sets too, subject to their own rules, so a change is justified and the old version is never deleted.

Mode 7 — Security probe. Modes 3–5 all start from a sentence someone wrote; a vulnerability is the sentence nobody wrote, so the engine that powers them cannot reach it (audit step 7 says why). Mode 7 changes the scenario source to a trust-boundary map plus an adversary model, and inverts what counts as evidence: not exploited is not a pass — a failed attempt and a clean scanner are silence and cap at [KKE] (R26); only a named control on every path to the asset promotes. R27 is R19 with the supplier changed. Read references/security-probe.md before the first security pass.

Beyond software. Modes 3/4/5 are domain-independent patterns (claim-vs-evidence hop audit; anomaly → rival-hypothesis registry; forward-commitment gate). For ANY domain outside software, read references/domain-adaptation.md — the five-question adaptation recipe, 14 per-domain hop maps and confound catalogs, and the constraints that transfer unchanged (append-only, DC-001 on individual hit rates, permanent [KKE] where symmetric controls are impossible).

Audit mode — full procedure

Moved verbatim from SKILL.md (v2.9, lines 124-214) when the body was pruned (EVAL-008/009). Nothing here was rewritten.

Audit mode — procedure

Read references/checklist.md before your first audit in a conversation; it lists the failure modes to hunt for and worked examples.

  1. Atomize. Decompose the document into individual checkable claims. A sentence like “you flagged r=0.997 as suspicious, which led to the init bug” is TWO claims (the flagging happened; it caused the discovery).
  2. Source each claim. For every claim, identify what evidence would verify it and whether that evidence is accessible (conversation history, files, commits, logs, web). Actually check what is checkable — open the file, search the history, run the number. A claim you cannot verify gets [H] with a note, not silent acceptance and not silent rejection.
  3. Tier each claim with the table above. Quote the claim, then the tag, then a one-line justification with the source.
  4. Hunt counter-examples. For every pattern-claim (“you always X”, “the system consistently Y”), actively search for instances of the opposite before accepting it. Report the search even when it comes up empty — “no counter-example found in N sources checked” is information; silence is not.
  5. Compute hit rates where possible. If the document praises someone’s judgment/predictions/instincts, reconstruct the full prediction record, not just the wins. A ~50-60% hit rate honestly reported is worth more than a 100% curated one — and say so.
  6. Name the missing card. Every summary format structurally omits something (failures, deferrals, abandoned lines, costs). State what this document’s format cannot show, and sketch it from available evidence.
  7. Probe the domain, not just the document. Every step above starts from a sentence someone wrote, so none of them can find a capability that was never claimed — an absence produces no claim to tier. Obtain a list of situations that actually occur in this field and test each against the model: expressible (fine), inexpressible and recorded as a deliberate boundary (fine — that is a decision), inexpressible and nowhere recorded (finding). You cannot write this list alone and must not pretend to: ask the domain owner, and record who supplied the scenarios as part of the coverage statement. Invented plausible-sounding scenarios are fiction wearing an evidence tag. Scenarios written after a gap was found are HARKing and prove nothing about coverage — say which ones were retrospective. Record it: probes.domain in the registry schema — the scenarios, supplied_by (auditor or none = self-report, and R19 refuses a tier-K coverage claim over it), and written_before_work. checklist.md item 12 carries an eight-class seed list for the asking — two at once, end of life, crossing a boundary, the person leaves, zero/one/many, undo and exit, out of order, who may see it. It is a prompt for the interview, not a list you may fill in alone. (Checklist item 12.)
  8. Re-assemble — audit the conjunctions, not only the claims. Step 1 took the document apart; a defect that exists only when two features hold at once was destroyed by that very act and cannot reappear in any later step. So put things back together deliberately: for each feature, list the existing guarantees it can touch, and for each pair ask “does that guarantee still hold while this feature is active?” Prioritise (a) derived signals — anything computed from an absence changes meaning the moment a new state exists — and (b) guarantees enforced call-site by call-site, which any new bulk surface can bypass wholesale. Check ordering too: some pairs are safe in one direction only, and the required order is part of the finding. A green test suite is not counter-evidence here: tests are written per feature, so they attest to the parts and are silent about the pair. Record it: probes.conjunction.pairs — feature, the guarantee it can reach, the order if the pair is asymmetric, and the outcome. A pair left unchecked blocks a tier-K coverage claim (R20); a pass that leaves no row cannot be told apart from one that never ran. (Checklist item 13.)
  9. Close the loop — for an ongoing target, CREATE the registry, do not offer it. When the audited thing is a living project (a repo, a backlog, a program) rather than a finished document, a one-shot audit cannot see gaps born after it ran. The audit’s final act is therefore to WRITE the registry file (Registry mode, seeded with the surviving [H] claims and every recurring failure class found) and to name the trigger that re-runs the audit — a phase boundary, a release, a fixed cadence. Ending with “shall I set up tracking?” is a known failure of this skill: the offer gets deferred, the project keeps closing tasks, and the next audit arrives only after a user stumbles on a gap. If the user declines the registry, record the refusal in the report so the absence of continuity is itself on the record. The registry is also where escapes land: when a defect surfaces later on ground this audit covered, RR-13 turns it into a class — probes.escaped with class_ref (the check that should have fired) or class_new (the probe that now exists because of it), which R21 requires. The scorecard already counted escapes; counting is not learning, and items 12 and 13 above were each born from an escape nobody recorded as one.
  10. Declare the HARKing status. Retrospective analysis selected its examples after seeing outcomes. Say this plainly in the report header — including about your own audit, which is also retrospective.
  11. Separate mechanism from motive. When explaining why a document is skewed, prefer structural explanations (selection pressure, format incentives) over intent attribution (“they designed it to flatter”) — unless intent is itself evidenced.

Registry mode — full procedure and schema rule history

Moved verbatim from SKILL.md (v2.9, lines 215-343) when the body was pruned (EVAL-008/009). Nothing here was rewritten.

Registry mode — procedure

  1. One entry per hypothesis, using the Registry Entry template (references/templates.md). The entry is written BEFORE the test runs.
  2. Lock thresholds numerically. “Improves things” is not a threshold; “ΔPPL ≤ −3%” or “counter-example rate < 1 per 10 sources” is. A threshold alone does not survive contact with a determined author, so three companion fields are locked with it and checked by R18: data status (does the data already exist, and have you seen it — registration against seen data is postdiction, not preregistration), stopping rule (the n or the condition that ends collection, without which “collect until it crosses” is always available), and exclusion rule (which observations get dropped, decided before seeing them; none is an answer, silence is not). The open-science preregistration templates ask all three before anything else, and each closes a door this file already argued should be shut.
  3. Write the refutation condition first, and check the two-sided informativeness requirement: both possible outcomes must teach something. If only success is informative, redesign the test.
  4. State the informativeness precondition where relevant: a test only counts if its preconditions held (e.g., a task-difference metric is meaningless if neither variant learned the task). Record “cell closed: precondition failed” as its own outcome type — distinct from [R].
  5. On surprising positive results: before promoting [H]→[K], ask what symmetric/confound control would distinguish “the specific claim” from “a generic alternative”, preregister that control as a sub-entry, and run it. The headline waits for the control.
  6. Status updates append, never overwrite. Each result gets a dated result block. Post-hoc reasoning is allowed but must be labeled “sonradan akıl yürütme / post-hoc, not preregistered”.
  7. Honesty annexes (dürüstlük şerhleri) are mandatory on every result: scope limits, sample size, single-seed caveats, instrument dependence.
  8. Prior art is declared, not discovered by reviewers. If the hypothesis has known relatives, name them in the entry and state where the originality claim actually lives.
  9. Name the arbiter of every threshold. A locked numeric threshold is only as strong as the judge that returns its verdict, and that judge is what quietly disappears when this discipline moves off code: in a test suite the runtime decides, in a strategy memo the author decides while the paperwork looks identical. Record the arbiter’s class — runtime (deterministic executor) / instrument (measurement independent of the author’s opinion) / third_party (a judge other than the author) / author (self-judged) / none — plus the concrete judge and the verdict latency. Two hard consequences: an author-arbitrated claim can never reach [K], it carries a permanent [KKE]; and with none the threshold is decorative, so say that and leave the entry at [S] rather than dressing an opinion in a number. Thresholds are calibrated against the arbiter’s own null and are never inherited across instruments.

From schema 1.4, four rules close the gaps between this file and the data. A [K] entry needs a threshold-meeting result or a cited external source (R9) — prior art is not evidence for the claim, only context for its originality. Thresholds name a quantity (R10), with a written justification as the escape hatch for a genuinely categorical verdict. Every entry carries a tier (R11), since “no untagged assertions” was never checked. And the mandatory fields of the entry template are present (R12): formal, a metric with a named instrument, cost, status, and prior_art — where “no known relatives” is an answer and an absent field is silence.

Bug entries and feature gates are hypotheses, not a different species. From schema 1.3 they carry the same fields and the same rules 1–9 above: a bug’s formal holds the MECHANISM (the symptom stays interpretation-free in its own field), a feature’s value metric IS metric and its success threshold IS threshold. Three rules are specific to them, and each was mandatory in references/ long before anything checked it: a feature names its kill condition (R13) and its alternatives including the null option (R14); a bug names at least one rival hypothesis (R15). The point of R15 is Mode 4’s whole point — never close on the first story that fits.

The Coverage Ledger lives in the registry (schema 1.5, rule R16). For a phased audit it is the one deliverable that DECIDES a tier — the whole-target coverage claim stays [H] until the MERGE row is done — and it used to live in a Markdown table no check could read. Keeping it in the registry follows code-audit.md’s own description of it as “the append-only registry used as the cross-phase carrier”, and it means R4 protects its rows for free: a re-scoped slice gets a new row, never an edit that erases the old one.

From schema 1.10, three rules close gaps the method itself had. All three came from one audit that MISSED four defects a user found by hand the same day — every one of them lying BETWEEN slices, and none reachable by the twenty-two rules that existed before them. Each closes a place the method trusted without checking:

The escapes, the numbers and what each rule would have caught are in references/code-audit.md and checklist.md items 14–16.

The validator also has a non-blocking warning channel (W1–W8): a missing two-sided statement, an entry written with no threshold or refutation and no result yet, a registry where every tiered entry is [K], and — from 1.10 — W6: every coverage phase done while the domain probe was never answered. R19 already blocks a tier-K claim over an unanswered probe, which is correct but narrow: an audit settling for [H] never meets it. In the run that produced W6 the probe was created on day one with supplied_by: none and seven phases closed around it. Waiving is a decision; silence is not. These advise rather than stop, for the same reason R8’s flag classes differ in force — a checker that can only block teaches people to write around it, which is a different skill from writing honestly.

Context economy

Moved verbatim from SKILL.md (v2.9, lines 344-373) when the body was pruned (EVAL-008/009). Nothing here was rewritten.

Context economy (long audits, long sessions)

An audit’s cost grows with the transcript, not with the finding. Every turn re-sends the whole conversation, so a large one-pass audit gets slower and more expensive with each exchange — and the last claims are graded under the worst conditions. The mechanism already exists in this skill: A5.1’s phased audit with an append-only registry as the carrier (references/code-audit.md). That is not a large-codebase special case; it is the general shape. Apply it whenever an audit will not finish in a few exchanges:

Tone, operating assumptions, anti-patterns — full text

Moved verbatim from SKILL.md (v2.9, lines 374-456) when the body was pruned (EVAL-008/009). Nothing here was rewritten.

Tone and framing rules

Operating assumptions (this skill runs inside someone else’s setup)

This skill is loaded into a host that already has its own instructions — a project’s CLAUDE.md, org policy, other skills. Those instructions take precedence over this file. That is correct, and it is also the most likely way Mizan fails: it degrades quietly. A short, softened audit still looks like an audit — tier tags in place, format intact, judgment gone. That is this skill’s own rigor cosplay anti-pattern, arrived at from the outside.

Anti-patterns (refuse these politely)

Reference index — full text

Moved verbatim from SKILL.md (v2.9, lines 457-488) when the body was pruned (EVAL-008/009). Nothing here was rewritten.

References