Pablo Zavala · AI Safety Evaluation · Research Engineering
Tribunal: When AI Decisions Need a Ledger
Tribunal turns high-stakes AI output into a reviewable decision record: blind proposals, sealed commitments, critique, vetoes, dissent, ratification, and a hash-chained ledger.
!Tribunal data dashboard built from real UI panels, ledger counts, and a decision-flow diagram
A Paragraph Leaves Too Little To Audit
When an AI system denies a loan, flags a benefit claim, rejects care, or removes a post, the person affected often receives little more than a paragraph. Fluent language can cite a policy while omitting the institutional facts that matter: what the system checked, who objected, what the panel overruled, and why one reason survived another.
The missing record is why I built Tribunal. High-stakes AI needs more than a confident rationale. High-stakes AI needs an inspectable procedure around the decision.
Tribunal Makes Explanation Procedural
Tribunal replaces one private completion with a recorded panel process. The decision is built one span at a time. A span is a small unit of final text: a disposition, a reason, a disclosure, or a STOP decision.
For each span, candidate proposals are drafted blind, sealed by hash, challenged, revised, reviewed for safety, ratified under a named rule, and committed to a ledger. Material dissent remains attached to the record instead of disappearing into the final answer.
The six seats carry distinct duties: evidence, adversary, law and policy, affected party, safety with veto power, and concision. The repo supports live modes where seats can be assigned across providers. Offline mode uses scripted stand-ins for CI and demos, so a recorded offline demo is verification evidence rather than live model evidence.
Tribunal changes the object being evaluated. A normal model gives the reader text. Tribunal gives the reader text plus a reviewable record of how the text survived challenge.
The mechanism stays simple enough to audit:
1. A case is opened.
2. Seats draft blind proposals.
3. Each proposal is sealed by hash before reveal.
4. The proposals are revealed and critiqued through seat-blinded feedback packets with identity fields stripped and per-recipient order rotation.
5. Seats revise after seeing objections.
6. Safety can veto.
7. A named ratification rule elects the span.
8. Dissent is preserved.
9. The span is committed, or STOP wins.
10. The next span repeats the process.
Because every event is typed and chained to the prior event, a changed reason, reordered step, or deleted late objection breaks hash-chain verification inside the supplied ledger.
Verifiable Surfaces
The public repository exposes the proof surfaces directly: `npm test`, `npm run demo`, local `POST /api/verify`, tamper detection, replayable ledgers, and anchored head hashes.
The public anchor table lists committed runs and their heads. One anchored offline lending run, `run_e6c6225c0d49`, contains 86 events, scores `12/12` on auditability, and verifies against its published head hash. The same ledger also shows the hard boundary of the project: the final answer adopts the false `52%` debt-to-income rationale from the draft case.
The flawed lending run shows why Tribunal's claim must stay procedural. A ledger can preserve dissent, expose the premise, name the rule, and make a flawed judgment easier to challenge. The record leaves a bad majority a bad majority.
Governance Demands Records
Across the cited regimes, the demand increasingly centers on reasons, records, oversight, and contestability.
In credit, the CFPB has said that complex algorithms still require specific and accurate adverse-action reasons. In content moderation, the EU Digital Services Act requires statements of reasons for certain restrictions. The EU AI Act framework sets logging, documentation, and human-oversight obligations for high-risk AI systems, with application dates varying by system and transition rule. GDPR Article 22 gives people rights around certain solely automated decisions with legal or similarly significant effects.
Tribunal falls short of compliance certification. Compliance depends on domain, jurisdiction, data, notice, retention, privacy controls, reviewer authority, appeal rights, and operating procedure.
Even with those limits, the governance pattern is procedural. A serious governance layer should help show which facts a system relied on, where automation entered, where a person could intervene, what objections were raised, and whether the record still verifies.
Tribunal is a prototype aimed at that pattern.
Auditability Rather Than Truth
The hardest limitation is also the most important honesty test: a ledger can faithfully record a flawed decision.
Tribunal's proof is procedural. The decision left a replayable record. A reviewer can inspect the premises, objections, rule, and dissent. Replayability is weaker than correctness, yet far more inspectable than a polished paragraph without an audit trail.
For that reason, the Tribunal scorecard is called A1-A12 auditability, rather than an accuracy benchmark. The checklist asks whether the artifact contains blind commitments, public warrants, seat-blinded feedback, order rotation, substantive revision, safety veto paths, named ratification, preserved dissent, deliberation memory, hash-chain verification, STOP handling, and typed events.
A single unledgered model completion scores `0/12` by construction because the completion lacks the artifact. The contrast should be read as a structural comparison: ordinary completions lack the kind of record this checklist measures.
SHAP, LIME, And Chain-Of-Thought Ask Different Questions
Feature-attribution tools answer a different question. SHAP and LIME can help explain which features moved a score in a model. Such tools can be useful for debugging feature-based systems, while a decision procedure requires a different artifact.
Tribunal asks who checked the file, which fact was disputed, what changed after critique, what safety vetoed, and whether the final answer still matches the committed spans.
The distinction matters because post-hoc explanations can be fragile. Cynthia Rudin has argued that high-stakes settings should prefer interpretable models over black-box explanations where possible. Slack, Hilgard, Jia, Singh, and Lakkaraju showed that adversarial scaffolding can make biased classifiers produce innocent-looking LIME and SHAP explanations. Turpin et al. show that chain-of-thought explanations can misrepresent why a model answered as it did.
Therefore, Tribunal avoids claims about a model's private mind. The system records public warrants and public objections during generation, then governs the procedure around the answer.
Affected People Need Contestable Records
If a borrower is denied, the record should show whether the debt ratio was recomputed from the file or copied from a bad worksheet.
If a patient is denied care, the record should show whether the reviewer used the current guideline, whether a clinician objected, and whether automation was the sole basis or a qualified human made an independent review.
If a benefits claimant is accused of fraud, the record should show whether an employer-name mismatch was reconciled against an EIN and wage total before penalties attached.
If a journalist's post is removed, the record should show whether the system distinguished the person making a threat from the reporter quoting the threat.
Loan, care, benefits, and moderation examples are procedural questions rather than abstract fairness slogans. Each asks what was checked before power moved.
Tribunal's answer is that the check itself should be visible.
Honest Claim
Tribunal is a prototype rather than a court, compliance certification, or proof that six models are wiser than one. The prototype turns an automated decision into a contestable artifact.
The promise is disciplined:
- record the proposal before the room sees it;
- record the objection before the final answer ships;
- record the veto when safety blocks a path;
- record the dissent when the minority loses;
- record the rule that binds the span;
- retain or publish enough of the ledger, under appropriate access and privacy controls, that an authorized reviewer can verify it.
The shift I care about is from model output toward reviewable record.
High-stakes AI should end with more than "the model said so": a decision an authorized reviewer or affected person can inspect, contest, replay where appropriate, and refuse to accept.
Sources And Boundaries
- Public system repository: `pazare/tribunal`.
- Anchored evidence: `runs/ANCHORS.md`, `runs/run_e6c6225c0d49/meta.json`, `audit.json`, and `ledger.json`.
- Local verification on July 6, 2026: `npm test` passed 31 tests; `npm run demo` printed `VERIFY: OK`; local anchor checking confirmed the cited anchored run matches its published head.
- Boundary: Tribunal supports auditability claims for the artifact, separate from answer accuracy, fairness certification, legal compliance, perfect anonymization, or live cross-provider proof in the offline run.
- Visual provenance: the image above starts from actual Tribunal UI panels, run metadata, audit counts, and ledger-event bars, then receives AI polishing for spacing and clarity.