Backstop · anatomy · The Agentic Bank

A second-line audit in the spirit of SR 11-7: it tests whether the decision is supported and consistent, and catches an inconsistent decline.

Second line. Audits the decision for soundness before the Keystone gate; a fail holds the commit.

Methodology

1 Consistency: is the decision consistent with what policy implies for this file, and with how similarly-situated applicants were decided? Name any inconsistent comparator.
2 Reason accuracy: do the stated factors genuinely support the decision? A decline must rest on a real, file-grounded basis.
3 Override / exception: a deviation from policy (a within-policy file declined, an out-of-policy file approved) must be a documented, justified override, or it is an exception finding.
4 Fair-lending soundness: weigh the fair-lending review if present; an unexplained inconsistent decline is itself a fair-lending risk. Never use a prohibited basis.
5 Rate pass / findings / fail. find_similar is for consistency on credit merit only, never protected attributes.

Sandboxed tools

In-process MCP tools only. Built-in tools disabled, settingSources [], permissionMode bypassPermissions. The only ground truth is the tools.

get_application read

The credit-relevant fields.

get_credit_report read

The bureau credit file.

get_policy read

The credit policy, to test whether the decision is supported.

find_similar read

Similarly-situated applicants for consistency testing on credit merit ONLY — the service strips prohibited basis from Backstop's comparators.

record_assessment commit

Commits the model-risk assessment. Called exactly once.

Outcome contract

A rating (pass / findings / fail) with the consistency check, the reason-accuracy check, whether any override is justified, a fair-lending note, the issues, and a recommendation.

▸ the validated schema (ModelRiskAssessment)

type ModelRiskAssessment = {
  rating: "pass" | "findings" | "fail"
  consistencyCheck: string
  reasonAccuracy: string
  overrideJustified: boolean | null   // null when no policy deviation
  fairLendingNote: string
  issues: string[]
  recommendation: string
}

Grounding

real SR 11-7 model-risk principles. The Federal Reserve / OCC model-risk-management framework, applied conceptually: test that the decision is defensible, do not re-underwrite.

synthetic Credit policy (the consistency benchmark). The thresholds and comparators a decision is tested against. Synthetic, internal.

Governance seams

The three trust boundaries a runtime governance layer clamps onto. Shown here ungoverned, so you can see exactly what would be inspected, redacted, or blocked.

Model plane

context in

The Backstop methodology plus the decision under audit, the file, and the fair-lending review when present.

governor → Confirm the audit reasons only from credit merit, never a protected attribute.

Tool plane

file + comparators

The reads plus find_similar, restricted to credit-merit consistency: the service strips the prohibited-basis block from Backstop's comparators.

governor → Confirm the merit-consistency path never receives protected attributes — here it is enforced at the service, not just instructed.

Data plane

the commit

The recorded rating written to the case journal; a fail feeds the Keystone gate.

governor → Route a fail to a hold for remediation rather than a pass-through.

Backstop does not re-underwrite; it tests whether the decision that was made is defensible. find_similar is restricted to credit-merit consistency: the service strips the prohibited-basis block from its comparators, so the restriction is enforced rather than merely instructed. A fail, an unjustified override, or a fair-lending violation holds the commit at the Keystone gate.

Evals & guardrails

3 / 3 golden Graded on the rating and naming the specific inconsistency or unsupported factor.

Inconsistent decline. Catches a decline that contradicts the policy thresholds the file actually met, and rates it findings / fail.
Sound decision. Clears a consistent, well-supported decision without manufacturing issues.

Orchestration

Specialist (second line) model-risk audit

⇢Audits Spark's decision, weighs Equipoise's review
⇢A fail holds the Keystone gate