Audit AI model routing evidence integrity

Audit every live AI-model route against current version-matched local evaluation, uncontaminated temporal holdout, pricing freshness, residency, retention, reliability and genuinely independent provider fallback evidence, counting each workload's value at risk once.

Forecasting & survival Audit CTOs Tech CEOs IT managers Team leads Investors

What it's for

Lets leaders see which production AI workloads are backed by current, locally relevant evidence—and where vendor concentration, stale tests or data-boundary gaps put value at risk.

What you give it

Inputs split into evidence read from your connected systems, calibration your team owns, and numerical controls that affect precision but never the result's meaning.

Field	Type	Role	Required
as_of_period	integer ≥ 0	Your calibration	Yes
evaluation_results	array of objects (14 fields)	Evidence	Yes
max_detail_rows	integer ≥ 1, ≤ 500	Numerical control	Optional
maximum_benchmark_overlap_fraction	number ≥ 0, ≤ 1	Your calibration	Optional
maximum_evaluation_age_periods	integer ≥ 0, ≤ 10000	Your calibration	Optional
maximum_pricing_age_periods	integer ≥ 0, ≤ 10000	Your calibration	Optional
maximum_terminal_failure_fraction	number ≥ 0, ≤ 1	Your calibration	Optional
maximum_unsupported_high_impact_fraction	number ≥ 0, ≤ 1	Your calibration	Optional
minimum_distinct_providers_for_high_impact	integer ≥ 1, ≤ 100	Your calibration	Optional
minimum_evaluated_cases	integer ≥ 1	Your calibration	Optional
minimum_evidence_coverage	number ≥ 0, ≤ 1	Your calibration	Optional
model_endpoints	array of objects (7 fields)	Evidence	Yes
routing_rules	array of objects (5 fields)	Evidence	Yes
workload_classes	array of objects (7 fields)	Evidence	Yes

Each `evaluation_results` record

Field	Type	Required
benchmark_overlap_fraction	number (≥ 0, ≤ 1)	Yes
blinded	boolean	Yes
endpoint_id	string (non-empty)	Yes
evaluated_case_count	integer (≥ 1)	Yes
evaluation_period	integer (≥ 0)	Yes
evidence_verified	boolean	Yes
id	string (non-empty)	Yes
independently_reviewed	boolean	Yes
model_version	string (non-empty)	Yes
p95_latency_ms	number (> 0)	Yes
quality_score	number (≥ 0, ≤ 1)	Yes
temporally_held_out	boolean	Yes
terminal_failure_fraction	number (≥ 0, ≤ 1)	Yes
workload_class_id	string (non-empty)	Yes

Example input

{
  "as_of_period": 10,
  "evaluation_results": [
    {
      "benchmark_overlap_fraction": 0,
      "blinded": true,
      "endpoint_id": "model-a",
      "evaluated_case_count": 50,
      "evaluation_period": 10,
      "evidence_verified": true,
      "id": "support-model-a",
      "independently_reviewed": true,
      "model_version": "a-1",
      "p95_latency_ms": 150,
      "quality_score": 0.9,
      "temporally_held_out": true,
      "terminal_failure_fraction": 0.01,
      "workload_class_id": "support"
    },
    {
      "benchmark_overlap_fraction": 0,
      "blinded": true,
      "endpoint_id": "model-b",
      "evaluated_case_count": 50,
      "evaluation_period": 10,
      "evidence_verified": true,
      "id": "support-model-b",
      "independently_reviewed": true,
      "model_version": "b-1",
      "p95_latency_ms": 150,
      "quality_score": 0.9,
      "temporally_held_out": true,
      "terminal_failure_fraction": 0.01,
      "workload_class_id": "support"
    }
  ],
  "model_endpoints": [
    {
      "data_retention_compliant": true,
      "evidence_verified": true,
      "id": "model-a",
      "model_version": "a-1",
      "pricing_as_of_period": 10,
      "provider_id": "provider-a",

Truncated for display — the full payload is 81 lines.

What you get back

This is the actual output of running the example above — computed by the same function the platform calls, not an illustration.

Example output

{
  "assumptions": [
    "Workload classes, endpoint model versions, prices, residency, retention posture, local evaluations and live routes share one frozen control-plane epoch. Evaluation cases represent future production demand rather than a benchmark selected after seeing results.",
    "A route is supported only by current blinded, temporally held-out, independently reviewed local evidence with controlled benchmark overlap, sufficient cases and workload-owned quality/latency/failure gates.",
    "Provider diversity is operational independence only when provider identities and endpoint serving regions reflect real common-mode infrastructure; several endpoint names from one provider do not create diversity.",
    "A failed audit is not proof that a model, vendor, team or person caused harm and never authorizes deployment, procurement, data transfer, privilege change, surveillance, investigation or vendor termination."
  ],
  "configuration": {
    "as_of_period": 10,
    "maximum_benchmark_overlap_fraction": 0,
    "maximum_evaluation_age_periods": 1,
    "maximum_pricing_age_periods": 1,
    "maximum_terminal_failure_fraction": 0.05,
    "maximum_unsupported_high_impact_fraction": 0,
    "minimum_distinct_providers_for_high_impact": 2,
    "minimum_evaluated_cases": 30,
    "minimum_evidence_coverage": 0.95,
    "value_rule": "count_each_unique_workload_class_value_once"
  },
  "decision": "ai_model_routing_evidence_integrity_supported",
  "duplicate_workload_endpoint_evaluations": [],
  "duplicate_workload_routing_rules": [],
  "evaluation_diagnostics": [
    {
      "benchmark_overlap_fraction": 0,
      "current_model_version": "a-1",
      "endpoint_id": "model-a",
      "evaluated_case_count": 50,
      "evaluated_model_version": "a-1",
      "failed_gates": [],
      "p95_latency_ms": 150,
      "provider_id": "provider-a",
      "quality_score": 0.9,
      "terminal_failure_fraction": 0.01,
      "valid": true,
      "workload_class_id": "support"
    },
    {
      "benchmark_overlap_fraction": 0,
      "current_model_version": "b-1",
      "endpoint_id": "model-b",
      "evaluated_case_count": 50,
      "evaluated_model_version": "b-1",
      "failed_gates": [],

Truncated for display — the full payload is 88 lines.

How it works

Forecasting & survival — Estimate when something completes or fails, with censoring and unresolved work handled honestly rather than dropped.

1 Freeze workload requirements, endpoint versions and price epochs, blinded local evaluation results and live routing rules at one point in time.
2 Require every primary and fallback endpoint to match the deployed model version and pass local case-count, temporal-holdout, benchmark-overlap, quality, latency, terminal-failure, residency, retention and freshness gates.
3 Retain duplicate claims, test high-impact routes for real provider diversity, union unsupported workload value once, and abstain when any route or evidence gate fails.

Before you trust it

Every tool in the catalog ships with the conditions under which its answer is meaningful — and the conditions under which it should abstain instead of guessing.

Assumptions & guardrails

Training examples precede their outcomes, censoring and unresolved work are represented, and deployment populations remain comparable to validation cohorts.
Endpoint IDs and model versions are immutable within the audit epoch; local test cases represent future production demand; provider identities reflect common-mode infrastructure rather than marketing labels.
A predictive interval or risk estimate is not a deadline promise, causal explanation, or individual-performance judgment.
A failed route audit is an abstention and reconciliation queue, not proof of vendor harm, security compromise, employee conduct or authority for deployment, procurement, data transfer, surveillance or termination.

Minimum evidence

workload_classes: required and organization-defined
model_endpoints: required and organization-defined
evaluation_results: required and organization-defined
routing_rules: required and organization-defined
as_of_period: required and organization-defined

How to validate it

Validate on future periods or held-out aggregate units, compare with a simple baseline, and require stability across plausible metric definitions and decision thresholds.

Calibrating it to your org

Same for everyone

The mathematical kernel, validation rules, method version, and JSON output semantics are organization-independent; no tenant-trained coefficients or company benchmark is embedded in the function.

Specific to you

point-in-time workload-endpoint evaluation projection joined on tenant, workload, immutable endpoint and exact model version before the route effective time
workload/impact taxonomy, quality/latency/failure requirements, value and currency horizon, provider common-mode identity, residency/retention policy, evaluation case independence, temporal holdout, benchmark overlap, review provenance, freshness and coverage gates

Calibration workflow

1 Define the management decision, target outcome, aggregate unit, privacy boundary, cadence, and prediction/intervention horizon for this organization.
2 Build a tenant-scoped historical cohort using only information available before each prediction or decision; preserve zero periods, censoring, assignment probabilities, and unresolved outcomes when the method requires them.
3 Estimate statistical parameters on training history, but obtain costs, utilities, risk tolerance, practical-effect thresholds, capacity, and policy constraints from accountable decision owners.
4 Validate on later time windows or held-out aggregate units at the deployment grain, against a simple baseline and the function-specific validation strategy.
5 Deploy only if the returned decision clears evidence, overlap, calibration, robustness, and guardrail checks; warning, unsupported, schema-gap, and fallback decisions are abstentions.
6 Monitor realized outcomes, data drift, coverage, and decision regret; recalibrate at a governed cadence or after a detected regime/definition change, never merely because a stakeholder dislikes the result.

Call it from your AI

You don't wire up 388 tools in your MCP client. The GitRevio MCP server exposes 18 tools, three of which let an agent search the catalog, read a tool's schema, and run it — so the assistant finds this one on its own.

gitrevio_capabilities_search
  { "q": "audit every live aimodel route against" }
  → finds "audit_ai_model_routing_evidence_integrity"

gitrevio_capability_describe
  { "capability_id": "audit_ai_model_routing_evidence_integrity" }
  → returns the input schema and agent guidance shown on this page

gitrevio_capability_run
  { "capability_id": "audit_ai_model_routing_evidence_integrity", "arguments": { ... } }
  → returns the result shown above

Works in Claude Desktop, Claude Code, Cursor, Cline, Continue.dev, Goose and Aider. See the MCP server.

Related tools

Forecast AI inference economics

Forecast full AI-inference cost, retry demand, terminal-failure loss, gross value and economic-loss VaR/CVaR with tenant-local Gamma-Poisson, Beta-Binomial and partially pooled lognormal models plus shared provider-outage scenarios.

Forecasting & survival

Optimize AI model routing portfolio

Choose one evidenced AI-model route per workload on a value/CVaR Pareto frontier under hard privacy, residency, retention, quality, latency, endpoint-capacity, route-availability, provider-diversity, concentration, budget and dependency constraints.

Constrained optimization

Audit AI capability fallback integrity

Prove that every aggregate capability required when AI is unavailable has a current approved runbook and a sufficiently large, timely, successful, independently observed exercise conducted with AI actually disabled.

Statistical audit & measurement

Audit AI code change evidence integrity

Prove that aggregate AI-assisted coding evidence comes from prospectively registered, nonoverlapping treatment/control studies with immutable assignment, configuration, trace and mature-outcome denominators before anyone estimates an effect.

Causal inference & experiment design

Audit AI inference cost allocation integrity

Reconcile provider AI invoices bottom-up to workload and route usage, price terms, cached requests, retries, fixed charges and credits without combining currencies or silently allocating unexplained spend.