Audit AI routing experiment integrity

Audit prospectively registered AI-route experiments at the randomization-unit/period/route grain, reconciling logged propensities, allocation fidelity, pre-assignment balance, crossover, outcome maturity, simultaneous-experiment overlap and unique value at risk before any causal effect is reported.

Causal inference & experiment design Audit CTOs Tech CEOs IT managers Team leads Investors

What it's for

Prevents AI model A/B tests from turning reconstructed traffic splits, immature quality labels or overlapping experiments into confident vendor and routing claims.

What you give it

Inputs split into evidence read from your connected systems, calibration your team owns, and numerical controls that affect precision but never the result's meaning.

Field	Type	Role	Required
as_of_period	integer ≥ 0	Your calibration	Yes
assignment_cells	array of objects (12 fields)	Evidence	Yes
covariate_balance_cells	array of objects (9 fields)	Evidence	Yes
experiments	array of objects (12 fields)	Evidence	Yes
max_detail_rows	integer ≥ 1, ≤ 500	Numerical control	Optional
maximum_absolute_allocation_deviation	number ≥ 0, ≤ 1	Your calibration	Optional
maximum_cross_experiment_overlap_fraction	number ≥ 0, ≤ 1	Your calibration	Optional
maximum_crossover_fraction	number ≥ 0, ≤ 1	Your calibration	Optional
maximum_standardized_mean_difference	number ≥ 0, ≤ 10	Your calibration	Optional
minimum_evidence_coverage	number ≥ 0, ≤ 1	Your calibration	Optional
minimum_mature_outcome_fraction	number ≥ 0, ≤ 1	Your calibration	Optional
minimum_randomization_units	integer ≥ 2	Your calibration	Optional
minimum_total_assignments	integer ≥ 1	Your calibration	Optional

Each `assignment_cells` record

Field	Type	Required
assigned_request_count	integer (≥ 0)	Yes
assignment_logged_before_request	boolean	Yes
assignment_probability	number (≥ 0, ≤ 1)	Yes
crossover_request_count	integer (≥ 0)	Yes
eligible_request_count	integer (≥ 0)	Yes
evidence_verified	boolean	Yes
experiment_id	string (non-empty)	Yes
id	string (non-empty)	Yes
matured_outcome_count	integer (≥ 0)	Yes
period	integer (≥ 0)	Yes
randomization_unit_id	string (non-empty)	Yes
route_id	string (non-empty)	Yes

Example input

{
  "as_of_period": 12,
  "assignment_cells": [
    {
      "assigned_request_count": 5,
      "assignment_logged_before_request": true,
      "assignment_probability": 0.5,
      "crossover_request_count": 0,
      "eligible_request_count": 10,
      "evidence_verified": true,
      "experiment_id": "support-route-test",
      "id": "support-cell-0-route-a",
      "matured_outcome_count": 5,
      "period": 10,
      "randomization_unit_id": "account-00",
      "route_id": "route-a"
    },
    {
      "assigned_request_count": 5,
      "assignment_logged_before_request": true,
      "assignment_probability": 0.5,
      "crossover_request_count": 0,
      "eligible_request_count": 10,
      "evidence_verified": true,
      "experiment_id": "support-route-test",
      "id": "support-cell-0-route-b",
      "matured_outcome_count": 5,
      "period": 10,
      "randomization_unit_id": "account-00",
      "route_id": "route-b"
    },
    {
      "assigned_request_count": 5,
      "assignment_logged_before_request": true,
      "assignment_probability": 0.5,
      "crossover_request_count": 0,
      "eligible_request_count": 10,
      "evidence_verified": true,
      "experiment_id": "support-route-test",
      "id": "support-cell-1-route-a",
      "matured_outcome_count": 5,
      "period": 10,
      "randomization_unit_id": "account-01",
      "route_id": "route-a"

Truncated for display — the full payload is 608 lines.

What you get back

This is the actual output of running the example above — computed by the same function the platform calls, not an illustration.

Example output

{
  "assumptions": [
    "Assignment cells are a complete zero-retaining projection at the declared randomization-unit, period and route grain. Logged propensities existed before requests and were not reconstructed from realized allocation shares.",
    "Covariates were measured before assignment, every declared route is represented, crossover and outcome maturity use the same eligible cohort, and overlapping experiments identify genuine simultaneous exposure rather than reused labels.",
    "Passing integrity permits a pre-registered effect analysis; it does not prove route superiority, causal value, model safety, provider independence or authorization to deploy, procure, transfer data or change traffic.",
    "Rows describe aggregate workloads and randomization units. The audit must not be used to infer named-person performance, intent, misconduct or employment action."
  ],
  "configuration": {
    "as_of_period": 12,
    "maximum_absolute_allocation_deviation": 0.1,
    "maximum_cross_experiment_overlap_fraction": 0,
    "maximum_crossover_fraction": 0.02,
    "maximum_standardized_mean_difference": 0.1,
    "minimum_evidence_coverage": 0.95,
    "minimum_mature_outcome_fraction": 0.9,
    "minimum_randomization_units": 20,
    "minimum_total_assignments": 100,
    "value_rule": "count_each_experiment_value_at_risk_once"
  },
  "decision": "ai_routing_experiment_integrity_supported",
  "experiment_diagnostics": [
    {
      "assignment_count": 200,
      "cross_experiment_overlap_fraction": 0,
      "crossover_fraction": 0,
      "experiment_id": "support-route-test",
      "experiment_integrity_supported": true,
      "failed_gates": [],
      "incomplete_balance_covariates": [],
      "invalid_propensity_stratum_count": 0,
      "mature_outcome_fraction": 1,
      "maximum_absolute_allocation_deviation": 0,
      "maximum_standardized_mean_difference": 0.01,
      "randomization_unit_count": 20,
      "route_count": 2,
      "workload_class_id": "support"
    }
  ],
  "failed_gates": [],
  "method": "prospectively_registered_logged_propensity_ai_route_experiment_audit_v1",
  "summary": {
    "evidence_coverage": 1,
    "experiment_count": 1,
    "supported_experiment_count": 1,

Truncated for display — the full payload is 49 lines.

How it works

Causal inference & experiment design — Separate what a change caused from what merely moved alongside it.

1 Freeze registered experiment scope, route set, randomization unit, period, primary metric, stopping rule and value before reading assignments or outcomes.
2 Require a complete route row for every unit-period stratum, propensities summing to one, a common eligible denominator, pre-request logging and observed allocation close to the declared probabilities.
3 Reconcile pre-assignment covariate balance, crossover, mature outcomes, cross-experiment exposure and evidence; count each unsupported experiment's value once and abstain before effect estimation when a gate fails.

Before you trust it

Every tool in the catalog ships with the conditions under which its answer is meaningful — and the conditions under which it should abstain instead of guessing.

Assumptions & guardrails

Assignment or adoption timing, interference policy, overlap, attrition, and outcome availability match the estimand encoded by the method.
Logged propensities existed before each request, zero route cells are retained, randomization units prevent the declared interference, covariates predate assignment and outcome maturity follows the registered horizon.
A causal label is warranted only when design and diagnostic requirements pass; otherwise treat the result as descriptive or abstain.
Passing permits the registered causal analysis but does not prove route superiority or safety, and never authorizes traffic, deployment, procurement, data transfer, surveillance or a decision about a provider, team or person.

Minimum evidence

experiments: required and organization-defined
assignment_cells: required and organization-defined
covariate_balance_cells: required and organization-defined
as_of_period: required and organization-defined

How to validate it

Preserve the assignment/adoption design and validate overlap, pre-period or placebo diagnostics, attrition, interference policy, and cluster-level uncertainty; never tune on the estimated effect.

Calibrating it to your org

Same for everyone

The mathematical kernel, validation rules, method version, and JSON output semantics are organization-independent; no tenant-trained coefficients or company benchmark is embedded in the function.

Specific to you

complete zero-inclusive route cells for every randomized unit-period stratum plus a simultaneous-experiment exposure join, preserving the exact registration and assignment-log versions visible before outcomes
experiment ownership and registration time, randomization/interference unit, allocation and stopping rules, outcome maturity, covariate timing, balance/support/crossover/overlap gates, unique value/currency/horizon and privacy-safe aggregate identity

Calibration workflow

1 Define the management decision, target outcome, aggregate unit, privacy boundary, cadence, and prediction/intervention horizon for this organization.
2 Build a tenant-scoped historical cohort using only information available before each prediction or decision; preserve zero periods, censoring, assignment probabilities, and unresolved outcomes when the method requires them.
3 Estimate statistical parameters on training history, but obtain costs, utilities, risk tolerance, practical-effect thresholds, capacity, and policy constraints from accountable decision owners.
4 Validate on later time windows or held-out aggregate units at the deployment grain, against a simple baseline and the function-specific validation strategy.
5 Deploy only if the returned decision clears evidence, overlap, calibration, robustness, and guardrail checks; warning, unsupported, schema-gap, and fallback decisions are abstentions.
6 Monitor realized outcomes, data drift, coverage, and decision regret; recalibrate at a governed cadence or after a detected regime/definition change, never merely because a stakeholder dislikes the result.

Call it from your AI

You don't wire up 388 tools in your MCP client. The GitRevio MCP server exposes 18 tools, three of which let an agent search the catalog, read a tool's schema, and run it — so the assistant finds this one on its own.

gitrevio_capabilities_search
  { "q": "audit prospectively registered airoute experiments at" }
  → finds "audit_ai_routing_experiment_integrity"

gitrevio_capability_describe
  { "capability_id": "audit_ai_routing_experiment_integrity" }
  → returns the input schema and agent guidance shown on this page

gitrevio_capability_run
  { "capability_id": "audit_ai_routing_experiment_integrity", "arguments": { ... } }
  → returns the result shown above

Works in Claude Desktop, Claude Code, Cursor, Cline, Continue.dev, Goose and Aider. See the MCP server.

Related tools

Forecast AI route quality cost drift

Forecast route-level quality, inference cost, p95 latency, breach timing, net value and economic-loss VaR/CVaR with partially pooled Bayesian trends and one common disruption state shared across every route on the same provider.

Sequential Bayesian & bandits

Optimize safe AI routing exploration portfolio

Allocate bounded production traffic to one safe challenger per AI workload by posterior-predictive knowledge gradient, maximizing net learning value under local quality/harm evidence, privacy, latency, provider diversity, shared endpoint capacity, exploration budget, provider concentration and regret CVaR constraints.

Sequential Bayesian & bandits

Audit AI capability fallback integrity

Prove that every aggregate capability required when AI is unavailable has a current approved runbook and a sufficiently large, timely, successful, independently observed exercise conducted with AI actually disabled.

Statistical audit & measurement

Audit AI code change evidence integrity

Prove that aggregate AI-assisted coding evidence comes from prospectively registered, nonoverlapping treatment/control studies with immutable assignment, configuration, trace and mature-outcome denominators before anyone estimates an effect.

Causal inference & experiment design

Audit AI inference cost allocation integrity

Reconcile provider AI invoices bottom-up to workload and route usage, price terms, cached requests, retries, fixed charges and credits without combining currencies or silently allocating unexplained spend.