Optimize safe AI routing exploration portfolio

Allocate bounded production traffic to one safe challenger per AI workload by posterior-predictive knowledge gradient, maximizing net learning value under local quality/harm evidence, privacy, latency, provider diversity, shared endpoint capacity, exploration budget, provider concentration and regret CVaR constraints.

Sequential Bayesian & bandits Optimize CTOs Tech CEOs IT managers Team leads Investors

What it's for

Lets CTOs learn which AI route is better without betting the whole workload: exploration traffic earns information only when local safety evidence, shared capacity and downside budgets permit it.

What you give it

Inputs split into evidence read from your connected systems, calibration your team owns, and numerical controls that affect precision but never the result's meaning.

Field	Type	Role	Required
allocation_step_fraction	number ≥ 0.001, ≤ 1	Your calibration	Optional
beam_width	integer ≥ 1, ≤ 10000	Numerical control	Optional
exact_enumeration_limit	integer ≥ 1, ≤ 1000000	Your calibration	Optional
exploration_request_budget	integer ≥ 0	Your calibration	Yes
harm_tolerance_per_request	number ≥ 0	Your calibration	Optional
max_detail_rows	integer ≥ 1, ≤ 500	Numerical control	Optional
maximum_cvar_exploration_regret	any	Your calibration	Optional
maximum_expected_exploration_regret	any	Your calibration	Optional
maximum_posterior_harm_probability	number ≥ 0, ≤ 1	Your calibration	Optional
maximum_provider_concentration_fraction	number ≥ 0, ≤ 1	Your calibration	Optional
minimum_distinct_providers_for_high_impact	integer ≥ 1, ≤ 100	Your calibration	Optional
minimum_posterior_quality_probability	number ≥ 0, ≤ 1	Your calibration	Optional
minimum_resolved_outcomes_per_route	integer ≥ 1	Your calibration	Optional
prior_failures	number > 0	Your calibration	Optional
prior_successes	number > 0	Your calibration	Optional
provider_scenarios	array of objects (4 fields)	Evidence	Yes
risk_aversion	number ≥ 0	Your calibration	Optional
route_candidates	array of objects (13 fields)	Evidence	Yes
seed	integer ≥ 0	Numerical control	Optional
simulations	integer ≥ 100, ≤ 100000	Numerical control	Optional
tail_probability	number ≥ 0.001, ≤ 0.5	Your calibration	Optional
workload_classes	array of objects (12 fields)	Evidence	Yes

Each `route_candidates` record

Field	Type	Required
cost_per_request	number (≥ 0)	Yes
data_retention_compliant	boolean	Yes
endpoint_capacity_requests	integer (≥ 0)	Yes
endpoint_id	string (non-empty)	Yes
evidence_verified	boolean	Yes
id	string (non-empty)	Yes
p95_latency_ms	number (> 0)	Yes
provider_id	string (non-empty)	Yes
resolved_failure_count	integer (≥ 0)	Yes
resolved_success_count	integer (≥ 0)	Yes
route_id	string (non-empty)	Yes
serving_region	string (non-empty)	Yes
workload_class_id	string (non-empty)	Yes

Example input

{
  "exploration_request_budget": 20,
  "maximum_posterior_harm_probability": 0.5,
  "minimum_posterior_quality_probability": 0.9,
  "provider_scenarios": [
    {
      "evidence_verified": true,
      "id": "base",
      "probability": 0.8,
      "unavailable_provider_ids": []
    },
    {
      "evidence_verified": true,
      "id": "provider-a-outage",
      "probability": 0.1,
      "unavailable_provider_ids": [
        "provider-a"
      ]
    },
    {
      "evidence_verified": true,
      "id": "provider-b-outage",
      "probability": 0.1,
      "unavailable_provider_ids": [
        "provider-b"
      ]
    }
  ],
  "route_candidates": [
    {
      "cost_per_request": 0.1,
      "data_retention_compliant": true,
      "endpoint_capacity_requests": 100,
      "endpoint_id": "endpoint-a",
      "evidence_verified": true,
      "id": "support-a",
      "p95_latency_ms": 150,
      "provider_id": "provider-a",
      "resolved_failure_count": 20,
      "resolved_success_count": 80,
      "route_id": "route-a",
      "serving_region": "eu",
      "workload_class_id": "support"
    },

Truncated for display — the full payload is 81 lines.

What you get back

This is the actual output of running the example above — computed by the same function the platform calls, not an illustration.

Example output

{
  "assumptions": [
    "Resolved successes and failures are prospective, exchangeable within the chosen recency epoch and measured identically across routes. Priors, quality floors, value, cost, latency and the deployment horizon are owned locally and frozen before optimization.",
    "Each workload explores at most one challenger in this decision epoch. The knowledge gradient values only the option to choose between that challenger and its declared baseline after observing the exploration batch; it is not perfect information and does not value unsubmitted routes.",
    "Immediate regret is simulated against the current all-baseline policy with one common provider-availability scenario across workloads. Endpoint capacity aggregates scheduled baseline and exploration traffic before selection; high-impact exploration cannot manufacture provider diversity with aliases.",
    "Posterior safety is model-conditional. A selected allocation is not deployment, procurement, privacy, security, data-transfer or legal approval, a vendor SLA, or a judgment about any provider, team or person. Runtime kill switches and accountable approval remain external hard controls."
  ],
  "baseline_current_state": {
    "endpoint_capacity_feasible": true,
    "feasible": true,
    "provider_concentration_fraction": 1
  },
  "candidate_diagnostics": [
    {
      "candidate_id": "support-a",
      "endpoint_id": "endpoint-a",
      "failed_gates": [],
      "is_baseline": true,
      "posterior_mean_quality": 0.797,
      "posterior_probability_harm_vs_baseline": 0,
      "posterior_probability_quality_floor": 1,
      "provider_id": "provider-a",
      "route_id": "route-a",
      "safe_for_exploration": true,
      "workload_class_id": "support"
    },
    {
      "candidate_id": "support-b",
      "endpoint_id": "endpoint-b",
      "failed_gates": [],
      "is_baseline": false,
      "posterior_mean_quality": 0.881,
      "posterior_probability_harm_vs_baseline": 0.21,
      "posterior_probability_quality_floor": 0.97,
      "provider_id": "provider-b",
      "route_id": "route-b",
      "safe_for_exploration": true,
      "workload_class_id": "support"
    }
  ],
  "constraints": {
    "allocation_step_fraction": 0.1,
    "exploration_request_budget": 20,
    "harm_tolerance_per_request": 0,

Truncated for display — the full payload is 128 lines.

How it works

Sequential Bayesian & bandits — Learn while deciding — update beliefs as evidence arrives and choose where the next unit of effort is worth spending.

1 Fit tenant-local Beta posteriors from resolved route successes/failures and reject challengers lacking outcome support, posterior quality, harm protection, residency, retention, latency or high-impact provider independence.
2 Generate bounded traffic allocations, simulate immediate regret against the all-baseline policy under one common provider-availability scenario, and estimate posterior-predictive knowledge-gradient value over the declared future deployment horizon—including observations lost to outages.
3 Aggregate scheduled baseline and challenger traffic into shared endpoint capacity and value-weighted provider concentration, then select the exact or explicitly heuristic multiple-choice portfolio under request, expected-regret and regret-CVaR limits.

Before you trust it

Every tool in the catalog ships with the conditions under which its answer is meaningful — and the conditions under which it should abstain instead of guessing.

Assumptions & guardrails

The likelihood or reward model, prior support, action logging, delayed outcomes, and any stationarity assumptions match the deployment process.
Resolved outcomes are prospective and exchangeable inside the recency epoch; success definitions align across routes; priors, value, cost, horizon, traffic and provider scenarios are frozen locally; runtime approval and kill switches remain external.
Posterior probability and adaptive selection are model-conditional; they are not substitutes for randomized propensities or guaranteed safety.
Knowledge value covers only submitted baseline/challenger choices and is model-conditional; exactness covers only the finite grid. A selected allocation is never deployment, procurement, privacy/security/legal or data-transfer approval.

Minimum evidence

workload_classes: required and organization-defined
route_candidates: required and organization-defined
provider_scenarios: required and organization-defined
exploration_request_budget: required and organization-defined

How to validate it

Backtest the chosen action against simple feasible baselines on held-out scenarios, sweep costs/constraints/risk tolerance, and require constraint feasibility under adverse inputs.

Calibrating it to your org

Same for everyone

The mathematical kernel, validation rules, method version, and JSON output semantics are organization-independent; no tenant-trained coefficients or company benchmark is embedded in the function.

Specific to you

one point-in-time route-candidate projection joining only mature prospective outcomes to the deployed endpoint and provider version, with shared endpoint capacity and common provider scenarios retained across workloads
success and harm definitions, outcome maturity/exchangeability epoch, Beta priors and support, high-impact provider independence, residency/retention, endpoint capacity, learning horizon and value, exploration traffic, provider concentration, expected/CVaR regret, runtime approval and kill switches

Calibration workflow

1 Define the management decision, target outcome, aggregate unit, privacy boundary, cadence, and prediction/intervention horizon for this organization.
2 Build a tenant-scoped historical cohort using only information available before each prediction or decision; preserve zero periods, censoring, assignment probabilities, and unresolved outcomes when the method requires them.
3 Estimate statistical parameters on training history, but obtain costs, utilities, risk tolerance, practical-effect thresholds, capacity, and policy constraints from accountable decision owners.
4 Validate on later time windows or held-out aggregate units at the deployment grain, against a simple baseline and the function-specific validation strategy.
5 Deploy only if the returned decision clears evidence, overlap, calibration, robustness, and guardrail checks; warning, unsupported, schema-gap, and fallback decisions are abstentions.
6 Monitor realized outcomes, data drift, coverage, and decision regret; recalibrate at a governed cadence or after a detected regime/definition change, never merely because a stakeholder dislikes the result.

Call it from your AI

You don't wire up 388 tools in your MCP client. The GitRevio MCP server exposes 18 tools, three of which let an agent search the catalog, read a tool's schema, and run it — so the assistant finds this one on its own.

gitrevio_capabilities_search
  { "q": "allocate bounded production traffic to one" }
  → finds "optimize_safe_ai_routing_exploration_portfolio"

gitrevio_capability_describe
  { "capability_id": "optimize_safe_ai_routing_exploration_portfolio" }
  → returns the input schema and agent guidance shown on this page

gitrevio_capability_run
  { "capability_id": "optimize_safe_ai_routing_exploration_portfolio", "arguments": { ... } }
  → returns the result shown above

Works in Claude Desktop, Claude Code, Cursor, Cline, Continue.dev, Goose and Aider. See the MCP server.

Related tools

Audit AI routing experiment integrity

Audit prospectively registered AI-route experiments at the randomization-unit/period/route grain, reconciling logged propensities, allocation fidelity, pre-assignment balance, crossover, outcome maturity, simultaneous-experiment overlap and unique value at risk before any causal effect is reported.

Causal inference & experiment design

Forecast AI route quality cost drift

Forecast route-level quality, inference cost, p95 latency, breach timing, net value and economic-loss VaR/CVaR with partially pooled Bayesian trends and one common disruption state shared across every route on the same provider.

Sequential Bayesian & bandits

Audit AI capability fallback integrity

Prove that every aggregate capability required when AI is unavailable has a current approved runbook and a sufficiently large, timely, successful, independently observed exercise conducted with AI actually disabled.

Statistical audit & measurement

Audit AI code change evidence integrity

Prove that aggregate AI-assisted coding evidence comes from prospectively registered, nonoverlapping treatment/control studies with immutable assignment, configuration, trace and mature-outcome denominators before anyone estimates an effect.