Forecast AI inference avoidable cost

Forecast AI inference spend and the safely avoidable portion from semantic response caching, retry prevention and batching using tenant-local empirical-Bayes rates, log-normal unit demand, shared scenarios, Shapley savings attribution and cost VaR/CVaR.

Forecasting & survival Forecast CTOs Tech CEOs IT managers Team leads Investors

What it's for

Splits AI spend into the part that buys something and the part that caching, retry prevention and batching could remove — with the savings attributed per driver.

Gives CTOs and investors a defensible answer to: how much will our AI bill become, how much is avoidable, which lever creates it, and what breaks under stress?

What you give it

Inputs split into evidence read from your connected systems, calibration your team owns, and numerical controls that affect precision but never the result's meaning.

Field	Type	Role	Required
current_workloads	array of objects (18 fields)	Evidence	Yes
historical_workload_periods	array of objects (14 fields)	Evidence	Yes
horizon_periods	integer ≥ 1, ≤ 120	Your calibration	Optional
max_detail_rows	integer ≥ 1, ≤ 500	Numerical control	Optional
minimum_active_periods	integer ≥ 3	Your calibration	Optional
minimum_historical_periods	integer ≥ 3	Your calibration	Optional
prior_strength	number > 0	Your calibration	Optional
scenarios	array of objects (11 fields)	Evidence	Yes
seed	integer ≥ 0	Numerical control	Optional
simulations	integer ≥ 100, ≤ 100000	Numerical control	Optional
tail_probability	number ≥ 0.001, ≤ 0.5	Your calibration	Optional

Each `current_workloads` record

Field	Type	Required
batch_discount_fraction	number (≥ 0, ≤ 1)	Yes
batch_latency_multiplier	number (≥ 1)	Yes
current_p95_latency_ms	number (> 0)	Yes
current_quality_score	number (≥ 0, ≤ 1)	Yes
evidence_verified	boolean	Yes
fixed_cost_per_period	number (≥ 0)	Yes
forecast_logical_request_count_per_period	number (≥ 0)	Yes
id	string (non-empty)	Yes
input_cost_per_million_units	number (≥ 0)	Yes
maximum_batch_fraction	number (≥ 0, ≤ 1)	Yes
maximum_p95_latency_ms	number (> 0)	Yes
maximum_safe_response_cache_fraction	number (≥ 0, ≤ 1)	Yes
maximum_safe_retry_reduction_fraction	number (≥ 0, ≤ 1)	Yes
minimum_quality_score	number (≥ 0, ≤ 1)	Yes
output_cost_per_million_units	number (≥ 0)	Yes
request_cost	number (≥ 0)	Yes
response_cache_lookup_cost	number (≥ 0)	Yes
workload_class	string (non-empty)	Yes

Example input

{
  "current_workloads": [
    {
      "batch_discount_fraction": 0.2,
      "batch_latency_multiplier": 1.2,
      "current_p95_latency_ms": 100,
      "current_quality_score": 0.9,
      "evidence_verified": true,
      "fixed_cost_per_period": 1,
      "forecast_logical_request_count_per_period": 1000,
      "id": "support-current",
      "input_cost_per_million_units": 2,
      "maximum_batch_fraction": 0.5,
      "maximum_p95_latency_ms": 200,
      "maximum_safe_response_cache_fraction": 0.4,
      "maximum_safe_retry_reduction_fraction": 1,
      "minimum_quality_score": 0.8,
      "output_cost_per_million_units": 4,
      "request_cost": 0.01,
      "response_cache_lookup_cost": 0.001,
      "workload_class": "support"
    }
  ],
  "historical_workload_periods": [
    {
      "avoidable_retry_count": 10,
      "billed_cost": 5,
      "evidence_verified": true,
      "id": "support-00",
      "inference_attempt_count": 90,
      "input_units": 90000,
      "logical_request_count": 100,
      "output_units": 18000,
      "p95_latency_ms": 100,
      "period": 0,
      "quality_score": 0.9,
      "response_cache_eligible_count": 40,
      "response_cache_hit_count": 20,
      "workload_class": "support"
    },
    {
      "avoidable_retry_count": 10,
      "billed_cost": 5,
      "evidence_verified": true,

Truncated for display — the full payload is 248 lines.

What you get back

This is the actual output of running the example above — computed by the same function the platform calls, not an illustration.

Example output

{
  "assumptions": [
    "History is consecutive and zero-inclusive with stable workload, cache-eligibility, retry, tokenizer, quality, latency and cost definitions; avoidable retries are independently governed rather than inferred from all retries.",
    "Maximum safe cache, retry-reduction and batch fractions are prospective operating constraints, not effects inferred from repository activity. Cache invalidation removes savings; batching is used only in simulations that clear the latency limit.",
    "Exact Shapley allocation fairly reconciles interaction savings across the three submitted levers; it does not make those savings causal or include implementation cost.",
    "This forecast is an avoidable-cost envelope, not a deployment instruction, quality/safety guarantee, vendor SLA, procurement decision, privacy authorization or judgment about a provider, team or person."
  ],
  "configuration": {
    "attribution_rule": "exact_three_lever_shapley_over_response_cache_retry_prevention_and_batching",
    "dependence_rule": "one_scenario_draw_shared_across_all_workloads_per_simulation",
    "horizon_periods": 6,
    "minimum_active_periods": 8,
    "minimum_historical_periods": 12,
    "prior_strength": 2,
    "seed": 37,
    "simulations": 200,
    "tail_probability": 0.1
  },
  "decision": "ai_inference_avoidable_cost_forecast_supported",
  "failed_gates": [],
  "method": "empirical_bayes_ai_inference_avoidable_cost_shapley_forecast_v1",
  "scenario_diagnostics": [
    {
      "probability": 0.8,
      "scenario_id": "base",
      "simulation_frequency": 0.845
    },
    {
      "probability": 0.2,
      "scenario_id": "stress",
      "simulation_frequency": 0.155
    }
  ],
  "summary": {
    "baseline_cost_conditional_value_at_risk": 167.4788,
    "baseline_cost_value_at_risk": 165.5537,
    "expected_avoidable_cost": 26.7322,
    "expected_baseline_cost": 89.1169,
    "expected_max_safe_lever_cost": 62.3847,
    "probability_of_positive_avoidable_cost": 1,
    "shapley_reconciliation_error": 0,
    "shapley_savings_by_lever": {
      "batching": 4.8099,
      "response_cache": 13.5475,

Truncated for display — the full payload is 71 lines.

How it works

Forecasting & survival — Estimate when something completes or fails, with censoring and unresolved work handled honestly rather than dropped.

1 Retain consecutive workload periods including zeros, then learn tenant-local cache eligibility/adoption and avoidable retry rates with partial pooling plus log-normal input/output units per attempt.
2 Draw one coherent demand, repeatability, retry, unit, price, invalidation, quality and latency scenario for the whole portfolio and simulate baseline versus maximum independently approved safe levers.
3 Attribute interactions exactly with three-lever Shapley values; expose latency/quality breaches, cost VaR/CVaR and history/evidence abstention rather than presenting savings as guaranteed.

Before you trust it

Every tool in the catalog ships with the conditions under which its answer is meaningful — and the conditions under which it should abstain instead of guessing.

Assumptions & guardrails

Training examples precede their outcomes, censoring and unresolved work are represented, and deployment populations remain comparable to validation cohorts.
Gateway counts, cache eligibility/invalidation, avoidable retries, token units, quality, latency and prices are stable enough within workload and use a common horizon, currency and scenario perimeter.
A predictive interval or risk estimate is not a deadline promise, causal explanation, or individual-performance judgment.
The result is a conditional safe-lever envelope, not a savings commitment, deployment authorization, service SLA or permission to cache sensitive responses.

Minimum evidence

historical_workload_periods: required and organization-defined
current_workloads: required and organization-defined
scenarios: required and organization-defined

How to validate it

Use chronological train/calibration/test windows, compare proper scores and decision value with a simple baseline, and recalibrate only from outcomes resolved after prediction time.

Calibrating it to your org

Same for everyone

The mathematical kernel, validation rules, method version, and JSON output semantics are organization-independent; no tenant-trained coefficients or company benchmark is embedded in the function.

Specific to you

mature point-in-time workload-period projection plus response-cache eligibility/invalidation and avoidable-retry labels, joined to effective pricing and prospectively validated quality/latency gates
workload stability, observation window and zero retention, cache eligibility/invalidation, retry avoidability, tokenizer, quality and latency semantics, safe lever caps, horizon, prices/currency, scenarios, priors, support and tail appetite

Calibration workflow

1 Define the management decision, target outcome, aggregate unit, privacy boundary, cadence, and prediction/intervention horizon for this organization.
2 Build a tenant-scoped historical cohort using only information available before each prediction or decision; preserve zero periods, censoring, assignment probabilities, and unresolved outcomes when the method requires them.
3 Estimate statistical parameters on training history, but obtain costs, utilities, risk tolerance, practical-effect thresholds, capacity, and policy constraints from accountable decision owners.
4 Validate on later time windows or held-out aggregate units at the deployment grain, against a simple baseline and the function-specific validation strategy.
5 Deploy only if the returned decision clears evidence, overlap, calibration, robustness, and guardrail checks; warning, unsupported, schema-gap, and fallback decisions are abstentions.
6 Monitor realized outcomes, data drift, coverage, and decision regret; recalibrate at a governed cadence or after a detected regime/definition change, never merely because a stakeholder dislikes the result.

Call it from your AI

You don't wire up 388 tools in your MCP client. The GitRevio MCP server exposes 18 tools, three of which let an agent search the catalog, read a tool's schema, and run it — so the assistant finds this one on its own.

gitrevio_capabilities_search
  { "q": "forecast ai inference spend and the" }
  → finds "forecast_ai_inference_avoidable_cost"

gitrevio_capability_describe
  { "capability_id": "forecast_ai_inference_avoidable_cost" }
  → returns the input schema and agent guidance shown on this page

gitrevio_capability_run
  { "capability_id": "forecast_ai_inference_avoidable_cost", "arguments": { ... } }
  → returns the result shown above

Works in Claude Desktop, Claude Code, Cursor, Cline, Continue.dev, Goose and Aider. See the MCP server.

Related tools

Audit AI inference cost allocation integrity

Reconcile provider AI invoices bottom-up to workload and route usage, price terms, cached requests, retries, fixed charges and credits without combining currencies or silently allocating unexplained spend.

Statistical audit & measurement

Optimize AI inference efficiency portfolio

Choose one governed AI inference efficiency design per workload across semantic caching, retry prevention, batching and unit reduction, maximizing risk-adjusted economic value under quality, latency, scenario availability, shared capacity, dependency, budget and CVaR-regret constraints.

Constrained optimization

Audit AI capability fallback integrity

Prove that every aggregate capability required when AI is unavailable has a current approved runbook and a sufficiently large, timely, successful, independently observed exercise conducted with AI actually disabled.

Statistical audit & measurement

Audit AI code change evidence integrity

Prove that aggregate AI-assisted coding evidence comes from prospectively registered, nonoverlapping treatment/control studies with immutable assignment, configuration, trace and mature-outcome denominators before anyone estimates an effect.

Causal inference & experiment design

Audit AI knowledge grounding integrity

Audit the complete AI knowledge supply chain from immutable source versions through indexed chunks and effective access policy to retrieved evidence, claim-level citations and honestly mature grounding outcomes, without treating unresolved answers as failures.