Rank experiments by expected information gain
Rank prospective experiments by Bayesian mutual information and decision-aware expected value of sample information across explicit hypotheses, result likelihoods and decision payoffs; price usability, monetary cost and decision delay, expose recommendation-change probability, and preserve a value-information-cost-delay Pareto set.
What it's for
Tells leaders which unanswered question is worth testing next—not merely which experiment is cheapest or most statistically interesting.
What you give it
Inputs split into evidence read from your connected systems, calibration your team owns, and numerical controls that affect precision but never the result's meaning.
| Field | Type | Role | Required |
|---|---|---|---|
| decisions | array of objects (2 fields) ≥ 1 item | Evidence | Yes |
| discount_rate_per_period | number ≥ -0.99, ≤ 10 | Your calibration | Optional |
| experiments | array of objects (6 fields) ≥ 1 item | Evidence | Yes |
| hypotheses | array of objects (2 fields) ≥ 2 items | Evidence | Yes |
| max_detail_rows | integer ≥ 1, ≤ 500 | Numerical control | Optional |
| minimum_information_gain_bits | number ≥ 0 | Your calibration | Optional |
| minimum_net_expected_decision_value | number | Your calibration | Optional |
| ranking_metric | one of "expected_information_gain_bits", "information_gain_per_cost", "net_expected_decision_value", "expected_decision_value_per_cost" | Your calibration | Optional |
Each experiments
record
| Field | Type | Required |
|---|---|---|
| decision_delay_periods | integer (≥ 0, ≤ 240) | Yes |
| execution_capacity_units | number (≥ 0) | Yes |
| id | string (non-empty) | Yes |
| monetary_cost | number (≥ 0) | Yes |
| outcomes | array of objects (2 fields) (≥ 2 items) | Yes |
| result_usability_probability | number (≥ 0, ≤ 1) | Yes |
{
"decisions": [
{
"id": "improve-onboarding",
"payoff_by_hypothesis": [
100,
20
]
},
{
"id": "improve-value",
"payoff_by_hypothesis": [
20,
100
]
}
],
"discount_rate_per_period": 0.02,
"experiments": [
{
"decision_delay_periods": 1,
"execution_capacity_units": 1,
"id": "onboarding-test",
"monetary_cost": 10,
"outcomes": [
{
"id": "adoption-signal",
"likelihood_by_hypothesis": [
0.8,
0.2
]
},
{
"id": "value-signal",
"likelihood_by_hypothesis": [
0.2,
0.8
]
}
],
"result_usability_probability": 0.9
},
{
"decision_delay_periods": 0,Truncated for display — the full payload is 77 lines.
What you get back
This is the actual output of running the example above — computed by the same function the platform calls, not an illustration.
{
"decision": "ranked_experiments_clear_governed_information_value_gates",
"guardrails": [
"Mutual information rewards uncertainty reduction, while EVSI rewards information that can change a real decision; inspect both because an experiment can be scientifically informative but managerially irrelevant.",
"Outcome likelihoods, priors, payoff tables, usability, cost and delay must be calibrated on this organization's point-in-time evidence; posterior calculations cannot repair omitted hypotheses or fabricated likelihoods.",
"The ranking treats experiments independently and does not account for correlated results, shared setup cost, sequencing or portfolio learning; use a portfolio or adaptive-design method when selecting several tests.",
"A high rank is not authority to experiment on people, customers or production systems; ethics, consent, privacy, safety, legal review and stopping rules remain external governed constraints."
],
"method": "bayesian_mutual_information_decision_evsi_ranking_v1",
"ranking": [
{
"decision_delay_periods": 1,
"discounted_expected_value_of_sample_information": 21.1765,
"execution_capacity_units": 1,
"expected_decision_value_per_cost": 2.1176,
"expected_information_gain_bits": 0.2503,
"expected_posterior_entropy_bits": 0.7497,
"expected_value_of_sample_information": 21.6,
"experiment_id": "onboarding-test",
"failed_gates": [],
"information_gain_per_cost": 0.025,
"monetary_cost": 10,
"net_expected_decision_value": 11.1765,
"on_information_value_cost_delay_pareto_frontier": true,
"outcome_diagnostics": [
{
"best_decision_id": "improve-onboarding",
"best_expected_payoff": 84,
"outcome_id": "adoption-signal",
"posterior_by_hypothesis": {
"adoption-limited": 0.8,
"value-limited": 0.2
},
"posterior_entropy_bits": 0.7219,
"prior_predictive_probability": 0.5
},
{
"best_decision_id": "improve-value",
"best_expected_payoff": 84,
"outcome_id": "value-signal",
"posterior_by_hypothesis": {
"adoption-limited": 0.2,
"value-limited": 0.8
},Truncated for display — the full payload is 113 lines.
How it works
Sequential Bayesian & bandits — Learn while deciding — update beliefs as evidence arrives and choose where the next unit of effort is worth spending.
- 1 Freeze mutually interpretable hypotheses and priors, feasible decisions and payoff by hypothesis, then define each experiment's exhaustive result likelihoods, usability, cost, delay and execution capacity.
- 2 Apply Bayes' rule for every possible result, calculate expected posterior entropy, mutual information, posterior-optimal decision value and probability that the current recommendation changes.
- 3 Discount usable EVSI for result delay, subtract experiment cost, rank by the selected governed metric, apply information/value gates and expose nondominated experiments across information, value, cost and delay.
Before you trust it
Every tool in the catalog ships with the conditions under which its answer is meaningful — and the conditions under which it should abstain instead of guessing.
Assumptions & guardrails
- The likelihood or reward model, prior support, action logging, delayed outcomes, and any stationarity assumptions match the deployment process.
- Hypotheses cover the material uncertainty, priors and likelihoods are point-in-time and locally calibrated, result states are exhaustive, payoffs are commensurable, and the experiment can actually inform the stated decision before value expires.
- Posterior probability and adaptive selection are model-conditional; they are not substitutes for randomized propensities or guaranteed safety.
- Entropy reduction and decision value are distinct; the rank is single-experiment and omits correlated or sequential portfolio learning; no score authorizes experimentation on people, customers or production without consent, safety and stopping governance.
Minimum evidence
- hypotheses: at least 2 rows/items
- decisions: at least 1 rows/items
- experiments: at least 1 rows/items
How to validate it
Backtest the chosen action against simple feasible baselines on held-out scenarios, sweep costs/constraints/risk tolerance, and require constraint feasibility under adverse inputs.
Calibrating it to your org
Same for everyone
The mathematical kernel, validation rules, method version, and JSON output semantics are organization-independent; no tenant-trained coefficients or company benchmark is embedded in the function.
Specific to you
- point-in-time experiment-design case combining calibrated priors and likelihoods, decision payoff model, feasibility, expected result usability and resolved historical calibration without outcome leakage
- decision and hypothesis perimeter, prior/likelihood evidence standard, result exhaustiveness, payoff/currency/horizon, experiment cost/capacity/delay/usability, discounting, ranking metric, information/value gates, ethics/privacy/safety/consent and stopping authority
Calibration workflow
- 1 Define the management decision, target outcome, aggregate unit, privacy boundary, cadence, and prediction/intervention horizon for this organization.
- 2 Build a tenant-scoped historical cohort using only information available before each prediction or decision; preserve zero periods, censoring, assignment probabilities, and unresolved outcomes when the method requires them.
- 3 Estimate statistical parameters on training history, but obtain costs, utilities, risk tolerance, practical-effect thresholds, capacity, and policy constraints from accountable decision owners.
- 4 Validate on later time windows or held-out aggregate units at the deployment grain, against a simple baseline and the function-specific validation strategy.
- 5 Deploy only if the returned decision clears evidence, overlap, calibration, robustness, and guardrail checks; warning, unsupported, schema-gap, and fallback decisions are abstentions.
- 6 Monitor realized outcomes, data drift, coverage, and decision regret; recalibrate at a governed cadence or after a detected regime/definition change, never merely because a stakeholder dislikes the result.
Call it from your AI
You don't wire up 388 tools in your MCP client. The GitRevio MCP server exposes 18 tools, three of which let an agent search the catalog, read a tool's schema, and run it — so the assistant finds this one on its own.
gitrevio_capabilities_search
{ "q": "rank prospective experiments by bayesian mutual" }
→ finds "rank_experiments_by_expected_information_gain"
gitrevio_capability_describe
{ "capability_id": "rank_experiments_by_expected_information_gain" }
→ returns the input schema and agent guidance shown on this page
gitrevio_capability_run
{ "capability_id": "rank_experiments_by_expected_information_gain", "arguments": { ... } }
→ returns the result shown aboveWorks in Claude Desktop, Claude Code, Cursor, Cline, Continue.dev, Goose and Aider. See the MCP server.
Related tools
Calculate break even delivery date
Find the latest economically supported delivery period across coherent value, remaining-cost, recurring-contribution, operating-cost, value-decay, cost-growth and delay-cost scenarios; enforce expected NPV, positive-NPV probability and CVaR gates while keeping the economic deadline distinct from a completion forecast.
Calculate incremental cost effectiveness ratio
Construct a probabilistic incremental cost-effectiveness frontier from jointly aligned cost and outcome scenarios; remove strict and extended dominance before calculating ICERs, and select by expected net benefit plus a cost-effectiveness acceptability curve at organization-owned willingness-to-pay thresholds.
Estimate productivity rebound effect
Estimate how much aggregate capacity released by a productivity intervention is absorbed by induced output or workload using stacked matched-cohort log difference-in-differences; separate fixed-output efficiency, induced output and total resource use, audit pretrends, reconcile the log identity, and cluster-bootstrap rebound uncertainty including backfire above 100 percent.
Audit commercial technical commitment integrity
Audit signed commercial promises against explicitly allocated technical scope, dependency order, funded capacity, acceptance criteria and evidence; expose orphan scope, double allocation, cycles, late plans and maximum contractual penalty without interpreting legal rights from engineering activity.
Audit decision execution fidelity
Audit whether approved decisions actually became verified implementation at the promised aggregate-unit and component grain, with whole-unit bootstrap uncertainty and simultaneous gates for fidelity, overdue scope, unverifiable evidence, exceptions and critical gaps.
Audit KPI threshold bunching
Detect a post-target excess concentration immediately above a governed KPI threshold: compare within-unit pre/post local mass and above-versus-below mirror asymmetry, bootstrap whole units, report density bins and a smoothed log-density jump, and explicitly refuse to equate bunching with individual gaming or intent.