Audit analytics transportability
Audit whether locally calibrated analytical functions retain decision-loss improvement across target-similar operating environments using similarity-weighted random-effects meta-analysis, between-environment variance, I-squared, sign consistency and a conservative target prediction interval.
What it's for
Shows executives which analytics generalize across the company's actual teams, repositories, products or operating regimes—and which require local recalibration before rollout.
What you give it
Inputs split into evidence read from your connected systems, calibration your team owns, and numerical controls that affect precision but never the result's meaning.
| Field | Type | Role | Required |
|---|---|---|---|
| confidence_level | number ≥ 0.5, < 1 | Your calibration | Optional |
| environment_summaries | array of objects (7 fields) ≥ 2 items | Evidence | Yes |
| max_detail_rows | integer ≥ 1, ≤ 500 | Numerical control | Optional |
| maximum_i_squared | number ≥ 0, ≤ 1 | Your calibration | Optional |
| minimum_environment_count | integer ≥ 2, ≤ 10000 | Your calibration | Optional |
| minimum_loss_improvement | number | Your calibration | Optional |
| minimum_positive_environment_fraction | number ≥ 0, ≤ 1 | Your calibration | Optional |
| minimum_target_similarity_weight | number ≥ 0, ≤ 1 | Your calibration | Optional |
| minimum_total_resolved_cases | integer ≥ 2, ≤ 10000000 | Your calibration | Optional |
Each environment_summaries
record
| Field | Type | Required |
|---|---|---|
| environment_id | string (non-empty) | Yes |
| function_id | string (non-empty) | Yes |
| id | string (non-empty) | Yes |
| mean_loss_improvement | number | Yes |
| resolved_case_count | integer (≥ 2, ≤ 1000000) | Yes |
| standard_error | number (> 0) | Yes |
| target_similarity_weight | number (≥ 0, ≤ 1) | Yes |
{
"environment_summaries": [
{
"environment_id": "operating-environment-0",
"function_id": "mcmc_project_completion_forecast",
"id": "transport-0",
"mean_loss_improvement": 1.95,
"resolved_case_count": 30,
"standard_error": 0.1,
"target_similarity_weight": 1
},
{
"environment_id": "operating-environment-1",
"function_id": "mcmc_project_completion_forecast",
"id": "transport-1",
"mean_loss_improvement": 2,
"resolved_case_count": 30,
"standard_error": 0.1,
"target_similarity_weight": 1
},
{
"environment_id": "operating-environment-2",
"function_id": "mcmc_project_completion_forecast",
"id": "transport-2",
"mean_loss_improvement": 2.05,
"resolved_case_count": 30,
"standard_error": 0.1,
"target_similarity_weight": 1
},
{
"environment_id": "operating-environment-3",
"function_id": "mcmc_project_completion_forecast",
"id": "transport-3",
"mean_loss_improvement": 1.95,
"resolved_case_count": 30,
"standard_error": 0.1,
"target_similarity_weight": 1
},
{
"environment_id": "operating-environment-4",
"function_id": "mcmc_project_completion_forecast",
"id": "transport-4",
"mean_loss_improvement": 2,
"resolved_case_count": 30,Truncated for display — the full payload is 60 lines.
What you get back
This is the actual output of running the example above — computed by the same function the platform calls, not an illustration.
{
"configuration": {
"confidence_level": 0.9,
"maximum_i_squared": 0.75,
"minimum_environment_count": 4,
"minimum_loss_improvement": 0,
"minimum_positive_environment_fraction": 0.75,
"minimum_target_similarity_weight": 0.1,
"minimum_total_resolved_cases": 100
},
"decision": "transportable_functions_available",
"function_diagnostics": [
{
"between_environment_variance": 0,
"decision": "transportability_supported_for_represented_target",
"excluded_low_similarity_environment_count": 0,
"failed_gates": [],
"function_id": "mcmc_project_completion_forecast",
"i_squared": 0,
"mean_improvement_confidence_interval": [
1.9328,
2.0672
],
"positive_environment_fraction": 1,
"random_effects_mean_improvement": 2,
"represented_environment_count": 6,
"target_prediction_interval": [
1.9328,
2.0672
],
"total_resolved_cases": 180
}
],
"guardrails": [
"Transportability requires prospective loss summaries from comparable environments and a target-similarity model frozen before seeing target outcomes; similarity weights are governance inputs, not learned excuses for a favorable result.",
"Random-effects pooling exposes heterogeneity and a target prediction interval; it does not repair unmeasured effect modifiers, selection, dataset shift or incompatible outcome definitions.",
"A supported result applies only to the represented target envelope and function version. It never licenses cross-tenant coefficient copying or named-person decisions."
],
"method": "target_similarity_weighted_random_effects_transportability_v1",
"summary": {
"function_count": 1,
"transportable_function_count": 1,
"transportable_function_ids": [
"mcmc_project_completion_forecast"Truncated for display — the full payload is 50 lines.
How it works
Forecasting & survival — Estimate when something completes or fails, with censoring and unresolved work handled honestly rather than dropped.
- 1 Freeze function version, prospective loss definition, environment taxonomy, resolved-case summaries and target-similarity weights before inspecting the target outcome.
- 2 Pool environment effects with inverse-variance and target-similarity weights, estimate random-effects heterogeneity, and form both mean-effect and new-target prediction intervals.
- 3 Require represented environments, resolved support, target lower-bound improvement, controlled heterogeneity and cross-environment sign consistency simultaneously.
Before you trust it
Every tool in the catalog ships with the conditions under which its answer is meaningful — and the conditions under which it should abstain instead of guessing.
Assumptions & guardrails
- Training examples precede their outcomes, censoring and unresolved work are represented, and deployment populations remain comparable to validation cohorts.
- Environment summaries use the same proper decision loss and prospective evaluation design; standard errors are valid; effect modifiers are measured enough for the frozen target-similarity model; environments are independent at the analysis grain.
- A predictive interval or risk estimate is not a deadline promise, causal explanation, or individual-performance judgment.
- Support applies only inside the represented target envelope; random-effects pooling does not repair hidden modifiers or dataset shift and never permits cross-tenant coefficient copying, individual scoring or autonomous deployment.
Minimum evidence
- environment_summaries: at least 2 rows/items
How to validate it
Use chronological train/calibration/test windows, compare proper scores and decision value with a simple baseline, and recalibrate only from outcomes resolved after prediction time.
Calibrating it to your org
Same for everyone
The mathematical kernel, validation rules, method version, and JSON output semantics are organization-independent; no tenant-trained coefficients or company benchmark is embedded in the function.
Specific to you
- versioned environment-level summaries derived from the same point-in-time held-out evaluation ledger, function version, proper loss, baseline, eligibility epoch and independent analysis unit
- function/loss/baseline versions, operating-environment taxonomy, target envelope and similarity model, standard-error method, minimum environments/cases/improvement/sign consistency, heterogeneity/confidence limits and rollout authority
Calibration workflow
- 1 Define the management decision, target outcome, aggregate unit, privacy boundary, cadence, and prediction/intervention horizon for this organization.
- 2 Build a tenant-scoped historical cohort using only information available before each prediction or decision; preserve zero periods, censoring, assignment probabilities, and unresolved outcomes when the method requires them.
- 3 Estimate statistical parameters on training history, but obtain costs, utilities, risk tolerance, practical-effect thresholds, capacity, and policy constraints from accountable decision owners.
- 4 Validate on later time windows or held-out aggregate units at the deployment grain, against a simple baseline and the function-specific validation strategy.
- 5 Deploy only if the returned decision clears evidence, overlap, calibration, robustness, and guardrail checks; warning, unsupported, schema-gap, and fallback decisions are abstentions.
- 6 Monitor realized outcomes, data drift, coverage, and decision regret; recalibrate at a governed cadence or after a detected regime/definition change, never merely because a stakeholder dislikes the result.
Call it from your AI
You don't wire up 388 tools in your MCP client. The GitRevio MCP server exposes 18 tools, three of which let an agent search the catalog, read a tool's schema, and run it — so the assistant finds this one on its own.
gitrevio_capabilities_search
{ "q": "audit whether locally calibrated analytical functions" }
→ finds "audit_analytics_transportability"
gitrevio_capability_describe
{ "capability_id": "audit_analytics_transportability" }
→ returns the input schema and agent guidance shown on this page
gitrevio_capability_run
{ "capability_id": "audit_analytics_transportability", "arguments": { ... } }
→ returns the result shown aboveWorks in Claude Desktop, Claude Code, Cursor, Cline, Continue.dev, Goose and Aider. See the MCP server.
Related tools
Calculate analytics portfolio realized ROI
Reconcile the analytics portfolio's realized ROI from unique finance-owned incremental benefit sources, causal-evidence weights, non-overlapping function allocations, implementation/recurring/shared costs and coherent joint scenarios, with positive-value probability and CVaR loss gates.
Construct calibration shared evidence graph
Discover stable candidate relationships for shared calibration design by correlating standardized held-out loss improvement across common environments, resampling whole environments, stability-selecting practical edges, controlling sign discoveries with Benjamini-Hochberg FDR and returning connected components without claiming parameter transfer.
Audit analytical specification multiverse
Audit whether an analytical conclusion survives a prespecified multiverse of admissible windows, cohorts, metrics, and models using aligned bootstrap draws, a weighted specification curve, practical-effect support gates, and descriptive choice-influence diagnostics.
Audit analytics challenger independence
Audit whether an analytical challenger supplies genuinely independent error information: use paired temporal moving-block bootstrap bounds on error correlation, incumbent-failure catch rate and common-mode joint failure, with simultaneous Bonferroni control across every screened challenger and explicit evidence gates.
Audit analytics function calibration readiness
Gate analytical functions on paired out-of-time decision loss against a frozen baseline using temporal moving-block bootstrap, autocorrelation- and weight-adjusted effective sample size, evidence coverage, lower confidence bounds, improvement probability and recent degradation rather than declaring a model calibrated from training fit.
Audit forecast ensemble lineage integrity
Audit whether a claimed forecast ensemble is a complete, independently sealed and point-in-time evidence set rather than duplicated consensus, then score only mature uncontaminated outcomes.