Audit release risk prediction integrity
Audit a complete eligible-change release-risk cohort for point-in-time prediction lineage, exact change-to-deployment linkage, mature mutually exclusive outcomes, selective labels, score-triggered intervention contamination, calibration and false alarms before the score influences a release decision.
What it's for
Turns Gitrevio's marketed per-PR release-risk score into an auditable product claim: the score must exist before the decision, link exactly to production, survive mature calibration and disclose selective deployment rather than merely repeat fixed weights.
What you give it
Inputs split into evidence read from your connected systems, calibration your team owns, and numerical controls that affect precision but never the result's meaning.
| Field | Type | Role | Required |
|---|---|---|---|
| as_of_ms | number ≥ 0 | Your calibration | Yes |
| eligible_changes | array of objects (7 fields) | Evidence | Yes |
| high_risk_threshold | number ≥ 0, ≤ 1 | Your calibration | Optional |
| max_detail_rows | integer ≥ 1, ≤ 500 | Numerical control | Optional |
| maximum_expected_calibration_error | number ≥ 0, ≤ 1 | Your calibration | Optional |
| maximum_false_positive_rate | number ≥ 0, ≤ 1 | Your calibration | Optional |
| maximum_intervention_contamination_fraction | number ≥ 0, ≤ 1 | Your calibration | Optional |
| maximum_missing_deployed_outcome_fraction | number ≥ 0, ≤ 1 | Your calibration | Optional |
| maximum_missing_prediction_fraction | number ≥ 0, ≤ 1 | Your calibration | Optional |
| minimum_mature_deployments | integer ≥ 1 | Your calibration | Optional |
| release_outcomes | array of objects (12 fields) ≥ 0 items | Evidence | Yes |
| risk_predictions | array of objects (11 fields) ≥ 0 items | Evidence | Yes |
Each release_outcomes
record
| Field | Type | Required |
|---|---|---|
| ambiguous_link_count | integer (≥ 0) | Yes |
| change_id | string (non-empty) | Yes |
| commit_count | integer (≥ 1) | Yes |
| deployed_at_ms | number (≥ 0) | Yes |
| deployment_link_method_id | string (non-empty) | Yes |
| evidence_verified | boolean | Yes |
| id | string (non-empty) | Yes |
| intervention_contaminated | boolean | Yes |
| label_mature_at_ms | number (≥ 0) | Yes |
| linked_commit_count | integer (≥ 0) | Yes |
| outcome_at_ms | number,null (≥ 0) | Yes |
| outcome_type_id | one of "clean", "rollback", "hotfix", "incident" | Yes |
{
"as_of_ms": 10000,
"eligible_changes": [
{
"change_class_id": "service-change",
"decision_at_ms": 1050,
"deployed": true,
"eligible_at_ms": 1000,
"evidence_verified": true,
"id": "audit-change-0",
"repository_id": "payments"
},
{
"change_class_id": "service-change",
"decision_at_ms": 1150,
"deployed": true,
"eligible_at_ms": 1100,
"evidence_verified": true,
"id": "audit-change-1",
"repository_id": "payments"
},
{
"change_class_id": "service-change",
"decision_at_ms": 1250,
"deployed": true,
"eligible_at_ms": 1200,
"evidence_verified": true,
"id": "audit-change-2",
"repository_id": "payments"
},
{
"change_class_id": "service-change",
"decision_at_ms": 1350,
"deployed": false,
"eligible_at_ms": 1300,
"evidence_verified": true,
"id": "audit-change-3",
"repository_id": "payments"
}
],
"maximum_expected_calibration_error": 0.3,
"minimum_mature_deployments": 3,
"release_outcomes": [
{Truncated for display — the full payload is 141 lines.
What you get back
This is the actual output of running the example above — computed by the same function the platform calls, not an illustration.
{
"assumptions": [
"The eligible-change perimeter includes deployed and undeployed candidates, and predictions, features, decisions, deployments and labels are immutable point-in-time records.",
"Rollback, hotfix and incident labels are mutually exclusive first outcomes under one governed maturity window; deployment linkage is exact rather than inferred from temporal proximity."
],
"change_diagnostics": [
{
"change_class_id": "service-change",
"change_id": "audit-change-3",
"deployed": false,
"mature_outcome": false,
"outcome_count": 0,
"outcome_type_id": null,
"prediction_count": 1,
"repository_id": "payments",
"scoreable": false
},
{
"change_class_id": "service-change",
"change_id": "audit-change-0",
"deployed": true,
"mature_outcome": true,
"outcome_count": 1,
"outcome_type_id": "rollback",
"prediction_count": 1,
"repository_id": "payments",
"scoreable": true
},
{
"change_class_id": "service-change",
"change_id": "audit-change-1",
"deployed": true,
"mature_outcome": true,
"outcome_count": 1,
"outcome_type_id": "clean",
"prediction_count": 1,
"repository_id": "payments",
"scoreable": true
},
{
"change_class_id": "service-change",
"change_id": "audit-change-2",
"deployed": true,
"mature_outcome": true,Truncated for display — the full payload is 108 lines.
How it works
Forecasting & survival — Estimate when something completes or fails, with censoring and unresolved work handled honestly rather than dropped.
- 1 Freeze the complete eligible change perimeter—including undeployed candidates—and reconcile exactly one immutable model/version/feature snapshot before each release decision.
- 2 Require exact commit-to-deployment linkage and one mature first clean, rollback, hotfix or incident outcome per deployed change; retain selective labels and exclude outcomes changed by score-triggered assurance.
- 3 Reconstruct Brier score, log loss, calibration and false-positive rates by repository/change class while rejecting automatic blocking and person-identity or protected-proxy features.
Before you trust it
Every tool in the catalog ships with the conditions under which its answer is meaningful — and the conditions under which it should abstain instead of guessing.
Assumptions & guardrails
- Training examples precede their outcomes, censoring and unresolved work are represented, and deployment populations remain comparable to validation cohorts.
- Eligibility, model and feature timestamps, release decisions, PR-to-commit-to-deployment links, intervention history and outcome maturity are complete immutable tenant records.
- A predictive interval or risk estimate is not a deadline promise, causal explanation, or individual-performance judgment.
- Passing validates submitted lineage and observed deployed-change performance only; it neither identifies causal risk factors nor authorizes an automatic merge block or person-level judgment.
Minimum evidence
- eligible_changes: required and organization-defined
- risk_predictions: at least 0 rows/items
- release_outcomes: at least 0 rows/items
- as_of_ms: required and organization-defined
How to validate it
Use chronological train/calibration/test windows, compare proper scores and decision value with a simple baseline, and recalibrate only from outcomes resolved after prediction time.
Calibrating it to your org
Same for everyone
The mathematical kernel, validation rules, method version, and JSON output semantics are organization-independent; no tenant-trained coefficients or company benchmark is embedded in the function.
Specific to you
- one versioned release-risk audit mart retaining deployed and undeployed candidates, exact commit coverage and ambiguity, score-triggered intervention contamination, mutually exclusive first outcome and maturity timestamp
- eligible change definition, decision cutoff, outcome taxonomy and maturity window, approved linkage methods, intervention-contamination rule, high-risk threshold, calibration/false-positive gates and human release authority
Calibration workflow
- 1 Define the management decision, target outcome, aggregate unit, privacy boundary, cadence, and prediction/intervention horizon for this organization.
- 2 Build a tenant-scoped historical cohort using only information available before each prediction or decision; preserve zero periods, censoring, assignment probabilities, and unresolved outcomes when the method requires them.
- 3 Estimate statistical parameters on training history, but obtain costs, utilities, risk tolerance, practical-effect thresholds, capacity, and policy constraints from accountable decision owners.
- 4 Validate on later time windows or held-out aggregate units at the deployment grain, against a simple baseline and the function-specific validation strategy.
- 5 Deploy only if the returned decision clears evidence, overlap, calibration, robustness, and guardrail checks; warning, unsupported, schema-gap, and fallback decisions are abstentions.
- 6 Monitor realized outcomes, data drift, coverage, and decision regret; recalibrate at a governed cadence or after a detected regime/definition change, never merely because a stakeholder dislikes the result.
Call it from your AI
You don't wire up 388 tools in your MCP client. The GitRevio MCP server exposes 18 tools, three of which let an agent search the catalog, read a tool's schema, and run it — so the assistant finds this one on its own.
gitrevio_capabilities_search
{ "q": "audit a complete eligiblechange releaserisk cohort" }
→ finds "audit_release_risk_prediction_integrity"
gitrevio_capability_describe
{ "capability_id": "audit_release_risk_prediction_integrity" }
→ returns the input schema and agent guidance shown on this page
gitrevio_capability_run
{ "capability_id": "audit_release_risk_prediction_integrity", "arguments": { ... } }
→ returns the result shown aboveWorks in Claude Desktop, Claude Code, Cursor, Cline, Continue.dev, Goose and Aider. See the MCP server.
Related tools
Forecast governed release competing risks
Forecast company-local rollback, hotfix and incident incidence conditional on deployment with inverse-propensity-corrected discrete-time competing risks, strict whole-release chronological validation, posterior intervals and mandatory improvement over both a simple baseline and the legacy PR score.
Optimize causal release assurance portfolio
Select one release, assurance or hold option per change from prospectively identified Beta-binomial relative-risk effects while pricing delay, failure and shared common-mode loss under budget, scarce resources, mandatory controls, expected-failure and CVaR constraints with an exact or disclosed beam-search Pareto frontier.
Audit commercial technical commitment integrity
Audit signed commercial promises against explicitly allocated technical scope, dependency order, funded capacity, acceptance criteria and evidence; expose orphan scope, double allocation, cycles, late plans and maximum contractual penalty without interpreting legal rights from engineering activity.
Audit decision execution fidelity
Audit whether approved decisions actually became verified implementation at the promised aggregate-unit and component grain, with whole-unit bootstrap uncertainty and simultaneous gates for fidelity, overdue scope, unverifiable evidence, exceptions and critical gaps.
Audit KPI threshold bunching
Detect a post-target excess concentration immediately above a governed KPI threshold: compare within-unit pre/post local mass and above-versus-below mirror asymmetry, bootstrap whole units, report density bins and a smoothed log-density jump, and explicitly refuse to equate bunching with individual gaming or intent.
Audit sunk cost escalation
Audit whether cumulative sunk cost predicts aggregate project continuation after project fixed effects, checkpoint time, forward value, success probability, remaining cost, and future irreversibility, with project-cluster bootstrap uncertainty.