Audit selective label partial identification

What it's for

Prevents one of the most dangerous AI analytics errors: Gitrevio shows what is and is not identifiable when the policy itself determines whose outcome becomes visible, instead of reporting falsely precise calibration from selected labels.

What you give it

Inputs split into evidence read from your connected systems, calibration your team owns, and numerical controls that affect precision but never the result's meaning.

{
  "decision_threshold": 0.5,
  "observations": [
    {
      "decision": 0,
      "id": "selective-label-0",
      "outcome": null,
      "predicted_probability": 0.2
    },
    {
      "decision": 0,
      "id": "selective-label-1",
      "outcome": 1,
      "predicted_probability": 0.2
    },
    {
      "decision": 0,
      "id": "selective-label-2",
      "outcome": 0,
      "predicted_probability": 0.2
    },
    {
      "decision": 0,
      "id": "selective-label-3",
      "outcome": 0,
      "predicted_probability": 0.2
    },
    {
      "decision": 0,
      "id": "selective-label-4",
      "outcome": 0,
      "predicted_probability": 0.2
    },
    {
      "decision": 0,
      "id": "selective-label-5",
      "outcome": null,
      "predicted_probability": 0.2
    },
    {
      "decision": 0,
      "id": "selective-label-6",
      "outcome": 0,
      "predicted_probability": 0.2

What you get back

This is the actual output of running the example above — computed by the same function the platform calls, not an illustration.

{
  "assumptions": [
    "Within each declared decision stratum, the odds of the event among missing labels differ from observed-label odds by no more than the governed sensitivity ratio.",
    "Observed outcomes are correctly resolved at one common horizon, population weights identify the intended eligible cohort, and missing outcomes are retained rather than coded as negatives.",
    "The supplied decision is the policy mechanism that controls label observation; threshold mismatches can indicate additional undocumented selection rules."
  ],
  "configuration": {
    "confidence_level": 0.9,
    "decision_threshold": 0.5,
    "maximum_decision_mismatch_rate": 0,
    "maximum_event_rate_bound_width": 0.2,
    "maximum_missing_fraction": 0.3,
    "posterior_draws": 500,
    "probability_bins": 10,
    "sensitivity_odds_ratio": 2
  },
  "decision": "selective_label_uncertainty_bounded",
  "decision_strata": [
    {
      "decision": 0,
      "event_rate_sensitivity_posterior_interval": [
        0.1056,
        0.1706
      ],
      "median_event_rate_bound_width": 0.0306,
      "missing_rows": 10,
      "observed_outcome_support_available": true,
      "observed_rows": 40,
      "rows": 50,
      "weighted_missing_fraction": 0.2
    },
    {
      "decision": 1,
      "event_rate_sensitivity_posterior_interval": [
        0.5668,
        0.67
      ],
      "median_event_rate_bound_width": 0.0632,
      "missing_rows": 10,
      "observed_outcome_support_available": true,
      "observed_rows": 40,
      "rows": 50,
      "weighted_missing_fraction": 0.2
    }

How it works

Sequential Bayesian & bandits — Learn while deciding — update beliefs as evidence arrives and choose where the next unit of effort is worth spending.

1. 1 Retain the full eligible cohort and represent unresolved or structurally unobserved outcomes as null, never as negative; verify that the recorded decision agrees with the governed probability threshold or expose an undocumented selection mechanism.
2. 2 Within actioned and unactioned strata, update a Jeffreys Beta posterior from genuinely observed outcomes and transform each draw into lower and upper missing-label event probabilities using the governed odds-ratio sensitivity envelope; use [0,1] when a stratum has no observed support.
3. 3 Combine fixed observed outcomes with the missing-label sensitivity draws to partially identify overall and stratum event rates, calibration gap, and Brier score while retaining population weights and probability-bin label coverage.
4. 4 Treat excessive missingness, a practically wide event-rate bound, threshold mismatch, or unsupported decision stratum as an abstention; the remedy is prospective label collection or design change rather than favorable imputation.

Before you trust it

Every tool in the catalog ships with the conditions under which its answer is meaningful — and the conditions under which it should abstain instead of guessing.

Calibrating it to your org

Calibration workflow

1. 1 Define the management decision, target outcome, aggregate unit, privacy boundary, cadence, and prediction/intervention horizon for this organization.
2. 2 Build a tenant-scoped historical cohort using only information available before each prediction or decision; preserve zero periods, censoring, assignment probabilities, and unresolved outcomes when the method requires them.
3. 3 Estimate statistical parameters on training history, but obtain costs, utilities, risk tolerance, practical-effect thresholds, capacity, and policy constraints from accountable decision owners.
4. 4 Validate on later time windows or held-out aggregate units at the deployment grain, against a simple baseline and the function-specific validation strategy.
5. 5 Deploy only if the returned decision clears evidence, overlap, calibration, robustness, and guardrail checks; warning, unsupported, schema-gap, and fallback decisions are abstentions.
6. 6 Monitor realized outcomes, data drift, coverage, and decision regret; recalibrate at a governed cadence or after a detected regime/definition change, never merely because a stakeholder dislikes the result.

Call it from your AI

You don't wire up 388 tools in your MCP client. The GitRevio MCP server exposes 18 tools, three of which let an agent search the catalog, read a tool's schema, and run it — so the assistant finds this one on its own.

gitrevio_capabilities_search
  { "q": "partially identify event risk calibration gap" }
  → finds "audit_selective_label_partial_identification"

gitrevio_capability_describe
  { "capability_id": "audit_selective_label_partial_identification" }
  → returns the input schema and agent guidance shown on this page

gitrevio_capability_run
  { "capability_id": "audit_selective_label_partial_identification", "arguments": { ... } }
  → returns the result shown above

Works in Claude Desktop, Claude Code, Cursor, Cline, Continue.dev, Goose and Aider. See the MCP server.

Related tools

See every tool in Measurement integrity →