Audit AI evaluation contamination integrity
Audit frozen AI evaluation suites for temporal or answer leakage, model-version mismatch, incomplete pre-label predictions, weak label provenance, missing subgroup support, cross-suite case reuse and near-duplicate content components before evaluation scores are trusted.
What it's for
Shows leaders whether an impressive AI evaluation is genuinely blind and independent—or inflated by answer leakage, old public cases, repeated content and unsupported subgroups.
What you give it
Inputs split into evidence read from your connected systems, calibration your team owns, and numerical controls that affect precision but never the result's meaning.
| Field | Type | Role | Required |
|---|---|---|---|
| as_of_period | integer ≥ 0 | Your calibration | Yes |
| case_results | array of objects (8 fields) | Evidence | Yes |
| case_similarity_edges | array of objects (5 fields) | Evidence | Yes |
| evaluation_cases | array of objects (9 fields) | Evidence | Yes |
| evaluation_suites | array of objects (13 fields) | Evidence | Yes |
| max_detail_rows | integer ≥ 1, ≤ 500 | Numerical control | Optional |
| maximum_cross_suite_case_reuse_fraction | number ≥ 0, ≤ 1 | Your calibration | Optional |
| minimum_evidence_coverage | number ≥ 0, ≤ 1 | Your calibration | Optional |
| minimum_subgroup_cases | integer ≥ 1 | Your calibration | Optional |
| minimum_temporal_holdout_fraction | number ≥ 0, ≤ 1 | Your calibration | Optional |
| minimum_unique_content_fraction | number ≥ 0, ≤ 1 | Your calibration | Optional |
| similarity_threshold | number > 0, ≤ 1 | Your calibration | Optional |
Each evaluation_suites
record
| Field | Type | Required |
|---|---|---|
| case_ids | array of string (≥ 2 items) | Yes |
| evaluation_period | integer (≥ 0) | Yes |
| evaluator_independent | boolean | Yes |
| evidence_verified | boolean | Yes |
| frozen_before_evaluation | boolean | Yes |
| high_impact | boolean | Yes |
| id | string (non-empty) | Yes |
| labels_blinded_until_prediction | boolean | Yes |
| model_version | string (non-empty) | Yes |
| required_subgroup_ids | array of string (≥ 1 item) | Yes |
| training_data_cutoff_period | integer (≥ 0) | Yes |
| value_at_risk | number (> 0) | Yes |
| workload_class_id | string (non-empty) | Yes |
{
"as_of_period": 7,
"case_results": [
{
"case_id": "ai-eval-case-00",
"evidence_verified": true,
"id": "result-ai-eval-case-00",
"model_version": "support-v2",
"prediction_logged_before_label_access": true,
"prediction_period": 5,
"score": 0.9,
"suite_id": "support-v2-eval"
},
{
"case_id": "ai-eval-case-01",
"evidence_verified": true,
"id": "result-ai-eval-case-01",
"model_version": "support-v2",
"prediction_logged_before_label_access": true,
"prediction_period": 5,
"score": 0.9,
"suite_id": "support-v2-eval"
},
{
"case_id": "ai-eval-case-02",
"evidence_verified": true,
"id": "result-ai-eval-case-02",
"model_version": "support-v2",
"prediction_logged_before_label_access": true,
"prediction_period": 5,
"score": 0.9,
"suite_id": "support-v2-eval"
},
{
"case_id": "ai-eval-case-03",
"evidence_verified": true,
"id": "result-ai-eval-case-03",
"model_version": "support-v2",
"prediction_logged_before_label_access": true,
"prediction_period": 5,
"score": 0.9,
"suite_id": "support-v2-eval"
},
{Truncated for display — the full payload is 989 lines.
What you get back
This is the actual output of running the example above — computed by the same function the platform calls, not an illustration.
{
"assumptions": [
"Suite scope, model version, training cutoff, cases, subgroups, labels and analysis rules were frozen before evaluation. Temporal separation indicates possible holdout eligibility but does not prove absence from training data.",
"Submitted similarity edges are a governed content comparison at the declared threshold; connected components prevent near-duplicates from being counted as independent evidence.",
"Predictions are immutable and precede label or answer access. Label provenance, evaluator independence and subgroup definitions are locally governed and evidence-backed.",
"Passing integrity permits evaluation analysis; it does not prove production validity, causal business value, model safety or authority to deploy, procure, route traffic or transfer data."
],
"configuration": {
"as_of_period": 7,
"content_rule": "connected_components_of_submitted_case_similarity_edges_at_or_above_threshold",
"maximum_cross_suite_case_reuse_fraction": 0,
"minimum_evidence_coverage": 0.95,
"minimum_subgroup_cases": 20,
"minimum_temporal_holdout_fraction": 0.9,
"minimum_unique_content_fraction": 0.9,
"similarity_threshold": 0.95,
"value_rule": "each_unsupported_suite_value_counted_once"
},
"decision": "ai_evaluation_contamination_integrity_supported",
"failed_gates": [],
"method": "point_in_time_ai_evaluation_contamination_graph_audit_v1",
"suite_diagnostics": [
{
"answer_protected_fraction": 1,
"case_count": 40,
"complete_pre_label_prediction_fraction": 1,
"cross_suite_case_reuse_fraction": 0,
"duplicate_content_clusters": [],
"evidence_coverage": 1,
"failed_gates": [],
"independent_content_component_count": 40,
"label_provenance_fraction": 1,
"model_version": "support-v2",
"required_subgroup_case_counts": {
"billing": 20,
"technical": 20
},
"suite_id": "support-v2-eval",
"supported": true,
"temporal_holdout_fraction": 1,
"truncated_duplicate_cluster_count": 0,
"unique_content_fraction": 1,
"value_at_risk": 100000,
"workload_class_id": "support"Truncated for display — the full payload is 54 lines.
How it works
Forecasting & survival — Estimate when something completes or fails, with censoring and unresolved work handled honestly rather than dropped.
- 1 Freeze suite, model version, training cutoff, cases, required subgroups, labels and evaluator independence before predictions or outcomes are inspected.
- 2 Reconcile every suite-case prediction to the exact model and pre-label timestamp, then build thresholded case-similarity components so near-duplicates contribute one independent content unit.
- 3 Gate temporal holdout, answer protection, label provenance, content independence, subgroup support, cross-suite reuse and evidence coverage; count each unsupported suite's value once and abstain before production claims.
Before you trust it
Every tool in the catalog ships with the conditions under which its answer is meaningful — and the conditions under which it should abstain instead of guessing.
Assumptions & guardrails
- Training examples precede their outcomes, censoring and unresolved work are represented, and deployment populations remain comparable to validation cohorts.
- The submitted similarity graph is complete enough at the governed threshold, training cutoffs cover every relevant training source, labels and answers remained inaccessible, and subgroup definitions predate results.
- A predictive interval or risk estimate is not a deadline promise, causal explanation, or individual-performance judgment.
- Passing establishes evaluation integrity, not production validity, causal value, model safety, vendor quality, deployment/procurement/data-transfer approval or a judgment about a provider, team or person.
Minimum evidence
- evaluation_suites: required and organization-defined
- evaluation_cases: required and organization-defined
- case_similarity_edges: required and organization-defined
- case_results: required and organization-defined
- as_of_period: required and organization-defined
How to validate it
Use chronological train/calibration/test windows, compare proper scores and decision value with a simple baseline, and recalibrate only from outcomes resolved after prediction time.
Calibrating it to your org
Same for everyone
The mathematical kernel, validation rules, method version, and JSON output semantics are organization-independent; no tenant-trained coefficients or company benchmark is embedded in the function.
Specific to you
- one point-in-time suite-case-model projection plus thresholded similarity graph, preserving missing results, near-duplicate connected components, cross-suite reuse and the evidence version visible before evaluation conclusions
- training-source cutoff semantics, suite freeze and blinding, evaluator independence, label provenance, answer access, subgroup taxonomy/support, similarity model/version/threshold, cross-suite reuse, evidence coverage, unique value/currency/horizon and privacy-safe case identity
Calibration workflow
- 1 Define the management decision, target outcome, aggregate unit, privacy boundary, cadence, and prediction/intervention horizon for this organization.
- 2 Build a tenant-scoped historical cohort using only information available before each prediction or decision; preserve zero periods, censoring, assignment probabilities, and unresolved outcomes when the method requires them.
- 3 Estimate statistical parameters on training history, but obtain costs, utilities, risk tolerance, practical-effect thresholds, capacity, and policy constraints from accountable decision owners.
- 4 Validate on later time windows or held-out aggregate units at the deployment grain, against a simple baseline and the function-specific validation strategy.
- 5 Deploy only if the returned decision clears evidence, overlap, calibration, robustness, and guardrail checks; warning, unsupported, schema-gap, and fallback decisions are abstentions.
- 6 Monitor realized outcomes, data drift, coverage, and decision regret; recalibrate at a governed cadence or after a detected regime/definition change, never merely because a stakeholder dislikes the result.
Call it from your AI
You don't wire up 388 tools in your MCP client. The GitRevio MCP server exposes 18 tools, three of which let an agent search the catalog, read a tool's schema, and run it — so the assistant finds this one on its own.
gitrevio_capabilities_search
{ "q": "audit frozen ai evaluation suites for" }
→ finds "audit_ai_evaluation_contamination_integrity"
gitrevio_capability_describe
{ "capability_id": "audit_ai_evaluation_contamination_integrity" }
→ returns the input schema and agent guidance shown on this page
gitrevio_capability_run
{ "capability_id": "audit_ai_evaluation_contamination_integrity", "arguments": { ... } }
→ returns the result shown aboveWorks in Claude Desktop, Claude Code, Cursor, Cline, Continue.dev, Goose and Aider. See the MCP server.
Related tools
Forecast AI evaluation production validity
Forecast whether offline AI evaluation scores will remain valid in production using a tenant-pooled Bayesian logit calibration with workload effects and time drift, coherent operating scenarios, false-promotion risk, breach timing, net value and quality-shortfall VaR/CVaR.
Optimize AI evaluation value of information portfolio
Select additional AI evaluation plans by multi-stratum posterior-predictive value of sample information, discounting duplicate content and optimizing budget, reviewer capacity, delay, quality lower bounds and expected/CVaR incremental false-deployment loss with exact or disclosed beam search.
Audit agentic action control integrity
Audit operational AI-agent actions from bounded least-privilege permission scope through independently tested authorization, approval, sandbox, monitoring, rollback or compensation, and kill-switch controls, counting unique value exposure once.
Audit AI configuration release integrity
Audit that the exact immutable AI configuration bundle evaluated and approved is the bundle exposed in every staged rollout, with consecutive parent lineage, complete blast-radius declaration, effective runtime controls, monotone traffic and a tested prior-version rollback path.
Audit AI data rights provenance integrity
Audit every AI training, fine-tuning, retrieval, evaluation, logging and persisted-output use against an immutable rights grant and the complete derivative lineage, including time, revocation, deletion, purpose, jurisdiction, consent, derivative and evidence gates.
Audit AI output IP provenance integrity
Audit aggregate AI outputs against the exact model and provider terms effective at generation, pre-commercialization similarity evidence, counsel-owned ownership/use rules, human review and any claimed indemnity coverage.