Regression Detection and Statistical Protocol

Specification for regression evaluation in EMEP, covering statistical protocols for detecting performance regression against baselines using bootstrap confidence intervals.

Regression evaluation determines whether a candidate model performs significantly worse than a baseline on critical metrics. EMEP uses statistical hypothesis testing with bootstrap confidence intervals to make this determination robustly. This page specifies the protocol, sample size requirements, and regression classification.

Regression Detection Flow

Statistical Protocol

Minimum Sample Size

The RegressionEngine requires at least 30 independent samples per metric for bootstrap confidence intervals. If fewer samples are available, the engine reports insufficient data rather than a regression verdict.

Bootstrap Procedure

1. Compute per-sample differences: candidate_score - baseline_score
2. Resample differences with replacement 10,000 times
3. Compute mean difference for each resample
4. Extract 2.5th and 97.5th percentiles as the 95% confidence interval

Regression Decision

A regression is detected when the entire 95% confidence interval for the mean difference is below zero. This means the candidate is worse than baseline with at least 95% confidence.

if CI_upper < 0: regression = True
else: regression = False

Regression Classes

When regression is detected, it is classified by severity and scope:

Class	Condition	Action
Critical Regression	Any safety metric below baseline	Candidate status: FAIL
Major Regression	Primary metric CI entirely below baseline with magnitude > 5%	Candidate status: REGRESSION
Minor Regression	Secondary metric CI below baseline, magnitude < 5%	Flag for review, status: PASS
No Regression	CI includes or is above zero	No action

Metric-Specific Thresholds

Different metrics may have different regression thresholds:

• Accuracy: 2 percentage point drop
• Exact Match: 3 percentage point drop
• ROUGE-L: 0.02 absolute drop
• BLEU-4: 0.01 absolute drop
• Safety Pass Rate: any drop is critical
• Latency: 20% increase

Thresholds are configurable per experiment. The default is a statistically significant drop with no minimum magnitude.

Multiple Comparisons

When testing many metrics, the chance of false positives increases. The RegressionEngine applies a simple Bonferroni correction: divide the significance level (0.05) by the number of metrics tested. For 10 metrics, the effective threshold is 0.005.

Baseline Selection

The baseline model is specified in the experiment configuration. Typical choices:

• The best-performing parent model
• The previous release candidate
• A fixed reference model maintained by the project

Baseline scores are cached after first computation. If the baseline changes mid-experiment, cached scores are invalidated.

Reporting

Regression reports include:

• Baseline and candidate model IDs
• Metric name and sample count
• Mean difference and 95% CI bounds
• Regression class and recommended action
• Raw score distributions (for manual review)

Integration

The RegressionEngine is called by the EvaluationEngine after benchmark aggregation. Results are stored in the ExperimentTracker and influence candidate status assignment. See Evaluation Framework for the full evaluation pipeline.

Regression evaluation uses the Optimization Set for evolution-phase checks. Post-evolution regression checks may use the Validation Set or Hidden Test Set for final verification.