> ## Documentation Index
> Fetch the complete documentation index at: https://doc.blueapi.ir/llms.txt
> Use this file to discover all available pages before exploring further.

# Regression Detection and Statistical Protocol

> Specification for regression evaluation in EMEP, covering statistical protocols for detecting performance regression against baselines using bootstrap confidence intervals.

Regression evaluation determines whether a candidate model performs significantly worse than a baseline on critical metrics. EMEP uses statistical hypothesis testing with bootstrap confidence intervals to make this determination robustly. This page specifies the protocol, sample size requirements, and regression classification.

## Regression Detection Flow

```mermaid theme={null}
flowchart TD
    START([START]) --> BASE[Load Baseline Scores]
    BASE --> CAND[Load Candidate Scores]
    CAND --> BOOT[Bootstrap Resampling]
    BOOT --> CI[Compute 95% CI for Difference]
    CI --> CHECK{CI Entirely Below Zero?}
    CHECK -->|Yes| REGRESS[Regression Detected]
    CHECK -->|No| NOREG[No Regression]
    REGRESS --> CLASS[Classify Regression]
    NOREG --> CLASS2[Classify: No Regression]
    CLASS --> REPORT[Report with CI Bounds]
    CLASS2 --> REPORT
    REPORT --> END([END])
```

## Statistical Protocol

**Minimum Sample Size**

The RegressionEngine requires at least 30 independent samples per metric for bootstrap confidence intervals. If fewer samples are available, the engine reports insufficient data rather than a regression verdict.

**Bootstrap Procedure**

1. Compute per-sample differences: candidate\_score - baseline\_score
2. Resample differences with replacement 10,000 times
3. Compute mean difference for each resample
4. Extract 2.5th and 97.5th percentiles as the 95% confidence interval

**Regression Decision**

A regression is detected when the entire 95% confidence interval for the mean difference is below zero. This means the candidate is worse than baseline with at least 95% confidence.

```text theme={null}
if CI_upper < 0: regression = True
else: regression = False
```

## Regression Classes

When regression is detected, it is classified by severity and scope:

| Class               | Condition                                                     | Action                        |
| ------------------- | ------------------------------------------------------------- | ----------------------------- |
| Critical Regression | Any safety metric below baseline                              | Candidate status: FAIL        |
| Major Regression    | Primary metric CI entirely below baseline with magnitude > 5% | Candidate status: REGRESSION  |
| Minor Regression    | Secondary metric CI below baseline, magnitude \< 5%           | Flag for review, status: PASS |
| No Regression       | CI includes or is above zero                                  | No action                     |

## Metric-Specific Thresholds

Different metrics may have different regression thresholds:

* **Accuracy**: 2 percentage point drop
* **Exact Match**: 3 percentage point drop
* **ROUGE-L**: 0.02 absolute drop
* **BLEU-4**: 0.01 absolute drop
* **Safety Pass Rate**: any drop is critical
* **Latency**: 20% increase

Thresholds are configurable per experiment. The default is a statistically significant drop with no minimum magnitude.

## Multiple Comparisons

When testing many metrics, the chance of false positives increases. The RegressionEngine applies a simple Bonferroni correction: divide the significance level (0.05) by the number of metrics tested. For 10 metrics, the effective threshold is 0.005.

## Baseline Selection

The baseline model is specified in the experiment configuration. Typical choices:

* The best-performing parent model
* The previous release candidate
* A fixed reference model maintained by the project

Baseline scores are cached after first computation. If the baseline changes mid-experiment, cached scores are invalidated.

## Reporting

Regression reports include:

* Baseline and candidate model IDs
* Metric name and sample count
* Mean difference and 95% CI bounds
* Regression class and recommended action
* Raw score distributions (for manual review)

## Integration

The RegressionEngine is called by the EvaluationEngine after benchmark aggregation. Results are stored in the ExperimentTracker and influence candidate status assignment. See [Evaluation Framework](/evaluation/framework) for the full evaluation pipeline.

<Info>
  Regression evaluation uses the Optimization Set for evolution-phase checks. Post-evolution regression checks may use the Validation Set or Hidden Test Set for final verification.
</Info>