> ## Documentation Index
> Fetch the complete documentation index at: https://doc.blueapi.ir/llms.txt
> Use this file to discover all available pages before exploring further.

# EMEP Observability: Logs, Metrics, Traces, and Alerts

> Observability pipeline for EMEP: logs, metrics, traces, model-level metrics, evaluation metrics, and alerting hooks with producer-collector-storage-dashboard flow.

EMEP observability covers logs, metrics, and traces across all 16 components. Model-level metrics track candidate throughput, fitness distribution, GPU utilization, and VRAM. Evaluation metrics feed into experiment dashboards. Alerting hooks trigger on experiment state transitions, resource thresholds, and benchmark anomalies. This page includes the observability pipeline diagram.

## Observability Pipeline

```mermaid theme={null}
flowchart LR
    subgraph PRODUCERS
        API[API Server]
        SCHED[Scheduler]
        GW[GPU Worker]
        EW[Eval Worker]
        EE[EvolutionEngine]
        BE[BenchmarkEngine]
    end
    subgraph COLLECTORS
        LOG[Log Collector]
        MET[Metrics Collector]
        TRACE[Trace Collector]
    end
    subgraph STORAGE
        LS[Log Store]
        TS[Time-Series DB]
        TRS[Trace Store]
        ES[Experiment State DB]
    end
    subgraph CONSUMERS
        DASH[Dashboards]
        ALERT[Alert Manager]
        AUDIT[Audit Log]
    end
    API --> LOG
    API --> MET
    SCHED --> LOG
    SCHED --> MET
    GW --> LOG
    GW --> MET
    GW --> TRACE
    EW --> LOG
    EW --> MET
    EW --> TRACE
    EE --> MET
    BE --> MET
    LOG --> LS
    MET --> TS
    TRACE --> TRS
    API --> ES
    SCHED --> ES
    LS --> DASH
    LS --> AUDIT
    TS --> DASH
    TS --> ALERT
    TRS --> DASH
    ES --> DASH
    ES --> ALERT
```

## Log Specification

Every component emits structured JSON logs with the following required fields.

| Field           | Type     | Example                                    |
| --------------- | -------- | ------------------------------------------ |
| `timestamp`     | ISO 8601 | `2024-01-15T09:23:47Z`                     |
| `level`         | string   | `INFO`, `WARN`, `ERROR`                    |
| `component`     | string   | `MergeEngine`                              |
| `trace_id`      | UUID     | `a1b2c3d4-...`                             |
| `experiment_id` | UUID     | `e5f6g7h8-...`                             |
| `message`       | string   | `Merge completed`                          |
| `context`       | object   | `{ strategy: "SLERP", duration_ms: 4523 }` |

Log levels:

* `DEBUG`: Tensor shape details, scheduler polling
* `INFO`: Job completion, state transitions, artifact storage
* `WARN`: Retry attempts, fallback to CPU, fair-share deficit
* `ERROR`: Merge failure, benchmark crash, GPU OOM

## Metrics Specification

Metrics are emitted in Prometheus exposition format. Key metric categories:

**System metrics:**

* `emep_gpu_utilization_percent{gpu_id, worker_id}`
* `emep_gpu_vram_used_bytes{gpu_id, worker_id}`
* `emep_gpu_vram_total_bytes{gpu_id, worker_id}`
* `emep_scheduler_queue_depth{queue_name}`
* `emep_scheduler_wait_time_seconds{queue_name, priority}`

**Model metrics:**

* `emep_candidate_throughput_per_hour{experiment_id}`
* `emep_fitness_distribution{experiment_id, percentile}`
* `emep_merge_duration_seconds{strategy}`
* `emep_eval_duration_seconds{benchmark_name}`

**Experiment metrics:**

* `emep_experiment_state_duration_seconds{experiment_id, state}`
* `emep_experiment_candidate_count{experiment_id, status}`
* `emep_population_generation{experiment_id}`

## Trace Specification

Distributed traces follow OpenTelemetry conventions. Spans are created at job submission, queue wait, GPU allocation, execution phases, and artifact storage.

Key spans:

* `merge_job.submit` (API Server)
* `merge_job.queue_wait` (Scheduler)
* `merge_job.gpu_allocate` (Scheduler)
* `merge_job.tensor_load` (TensorEngine)
* `merge_job.merge_execute` (MergeEngine)
* `merge_job.artifact_store` (ArtifactStore)

Trace sampling: 100% for failed jobs, 10% for successful jobs, 100% for critical priority.

## Dashboards

| Dashboard            | Data Source                  | Refresh Rate     |
| -------------------- | ---------------------------- | ---------------- |
| Experiment Overview  | Experiment State DB          | 5 seconds        |
| GPU Utilization      | Time-Series DB               | 10 seconds       |
| Fitness Distribution | Time-Series DB               | 30 seconds       |
| Merge Performance    | Time-Series DB + Trace Store | 1 minute         |
| Queue Depth          | Time-Series DB               | 5 seconds        |
| Audit Log            | Log Store                    | Real-time search |

## Alerting Hooks

Alerts are configured as Prometheus alert rules routed through Alertmanager.

| Alert             | Condition                                                     | Severity | Route             |
| ----------------- | ------------------------------------------------------------- | -------- | ----------------- |
| ExperimentFailed  | `emep_experiment_state_duration_seconds{state="FAILED"} > 0`  | Critical | PagerDuty + Slack |
| GPUOOM            | `emep_gpu_vram_used_bytes / emep_gpu_vram_total_bytes > 0.95` | Warning  | Slack             |
| QueueStalled      | `emep_scheduler_queue_depth > 100` for 10 minutes             | Warning  | Slack             |
| FitnessRegression | `emep_fitness_distribution{percentile="p50"} < baseline`      | Critical | PagerDuty         |
| BenchmarkAnomaly  | `emep_eval_duration_seconds > 2 * historical_median`          | Warning  | Email             |
| WorkerDown        | `up{job="gpu_worker"} == 0` for 2 minutes                     | Critical | PagerDuty         |

## Traceability Footer

| Spec Reference                                             | Phase   |
| ---------------------------------------------------------- | ------- |
| [Runtime Architecture](/architecture/runtime-architecture) | Phase 3 |
| [GPU Orchestration](/architecture/gpu-orchestration)       | Phase 3 |
| [Monitoring](/operations/monitoring)                       | Phase 4 |
| [Alerting](/operations/alerting)                           | Phase 4 |