> ## Documentation Index
> Fetch the complete documentation index at: https://doc.blueapi.ir/llms.txt
> Use this file to discover all available pages before exploring further.

# Monitoring Pipeline and Metrics Catalog

> Metrics catalog, dashboard definitions, and the OpenTelemetry pipeline for GPU health, experiment throughput, evaluation queues, and storage capacity.

EMEP exposes metrics from every component for real-time visibility into experiments, GPU utilization, and system health. This page defines the metrics catalog, the dashboards operators use, and the telemetry pipeline that carries measurements from producers to storage.

## Metrics Catalog

| Metric                     | Type      | Source             | Description                               |
| -------------------------- | --------- | ------------------ | ----------------------------------------- |
| experiment.active          | Gauge     | ExperimentTracker  | Number of experiments in RUNNING state    |
| experiment.queue\_depth    | Gauge     | ExperimentTracker  | Experiments waiting for GPU allocation    |
| experiment.duration        | Histogram | ExperimentTracker  | Wall-clock time from CREATED to COMPLETED |
| merge.tensor\_ops\_rate    | Counter   | MergeEngine        | Tensor operations per second during merge |
| eval.tasks\_completed      | Counter   | EvaluationEngine   | Evaluation tasks finished                 |
| eval.queue\_wait           | Histogram | EvaluationEngine   | Time from task submission to start        |
| gpu.utilization            | Gauge     | GPU Health Agent   | GPU compute utilization percentage        |
| gpu.memory.used            | Gauge     | GPU Health Agent   | GPU memory allocated in bytes             |
| gpu.ecc.errors             | Counter   | GPU Health Agent   | Corrected ECC errors since boot           |
| gpu.xid.errors             | Counter   | GPU Health Agent   | Uncorrectable XID errors                  |
| storage.capacity.bytes     | Gauge     | Storage Operations | Total object store capacity               |
| storage.used.bytes         | Gauge     | Storage Operations | Used object store bytes                   |
| registry.models.total      | Gauge     | ModelRegistry      | Total models in REGISTERED or later state |
| registry.models.deprecated | Gauge     | ModelRegistry      | Models in DEPRECATED or ARCHIVED state    |
| api.requests               | Counter   | API Gateway        | Total API requests                        |
| api.latency                | Histogram | API Gateway        | Request latency in milliseconds           |

## Dashboards

| Dashboard             | Panels                                                | Audience                 |
| --------------------- | ----------------------------------------------------- | ------------------------ |
| GPU Health            | ECC errors, thermal, XID errors, memory, PCIe errors  | Infrastructure operators |
| Experiment Throughput | Active experiments, queue depth, duration percentiles | Experiment leads         |
| Evaluation Queue      | Tasks completed, wait time, failure rate              | Evaluation engineers     |
| Storage Capacity      | Used vs total, growth rate, cold tier ratio           | Storage operators        |
| Registry Activity     | Models by lifecycle state, daily registrations        | Registry curators        |

## Telemetry Pipeline

```mermaid theme={null}
flowchart LR
    subgraph PRODUCERS["Metric Producers"]
        E[ExperimentTracker]
        M[MergeEngine]
        V[EvaluationEngine]
        G[GPU Health Agent]
        S[Storage Operations]
        R[ModelRegistry]
        A[API Gateway]
    end

    subgraph COLLECTORS["OTel Collectors"]
        OC1[Collector Pool 1]
        OC2[Collector Pool 2]
    end

    subgraph STORAGE["Storage"]
        TSDB[Time-Series DB]
        RAW[Raw Log Archive]
    end

    subgraph CONSUMERS["Consumers"]
        DASH[Dashboards]
        ALERT[Alert Manager]
    end

    PRODUCERS -->|OTLP| COLLECTORS
    COLLECTORS -->|Remote Write| TSDB
    COLLECTORS -->|Batch Export| RAW
    TSDB --> DASH
    TSDB --> ALERT
```

Producers emit metrics in OpenTelemetry Protocol (OTLP). Collectors batch, filter, and forward to the time-series database. Raw logs archive to cold storage for forensic replay. Dashboards and alerts read from the time-series database.

## Retention

| Data Type           | Hot Retention | Cold Retention |
| ------------------- | ------------- | -------------- |
| Metrics             | 30 days       | 1 year         |
| Raw OTLP logs       | 7 days        | 90 days        |
| Dashboard snapshots | 90 days       | 1 year         |

## Cross-Links

* [Alerting](/operations/alerting)
* [GPU Health](/operations/gpu-health)
* [Storage Operations](/operations/storage-operations)