Monitoring Pipeline and Metrics Catalog

Metrics catalog, dashboard definitions, and the OpenTelemetry pipeline for GPU health, experiment throughput, evaluation queues, and storage capacity.

EMEP exposes metrics from every component for real-time visibility into experiments, GPU utilization, and system health. This page defines the metrics catalog, the dashboards operators use, and the telemetry pipeline that carries measurements from producers to storage.

Metrics Catalog

Metric	Type	Source	Description
experiment.active	Gauge	ExperimentTracker	Number of experiments in RUNNING state
experiment.queue_depth	Gauge	ExperimentTracker	Experiments waiting for GPU allocation
experiment.duration	Histogram	ExperimentTracker	Wall-clock time from CREATED to COMPLETED
merge.tensor_ops_rate	Counter	MergeEngine	Tensor operations per second during merge
eval.tasks_completed	Counter	EvaluationEngine	Evaluation tasks finished
eval.queue_wait	Histogram	EvaluationEngine	Time from task submission to start
gpu.utilization	Gauge	GPU Health Agent	GPU compute utilization percentage
gpu.memory.used	Gauge	GPU Health Agent	GPU memory allocated in bytes
gpu.ecc.errors	Counter	GPU Health Agent	Corrected ECC errors since boot
gpu.xid.errors	Counter	GPU Health Agent	Uncorrectable XID errors
storage.capacity.bytes	Gauge	Storage Operations	Total object store capacity
storage.used.bytes	Gauge	Storage Operations	Used object store bytes
registry.models.total	Gauge	ModelRegistry	Total models in REGISTERED or later state
registry.models.deprecated	Gauge	ModelRegistry	Models in DEPRECATED or ARCHIVED state
api.requests	Counter	API Gateway	Total API requests
api.latency	Histogram	API Gateway	Request latency in milliseconds

Dashboards

Dashboard	Panels	Audience
GPU Health	ECC errors, thermal, XID errors, memory, PCIe errors	Infrastructure operators
Experiment Throughput	Active experiments, queue depth, duration percentiles	Experiment leads
Evaluation Queue	Tasks completed, wait time, failure rate	Evaluation engineers
Storage Capacity	Used vs total, growth rate, cold tier ratio	Storage operators
Registry Activity	Models by lifecycle state, daily registrations	Registry curators

Telemetry Pipeline

Producers emit metrics in OpenTelemetry Protocol (OTLP). Collectors batch, filter, and forward to the time-series database. Raw logs archive to cold storage for forensic replay. Dashboards and alerts read from the time-series database.

Retention

Data Type	Hot Retention	Cold Retention
Metrics	30 days	1 year
Raw OTLP logs	7 days	90 days
Dashboard snapshots	90 days	1 year

Cross-Links

• Alerting
• GPU Health
• Storage Operations