EMEP Observability: Logs, Metrics, Traces, and Alerts

Observability pipeline for EMEP: logs, metrics, traces, model-level metrics, evaluation metrics, and alerting hooks with producer-collector-storage-dashboard flow.

EMEP observability covers logs, metrics, and traces across all 16 components. Model-level metrics track candidate throughput, fitness distribution, GPU utilization, and VRAM. Evaluation metrics feed into experiment dashboards. Alerting hooks trigger on experiment state transitions, resource thresholds, and benchmark anomalies. This page includes the observability pipeline diagram.

Observability Pipeline

Log Specification

Every component emits structured JSON logs with the following required fields.

Field	Type	Example
timestamp	ISO 8601	2024-01-15T09:23:47Z
level	string	INFO, WARN, ERROR
component	string	MergeEngine
trace_id	UUID	a1b2c3d4-...
experiment_id	UUID	e5f6g7h8-...
message	string	Merge completed
context	object	{ strategy: "SLERP", duration_ms: 4523 }

Log levels:

• DEBUG: Tensor shape details, scheduler polling
• INFO: Job completion, state transitions, artifact storage
• WARN: Retry attempts, fallback to CPU, fair-share deficit
• ERROR: Merge failure, benchmark crash, GPU OOM

Metrics Specification

Metrics are emitted in Prometheus exposition format. Key metric categories:

System metrics:

• emep_gpu_utilization_percent{gpu_id, worker_id}
• emep_gpu_vram_used_bytes{gpu_id, worker_id}
• emep_gpu_vram_total_bytes{gpu_id, worker_id}
• emep_scheduler_queue_depth{queue_name}
• emep_scheduler_wait_time_seconds{queue_name, priority}

Model metrics:

• emep_candidate_throughput_per_hour{experiment_id}
• emep_fitness_distribution{experiment_id, percentile}
• emep_merge_duration_seconds{strategy}
• emep_eval_duration_seconds{benchmark_name}

Experiment metrics:

• emep_experiment_state_duration_seconds{experiment_id, state}
• emep_experiment_candidate_count{experiment_id, status}
• emep_population_generation{experiment_id}

Trace Specification

Distributed traces follow OpenTelemetry conventions. Spans are created at job submission, queue wait, GPU allocation, execution phases, and artifact storage.

Key spans:

• merge_job.submit (API Server)
• merge_job.queue_wait (Scheduler)
• merge_job.gpu_allocate (Scheduler)
• merge_job.tensor_load (TensorEngine)
• merge_job.merge_execute (MergeEngine)
• merge_job.artifact_store (ArtifactStore)

Trace sampling: 100% for failed jobs, 10% for successful jobs, 100% for critical priority.

Dashboards

Dashboard	Data Source	Refresh Rate
Experiment Overview	Experiment State DB	5 seconds
GPU Utilization	Time-Series DB	10 seconds
Fitness Distribution	Time-Series DB	30 seconds
Merge Performance	Time-Series DB + Trace Store	1 minute
Queue Depth	Time-Series DB	5 seconds
Audit Log	Log Store	Real-time search

Alerting Hooks

Alerts are configured as Prometheus alert rules routed through Alertmanager.

Alert	Condition	Severity	Route
ExperimentFailed	emep_experiment_state_duration_seconds{state="FAILED"} > 0	Critical	PagerDuty + Slack
GPUOOM	emep_gpu_vram_used_bytes / emep_gpu_vram_total_bytes > 0.95	Warning	Slack
QueueStalled	emep_scheduler_queue_depth > 100 for 10 minutes	Warning	Slack
FitnessRegression	emep_fitness_distribution{percentile="p50"} < baseline	Critical	PagerDuty
BenchmarkAnomaly	emep_eval_duration_seconds > 2 * historical_median	Warning	Email
WorkerDown	up{job="gpu_worker"} == 0 for 2 minutes	Critical	PagerDuty

Traceability Footer

Spec Reference	Phase
Runtime Architecture	Phase 3
GPU Orchestration	Phase 3
Monitoring	Phase 4
Alerting	Phase 4