EMEP Runtime Architecture: Processes, Workers, and Queues

Runtime topology for EMEP: API server, scheduler, GPU workers, CPU workers, evaluation workers, and message flow. Includes merge job sequence diagram.

EMEP runtime separates control plane from execution plane. The API server and scheduler run on the control plane. GPU workers, CPU workers, evaluation workers, artifact writers, and tracker publishers run on the execution plane. This page specifies process topology, queue design, and message flow with a sequence diagram of a merge job traversing the runtime.

Runtime Topology

Process Descriptions

API Server: Receives external requests via REST and CLI gateway. Validates authentication, enforces rate limits, and submits jobs to the scheduler. Stateless. Horizontally scalable.

Scheduler: Maintains job queues, assigns priorities, and dispatches to workers. Tracks GPU allocation state. Supports preemption for high-priority experiments.

GPU Workers: Execute TensorEngine, MergeEngine, and EvolutionEngine operations. One worker owns one or more GPUs. Runs in a container with NVIDIA/ROCm drivers mounted.

CPU Workers: Handle data preprocessing, tokenizer operations, and lightweight compatibility checks. Also serves as GPU worker fallback on OOM.

Evaluation Workers: Run BenchmarkEngine and EvaluationEngine. Isolated from merge workers to prevent resource contention. Loads models from ArtifactStore into InferenceBackend.

Artifact Writer: Asynchronously writes large artifacts to ArtifactStore. Decouples workers from slow object-store uploads.

Tracker Publisher: Batches experiment events and writes to ExperimentTracker. Uses at-least-once delivery with idempotent writes.

Queue Design

Queue	Type	Priority	Preemption	TTL
Merge Queue	GPU-bound	High/Medium/Low	High can preempt Medium	24 hours
Eval Queue	GPU/CPU-bound	Medium	No	48 hours
Artifact Queue	I/O-bound	Low	No	1 hour
Tracker Queue	Memory-bound	Lowest	No	5 minutes

Messages carry trace_id, experiment_id, and job_type. Dead-letter queues capture failed jobs for manual inspection.

Merge Job Sequence Diagram

Event Loop and Concurrency

Each worker runs an asyncio event loop. GPU workers pin one process per GPU to avoid CUDA context switching. Evaluation workers use multiprocessing for backend isolation. CPU workers use thread pools for I/O-bound tasks.

Failure Handling

Worker crashes trigger scheduler-level retry with exponential backoff. GPU OOM events promote the job to a CPU worker with chunked execution. Tracker Publisher failures buffer events to local disk for replay.

Traceability Footer

Spec Reference	Phase
System Architecture	Phase 1
Component Architecture	Phase 2
GPU Orchestration	Phase 3
Observability	Phase 3