EMEP GPU Orchestration: Scheduling and Allocation

GPU scheduling for EMEP: queue design, priority, preemption, fair-share, per-experiment allocation, multi-node support, and topology awareness for NVLink groups.

EMEP GPU orchestration manages heterogeneous GPU pools for merge, evolution, and evaluation workloads. The scheduler supports priority queues, preemption, fair-share allocation, multi-node jobs, and topology-aware placement for NVLink groups. This page includes a scheduling pipeline flowchart and a sequence diagram of a job requesting GPUs.

Scheduling Pipeline Flowchart

Queue Design

Queue	Priority Levels	Preemption	Fair-Share Weight
Merge Queue	Critical / High / Normal / Low	Critical preempts Normal and Low	2.0
Eval Queue	High / Normal / Low	No	1.0
Inference Queue	Realtime / Normal	Realtime preempts Normal	3.0

Jobs specify gpu_count, gpu_memory_min, nvlink_required, and max_duration. The scheduler sorts by priority, then by fair-share deficit, then by submission time.

Priority and Preemption

Critical priority is reserved for regression gates and production deployment validation. Critical jobs can preempt Normal and Low jobs. Preempted jobs are checkpointed and requeued at their original priority.

Preemption requires:

1. Experiment state checkpoint in ArtifactStore
2. GPU memory state saved via framework checkpoint (PyTorch checkpoint)
3. Job requeued with preempted=true flag

Fair-Share Allocation

Each user or project receives a fair-share weight. The scheduler tracks GPU-hours consumed per entity. Entities below their fair-share target receive scheduling preference. Weights are configurable per project.

Formula: score = priority * (1 + (target - consumed) / target)

Multi-Node and Topology Awareness

Multi-node jobs request node_count * gpu_per_node. The scheduler attempts to place all GPUs within the same rack or NVLink domain to minimize cross-node communication.

NVLink groups: GPUs connected via NVLink are treated as a single scheduling unit. Jobs requesting NVLink are placed entirely within one group or rejected if no group has sufficient capacity.

Topology labels: Each GPU worker advertises nvlink_group_id, rack_id, and network_bandwidth_gbps. The scheduler uses these labels for affinity and anti-affinity rules.

GPU Job Sequence Diagram

GPU Health and Reclamation

GPU workers report utilization, temperature, and memory state every 10 seconds. The scheduler marks GPUs as UNHEALTHY if three consecutive health checks fail. Jobs on unhealthy GPUs are preempted and requeued.

Stale allocations are reclaimed if a worker fails to heartbeat for 60 seconds. Checkpoint recovery is attempted before requeue.

Traceability Footer

Spec Reference	Phase
Runtime Architecture	Phase 3
Component Architecture	Phase 2
Observability	Phase 3
GPU Health	Phase 4