> ## Documentation Index
> Fetch the complete documentation index at: https://doc.blueapi.ir/llms.txt
> Use this file to discover all available pages before exploring further.

# EMEP GPU Orchestration: Scheduling and Allocation

> GPU scheduling for EMEP: queue design, priority, preemption, fair-share, per-experiment allocation, multi-node support, and topology awareness for NVLink groups.

EMEP GPU orchestration manages heterogeneous GPU pools for merge, evolution, and evaluation workloads. The scheduler supports priority queues, preemption, fair-share allocation, multi-node jobs, and topology-aware placement for NVLink groups. This page includes a scheduling pipeline flowchart and a sequence diagram of a job requesting GPUs.

## Scheduling Pipeline Flowchart

```mermaid theme={null}
flowchart TB
    START([Job Submitted]) --> PARSE[Parse Resource Request]
    PARSE --> VALIDATE{Valid Request?}
    VALIDATE -- No --> REJECT[Reject Job]
    REJECT --> END1([END])
    VALIDATE -- Yes --> PRIORITY[Assign Priority]
    PRIORITY --> QUEUE[Enqueue]
    QUEUE --> POLL[Scheduler Poll]
    POLL --> CHECK{GPU Available?}
    CHECK -- No --> WAIT[Wait / Preempt Lower Priority]
    WAIT --> POLL
    CHECK -- Yes --> TOPO[Topology Check]
    TOPO --> NVLINK{NVLink Group Required?}
    NVLINK -- Yes --> PLACE_NVLINK[Place in NVLink Group]
    NVLINK -- No --> PLACE_ANY[Place on Any GPU]
    PLACE_NVLINK --> ALLOC[Allocate GPUs]
    PLACE_ANY --> ALLOC
    ALLOC --> RUN[Run Job]
    RUN --> COMPLETE{Job Complete?}
    COMPLETE -- Yes --> RELEASE[Release GPUs]
    COMPLETE -- No --> RETRY{Retryable?}
    RETRY -- Yes --> QUEUE
    RETRY -- No --> FAIL[Mark Failed]
    RELEASE --> END2([END])
    FAIL --> END2
```

## Queue Design

| Queue           | Priority Levels                | Preemption                       | Fair-Share Weight |
| --------------- | ------------------------------ | -------------------------------- | ----------------- |
| Merge Queue     | Critical / High / Normal / Low | Critical preempts Normal and Low | 2.0               |
| Eval Queue      | High / Normal / Low            | No                               | 1.0               |
| Inference Queue | Realtime / Normal              | Realtime preempts Normal         | 3.0               |

Jobs specify `gpu_count`, `gpu_memory_min`, `nvlink_required`, and `max_duration`. The scheduler sorts by priority, then by fair-share deficit, then by submission time.

## Priority and Preemption

Critical priority is reserved for regression gates and production deployment validation. Critical jobs can preempt Normal and Low jobs. Preempted jobs are checkpointed and requeued at their original priority.

Preemption requires:

1. Experiment state checkpoint in ArtifactStore
2. GPU memory state saved via framework checkpoint (PyTorch checkpoint)
3. Job requeued with `preempted=true` flag

## Fair-Share Allocation

Each user or project receives a fair-share weight. The scheduler tracks GPU-hours consumed per entity. Entities below their fair-share target receive scheduling preference. Weights are configurable per project.

Formula: `score = priority * (1 + (target - consumed) / target)`

## Multi-Node and Topology Awareness

Multi-node jobs request `node_count * gpu_per_node`. The scheduler attempts to place all GPUs within the same rack or NVLink domain to minimize cross-node communication.

**NVLink groups:** GPUs connected via NVLink are treated as a single scheduling unit. Jobs requesting NVLink are placed entirely within one group or rejected if no group has sufficient capacity.

**Topology labels:** Each GPU worker advertises `nvlink_group_id`, `rack_id`, and `network_bandwidth_gbps`. The scheduler uses these labels for affinity and anti-affinity rules.

## GPU Job Sequence Diagram

```mermaid theme={null}
sequenceDiagram
    participant API as API Server
    participant SCHED as Scheduler
    participant Q as Merge Queue
    participant GW as GPU Worker
    participant TE as TensorEngine
    participant ME as MergeEngine
    participant AS as ArtifactStore
    API->>SCHED: submit_job(gpu_count=2, nvlink=true, duration=4h)
    SCHED->>SCHED: validate_request
    SCHED->>SCHED: assign_priority=High
    SCHED->>Q: enqueue(job)
    loop Scheduler Poll
        SCHED->>Q: peek_next()
        Q-->>SCHED: job
        SCHED->>SCHED: check_fair_share
        SCHED->>SCHED: find_nvlink_group(size=2)
    end
    SCHED->>GW: allocate(gpus=[GPU3, GPU4], nvlink=true)
    GW-->>SCHED: allocated
    GW->>TE: load_tensors(model_a, model_b)
    TE-->>GW: tensors
    GW->>ME: merge(strategy=SLERP, t=0.5)
    ME-->>GW: candidate
    GW->>AS: store(candidate)
    AS-->>GW: artifact_id
    GW->>SCHED: release(gpus=[GPU3, GPU4])
    SCHED-->>GW: released
    SCHED->>Q: mark_complete(job)
```

## GPU Health and Reclamation

GPU workers report utilization, temperature, and memory state every 10 seconds. The scheduler marks GPUs as UNHEALTHY if three consecutive health checks fail. Jobs on unhealthy GPUs are preempted and requeued.

Stale allocations are reclaimed if a worker fails to heartbeat for 60 seconds. Checkpoint recovery is attempted before requeue.

## Traceability Footer

| Spec Reference                                                 | Phase   |
| -------------------------------------------------------------- | ------- |
| [Runtime Architecture](/architecture/runtime-architecture)     | Phase 3 |
| [Component Architecture](/architecture/component-architecture) | Phase 2 |
| [Observability](/architecture/observability)                   | Phase 3 |
| [GPU Health](/operations/gpu-health)                           | Phase 4 |