> ## Documentation Index
> Fetch the complete documentation index at: https://doc.blueapi.ir/llms.txt
> Use this file to discover all available pages before exploring further.

# EMEP Runtime Architecture: Processes, Workers, and Queues

> Runtime topology for EMEP: API server, scheduler, GPU workers, CPU workers, evaluation workers, and message flow. Includes merge job sequence diagram.

EMEP runtime separates control plane from execution plane. The API server and scheduler run on the control plane. GPU workers, CPU workers, evaluation workers, artifact writers, and tracker publishers run on the execution plane. This page specifies process topology, queue design, and message flow with a sequence diagram of a merge job traversing the runtime.

## Runtime Topology

```mermaid theme={null}
flowchart TB
    subgraph CONTROL_PLANE
        API[API Server]
        SCHED[Scheduler]
        TRACK[Tracker Publisher]
    end
    subgraph EXECUTION_PLANE
        subgraph GPU_POOL
            GW1[GPU Worker 1]
            GW2[GPU Worker 2]
            GWN[GPU Worker N]
        end
        subgraph CPU_POOL
            CW1[CPU Worker 1]
            CW2[CPU Worker 2]
        end
        subgraph EVAL_POOL
            EW1[Evaluation Worker 1]
            EW2[Evaluation Worker 2]
        end
        AW[Artifact Writer]
    end
    subgraph QUEUES
        MQ[Merge Queue]
        EQ[Eval Queue]
        AQ[Artifact Queue]
        TQ[Tracker Queue]
    end
    API --> SCHED
    SCHED --> MQ
    SCHED --> EQ
    MQ --> GW1
    MQ --> GW2
    EQ --> EW1
    EQ --> EW2
    GW1 --> AQ
    EW1 --> AQ
    AQ --> AW
    AW --> TQ
    GW1 --> TQ
    EW1 --> TQ
    TQ --> TRACK
    CW1 --> MQ
    CW1 --> EQ
```

## Process Descriptions

**API Server:** Receives external requests via REST and CLI gateway. Validates authentication, enforces rate limits, and submits jobs to the scheduler. Stateless. Horizontally scalable.

**Scheduler:** Maintains job queues, assigns priorities, and dispatches to workers. Tracks GPU allocation state. Supports preemption for high-priority experiments.

**GPU Workers:** Execute TensorEngine, MergeEngine, and EvolutionEngine operations. One worker owns one or more GPUs. Runs in a container with NVIDIA/ROCm drivers mounted.

**CPU Workers:** Handle data preprocessing, tokenizer operations, and lightweight compatibility checks. Also serves as GPU worker fallback on OOM.

**Evaluation Workers:** Run BenchmarkEngine and EvaluationEngine. Isolated from merge workers to prevent resource contention. Loads models from ArtifactStore into InferenceBackend.

**Artifact Writer:** Asynchronously writes large artifacts to ArtifactStore. Decouples workers from slow object-store uploads.

**Tracker Publisher:** Batches experiment events and writes to ExperimentTracker. Uses at-least-once delivery with idempotent writes.

## Queue Design

| Queue          | Type          | Priority        | Preemption              | TTL       |
| -------------- | ------------- | --------------- | ----------------------- | --------- |
| Merge Queue    | GPU-bound     | High/Medium/Low | High can preempt Medium | 24 hours  |
| Eval Queue     | GPU/CPU-bound | Medium          | No                      | 48 hours  |
| Artifact Queue | I/O-bound     | Low             | No                      | 1 hour    |
| Tracker Queue  | Memory-bound  | Lowest          | No                      | 5 minutes |

Messages carry `trace_id`, `experiment_id`, and `job_type`. Dead-letter queues capture failed jobs for manual inspection.

## Merge Job Sequence Diagram

```mermaid theme={null}
sequenceDiagram
    participant API as API Server
    participant SCHED as Scheduler
    participant MQ as Merge Queue
    participant GW as GPU Worker
    participant TE as TensorEngine
    participant ME as MergeEngine
    participant AW as Artifact Writer
    participant AQ as Artifact Queue
    participant TP as Tracker Publisher
    participant TQ as Tracker Queue
    participant ET as ExperimentTracker
    API->>SCHED: submit_merge_job(parent_a, parent_b, strategy)
    SCHED->>MQ: enqueue(job)
    MQ->>GW: dequeue(job)
    GW->>TE: load_tensors(parent_a, parent_b)
    TE-->>GW: TensorDict
    GW->>TE: validate_shapes(a, b)
    TE-->>GW: valid
    GW->>ME: merge(strategy, params)
    ME-->>GW: candidate
    GW->>AQ: enqueue_artifact(candidate)
    AQ->>AW: dequeue(candidate)
    AW->>AW: write_to_store
    AW->>TQ: enqueue_event(artifact_stored)
    GW->>TQ: enqueue_event(merge_complete)
    TQ->>TP: dequeue_batch(events)
    TP->>ET: write_events
    ET-->>TP: ack
    TP-->>TQ: ack
    GW-->>MQ: ack(job)
    MQ-->>SCHED: notify_complete
```

## Event Loop and Concurrency

Each worker runs an asyncio event loop. GPU workers pin one process per GPU to avoid CUDA context switching. Evaluation workers use multiprocessing for backend isolation. CPU workers use thread pools for I/O-bound tasks.

## Failure Handling

Worker crashes trigger scheduler-level retry with exponential backoff. GPU OOM events promote the job to a CPU worker with chunked execution. Tracker Publisher failures buffer events to local disk for replay.

## Traceability Footer

| Spec Reference                                                 | Phase   |
| -------------------------------------------------------------- | ------- |
| [System Architecture](/architecture/system-architecture)       | Phase 1 |
| [Component Architecture](/architecture/component-architecture) | Phase 2 |
| [GPU Orchestration](/architecture/gpu-orchestration)           | Phase 3 |
| [Observability](/architecture/observability)                   | Phase 3 |