> ## Documentation Index
> Fetch the complete documentation index at: https://doc.blueapi.ir/llms.txt
> Use this file to discover all available pages before exploring further.

# Operations Runbook: Startup, Shutdown, and Tasks

> Cold start, warm start, graceful shutdown, forced shutdown, and common operational tasks for EMEP production deployments.

This runbook covers the standard procedures for starting, stopping, and maintaining an EMEP cluster. Follow the startup sequence in order. Shutdown procedures minimize data loss and registry inconsistency.

## Startup Sequence

```mermaid theme={null}
sequenceDiagram
    participant OP as Operator
    participant CFG as Config Loader
    participant DEP as Dependency Check
    participant REG as Registry Connect
    participant WP as Worker Pool
    participant HC as Health Check

    OP->>CFG: Start command
    CFG->>DEP: Validate services
    DEP->>REG: Check registry
    REG->>WP: Initialize workers
    WP->>HC: Run health checks
    HC-->>OP: Ready / Not Ready
```

### Cold Start

1. Load configuration from canonical source (file or vault)
2. Validate all external dependencies: database, object store, message queue, GPU nodes
3. Connect to ModelRegistry and DatasetRegistry; verify schema version
4. Initialize worker pools: MergeEngine, EvaluationEngine, EvolutionEngine
5. Run health checks on every service endpoint
6. Mark cluster READY; expose metrics and API

A cold start assumes no prior state. All caches are empty. The first model load may take longer than usual.

### Warm Start

A warm start skips dependency validation if the operator confirms all dependencies are healthy. It resumes from checkpointed worker pool state. Use warm start after a rolling restart or configuration reload.

1. Load configuration
2. Skip dependency validation (operator flag)
3. Restore worker pool state from snapshot
4. Run abbreviated health checks
5. Mark cluster READY

## Shutdown Procedures

### Graceful Shutdown

1. Reject new experiment submissions via API
2. Wait for active merges and evaluations to complete (configurable timeout, default 300 seconds)
3. Flush ExperimentTracker buffers to persistent store
4. Close registry connections
5. Terminate worker pools
6. Mark cluster STOPPED

### Forced Shutdown

Use only when graceful shutdown would exceed operational limits (e.g., imminent power loss).

1. Immediately cancel all active tasks
2. Flush buffers with best effort
3. Terminate all processes
4. On next startup, the cluster runs a consistency check against the registry and replays incomplete experiments from the last checkpoint

## Common Tasks

| Task                       | Command / Action                    | Verification                                |
| -------------------------- | ----------------------------------- | ------------------------------------------- |
| Check cluster health       | API GET /health or metrics endpoint | All services report healthy                 |
| Reload configuration       | API POST /admin/reload or signal    | Config version increments                   |
| Drain node for maintenance | API POST /admin/drain               | No new tasks scheduled; active tasks finish |
| Flush experiment logs      | API POST /admin/flush               | Log sequence number advances                |
| Rotate signing keys        | Admin CLI key-rotate command        | New key ID appears in registry              |
| Clear model cache          | API DELETE /admin/cache             | Cache hit rate drops to zero                |

## Cross-Links

* [Disaster Recovery](/operations/disaster-recovery)
* [Monitoring](/operations/monitoring)
* [Alerting](/operations/alerting)