Operations Runbook: Startup, Shutdown, and Tasks

Cold start, warm start, graceful shutdown, forced shutdown, and common operational tasks for EMEP production deployments.

This runbook covers the standard procedures for starting, stopping, and maintaining an EMEP cluster. Follow the startup sequence in order. Shutdown procedures minimize data loss and registry inconsistency.

Startup Sequence

Cold Start

1. Load configuration from canonical source (file or vault)
2. Validate all external dependencies: database, object store, message queue, GPU nodes
3. Connect to ModelRegistry and DatasetRegistry; verify schema version
4. Initialize worker pools: MergeEngine, EvaluationEngine, EvolutionEngine
5. Run health checks on every service endpoint
6. Mark cluster READY; expose metrics and API

A cold start assumes no prior state. All caches are empty. The first model load may take longer than usual.

Warm Start

A warm start skips dependency validation if the operator confirms all dependencies are healthy. It resumes from checkpointed worker pool state. Use warm start after a rolling restart or configuration reload.

1. Load configuration
2. Skip dependency validation (operator flag)
3. Restore worker pool state from snapshot
4. Run abbreviated health checks
5. Mark cluster READY

Shutdown Procedures

Graceful Shutdown

1. Reject new experiment submissions via API
2. Wait for active merges and evaluations to complete (configurable timeout, default 300 seconds)
3. Flush ExperimentTracker buffers to persistent store
4. Close registry connections
5. Terminate worker pools
6. Mark cluster STOPPED

Forced Shutdown

Use only when graceful shutdown would exceed operational limits (e.g., imminent power loss).

1. Immediately cancel all active tasks
2. Flush buffers with best effort
3. Terminate all processes
4. On next startup, the cluster runs a consistency check against the registry and replays incomplete experiments from the last checkpoint

Common Tasks

Task	Command / Action	Verification
Check cluster health	API GET /health or metrics endpoint	All services report healthy
Reload configuration	API POST /admin/reload or signal	Config version increments
Drain node for maintenance	API POST /admin/drain	No new tasks scheduled; active tasks finish
Flush experiment logs	API POST /admin/flush	Log sequence number advances
Rotate signing keys	Admin CLI key-rotate command	New key ID appears in registry
Clear model cache	API DELETE /admin/cache	Cache hit rate drops to zero

Cross-Links

• Disaster Recovery
• Monitoring
• Alerting