EMEP Storage Architecture: Tiers, Layout, and Retention

Storage tiers for EMEP: hot NVMe, warm object, cold archive. Layout for model artifacts, datasets, experiment DB, metrics, logs, and signing keys with per-tier retention.

EMEP storage is organized into three tiers with distinct latency, cost, and retention characteristics. This page specifies the layout for model artifacts, datasets, experiment databases, metrics stores, logs, and signing keys. A tiered flowchart illustrates data movement between tiers.

Tiered Storage Flowchart

Hot Tier: NVMe SSD

Purpose: Sub-millisecond latency for active workloads.

Contents:

• Active model artifacts (checkpoints, merged candidates)
• Active dataset splits (Optimization Set, Validation Set)
• Experiment database primary (PostgreSQL or equivalent)
• Recent metrics (last 30 days)
• Recent logs (last 7 days)

Retention: Evicted to warm tier after idle period. Model artifacts: 30 days. Dataset splits: 7 days post-evaluation. Logs: 7 days.

Capacity planning: Sized for 2x the largest expected single experiment working set.

Warm Tier: Object Store

Purpose: Cost-effective storage for completed experiments and historical data.

Contents:

• Completed experiment artifacts (merged models, evaluation outputs)
• Full dataset archives (all splits, immutable)
• Experiment database replicas (read replicas, analytics)
• Historical metrics (30 days to 1 year)
• Historical logs (7 days to 90 days)

Retention: 1 year for artifacts and datasets. 2 years for database replicas. 90 days for logs.

Access pattern: Batch analytics, audit queries, model rollback.

Cold Tier: Archive

Purpose: Long-term retention for compliance, reproducibility, and key backup.

Contents:

• Deprecated model versions (post-DEPRECATED lifecycle state)
• Obsolete datasets (superseded by newer versions)
• Archived experiment records (post-ARCHIVED state)
• Aggregated metrics snapshots (yearly rollups)
• Log archives (compressed, indexed)
• Signing key backups (encrypted, offline)

Retention: 7 years for signing keys and experiment records. 3 years for model artifacts. 1 year for logs.

Access pattern: Rare. Retrieval SLA: 24 hours.

Data Layout

Data Type	Hot Path	Warm Path	Cold Path
Model artifacts	/hot/models/{model_id}/	/warm/models/{model_id}/	/cold/models/{model_id}/
Datasets	/hot/datasets/{dataset_id}/splits/	/warm/datasets/{dataset_id}/	/cold/datasets/{dataset_id}/
Experiment DB	/hot/experiments/db/	/warm/experiments/replicas/	/cold/experiments/archives/
Metrics	/hot/metrics/{experiment_id}/	/warm/metrics/{year}/{month}/	/cold/metrics/snapshots/
Logs	/hot/logs/{date}/	/warm/logs/{week}/	/cold/logs/{year}/
Signing keys	HSM only	Encrypted backup	Offline vault

Tier Movement Policy

Data moves between tiers based on lifecycle state and access patterns.

• Model artifacts transition from hot to warm when state reaches EVALUATED.
• Model artifacts transition from warm to cold when state reaches DEPRECATED.
• Dataset splits move to warm 7 days after last benchmark reference.
• Logs are compressed and moved weekly.
• Signing key backups are immutable and never leave cold tier.

Traceability Footer

Spec Reference	Phase
System Architecture	Phase 1
Component Architecture	Phase 2
Artifact Registry	Phase 2
Backup Strategy	Phase 3