> ## Documentation Index
> Fetch the complete documentation index at: https://doc.blueapi.ir/llms.txt
> Use this file to discover all available pages before exploring further.

# EMEP Storage Architecture: Tiers, Layout, and Retention

> Storage tiers for EMEP: hot NVMe, warm object, cold archive. Layout for model artifacts, datasets, experiment DB, metrics, logs, and signing keys with per-tier retention.

EMEP storage is organized into three tiers with distinct latency, cost, and retention characteristics. This page specifies the layout for model artifacts, datasets, experiment databases, metrics stores, logs, and signing keys. A tiered flowchart illustrates data movement between tiers.

## Tiered Storage Flowchart

```mermaid theme={null}
flowchart LR
    subgraph HOT["Hot Tier: NVMe SSD"]
        HA[Active Model Artifacts]
        HD[Active Dataset Splits]
        HE[Experiment DB Primary]
        HM[Recent Metrics]
        HL[Recent Logs]
    end
    subgraph WARM["Warm Tier: Object Store"]
        WA[Completed Experiment Artifacts]
        WD[Full Dataset Archives]
        WE[Experiment DB Replicas]
        WM[Historical Metrics]
        WL[Historical Logs]
    end
    subgraph COLD["Cold Tier: Archive"]
        CA[Deprecated Model Versions]
        CD[Obsolete Datasets]
        CE[Archived Experiment Records]
        CM[Aggregated Metrics Snapshots]
        CL[Log Archives]
        CK[Signing Key Backups]
    end
    HA -->|30 days idle| WA
    HD -->|7 days post-eval| WD
    HE -->|real-time replication| WE
    HM -->|30 days| WM
    HL -->|7 days| WL
    WA -->|1 year| CA
    WD -->|1 year| CD
    WE -->|2 years| CE
    WM -->|1 year| CM
    WL -->|90 days| CL
    CK -->|7 years| CK
```

## Hot Tier: NVMe SSD

**Purpose:** Sub-millisecond latency for active workloads.

**Contents:**

* Active model artifacts (checkpoints, merged candidates)
* Active dataset splits (Optimization Set, Validation Set)
* Experiment database primary (PostgreSQL or equivalent)
* Recent metrics (last 30 days)
* Recent logs (last 7 days)

**Retention:** Evicted to warm tier after idle period. Model artifacts: 30 days. Dataset splits: 7 days post-evaluation. Logs: 7 days.

**Capacity planning:** Sized for 2x the largest expected single experiment working set.

## Warm Tier: Object Store

**Purpose:** Cost-effective storage for completed experiments and historical data.

**Contents:**

* Completed experiment artifacts (merged models, evaluation outputs)
* Full dataset archives (all splits, immutable)
* Experiment database replicas (read replicas, analytics)
* Historical metrics (30 days to 1 year)
* Historical logs (7 days to 90 days)

**Retention:** 1 year for artifacts and datasets. 2 years for database replicas. 90 days for logs.

**Access pattern:** Batch analytics, audit queries, model rollback.

## Cold Tier: Archive

**Purpose:** Long-term retention for compliance, reproducibility, and key backup.

**Contents:**

* Deprecated model versions (post-DEPRECATED lifecycle state)
* Obsolete datasets (superseded by newer versions)
* Archived experiment records (post-ARCHIVED state)
* Aggregated metrics snapshots (yearly rollups)
* Log archives (compressed, indexed)
* Signing key backups (encrypted, offline)

**Retention:** 7 years for signing keys and experiment records. 3 years for model artifacts. 1 year for logs.

**Access pattern:** Rare. Retrieval SLA: 24 hours.

## Data Layout

| Data Type       | Hot Path                             | Warm Path                       | Cold Path                      |
| --------------- | ------------------------------------ | ------------------------------- | ------------------------------ |
| Model artifacts | `/hot/models/{model_id}/`            | `/warm/models/{model_id}/`      | `/cold/models/{model_id}/`     |
| Datasets        | `/hot/datasets/{dataset_id}/splits/` | `/warm/datasets/{dataset_id}/`  | `/cold/datasets/{dataset_id}/` |
| Experiment DB   | `/hot/experiments/db/`               | `/warm/experiments/replicas/`   | `/cold/experiments/archives/`  |
| Metrics         | `/hot/metrics/{experiment_id}/`      | `/warm/metrics/{year}/{month}/` | `/cold/metrics/snapshots/`     |
| Logs            | `/hot/logs/{date}/`                  | `/warm/logs/{week}/`            | `/cold/logs/{year}/`           |
| Signing keys    | HSM only                             | Encrypted backup                | Offline vault                  |

## Tier Movement Policy

Data moves between tiers based on lifecycle state and access patterns.

* Model artifacts transition from hot to warm when state reaches EVALUATED.
* Model artifacts transition from warm to cold when state reaches DEPRECATED.
* Dataset splits move to warm 7 days after last benchmark reference.
* Logs are compressed and moved weekly.
* Signing key backups are immutable and never leave cold tier.

## Traceability Footer

| Spec Reference                                                 | Phase   |
| -------------------------------------------------------------- | ------- |
| [System Architecture](/architecture/system-architecture)       | Phase 1 |
| [Component Architecture](/architecture/component-architecture) | Phase 2 |
| [Artifact Registry](/tracking/artifact-registry)               | Phase 2 |
| [Backup Strategy](/operations/backup-strategy)                 | Phase 3 |