ArtifactStore for Content-Addressed Model Storage

Specification for the ArtifactStore in EMEP, covering content-addressed storage, tiering, garbage collection, and digital signatures.

The ArtifactStore is the blob storage layer for EMEP. It stores model weights, experiment artifacts, evaluation outputs, and provenance records. Content addressing guarantees integrity and deduplication. This page specifies the storage model, tiering policy, and garbage collection rules.

Content-Addressed Storage

All artifacts are stored by their SHA-256 hash. The store API is:

ArtifactStore.put(data) -> hash
ArtifactStore.get(hash) -> data
ArtifactStore.exists(hash) -> bool
ArtifactStore.verify(hash) -> bool

Content addressing provides:

• Integrity verification: the hash proves the data matches the record
• Deduplication: identical artifacts store once regardless of origin
• Immutability: artifacts cannot be modified without changing their hash
• Referential stability: hashes are permanent identifiers

Artifact Types

Type	Description	Typical Size
model_weights	Serialized model parameters	1-100 GB
experiment_record	JSON experiment metadata	1-100 KB
evaluation_output	Raw benchmark results	10 MB - 1 GB
provenance_chain	Signed provenance records	10-50 KB
genome	JSON genome specification	1-10 KB
event_stream	ExperimentTracker logs	100 KB - 10 MB

Storage Tiers

Tier	Latency	Cost	Contents
Hot	< 10ms	High	Active experiment artifacts, recent models
Warm	< 1s	Medium	Completed experiment outputs, validated models
Cold	< 1min	Low	Archived experiments, deprecated models
Glacier	Hours	Very Low	Long-term audit retention

Tiering is automatic based on access patterns and artifact age. Active experiment artifacts stay in hot storage. Artifacts from COMPLETED experiments move to warm after 7 days. ARCHIVED experiments move to cold after 30 days. Audit-mandated artifacts move to glacier after 90 days.

Garbage Collection

Artifacts are eligible for garbage collection when:

• No ModelRegistry record references them
• No ExperimentTracker record references them
• They are not part of a signed provenance chain
• Retention policy allows deletion

The garbage collector runs weekly. It scans for unreferenced artifacts and moves them to a quarantine zone for 30 days before permanent deletion. Quarantined artifacts can be restored by hash.

Digital Signatures

Critical artifacts (model weights, provenance chains, release packages) are signed at upload time. Signatures are stored alongside the artifact and verified on retrieval.

signature = sign(private_key, hash)
verify(public_key, hash, signature) -> bool

See Artifact Signing for key management and verification protocols.

API Operations

ArtifactStore.put(data, artifact_type, metadata) -> hash
ArtifactStore.get(hash, tier_preference) -> data
ArtifactStore.get_metadata(hash) -> ArtifactMetadata
ArtifactStore.delete(hash) -> bool  # soft delete to quarantine
ArtifactStore.restore(hash) -> bool  # from quarantine
ArtifactStore.get_referencing_records(hash) -> List[RecordRef]

Integration

Component	Stored Artifacts
ModelLoader	Retrieves model weights by hash
ExperimentTracker	Stores event streams and records
EvaluationEngine	Stores raw evaluation outputs
ModelRegistry	References weight archives by hash
Provenance system	Stores signed provenance chains

Garbage collection does not delete artifacts referenced by active experiments, registered models, or signed provenance chains. An artifact with any live reference is retained regardless of age.

Content addressing means that uploading the same artifact twice returns the same hash. The ArtifactStore detects duplicates and stores only one copy. This saves storage when multiple experiments use the same parent model.