Implementieren Sie die Versionskontrolle von Datensätzen mithilfe von MLOps.

Option A — DVC (Data Version Control) for file-based datasets

Best when your training data is stored as files (CSV/Parquet/images/audio), you already use Git, and you want a developer-friendly workflow. DVC keeps lightweight pointers in Git while storing heavy files in remote storage (S3/GCS/Azure Blob/NFS).

• Great for: data science teams, reproducible experiments, pipeline stages, dataset registries.
• Watch out for: organizational scaling if you need branching over massive shared object stores with many concurrent users.

Option B — lakeFS for Git-like branching on object storage

Best when you have a data lake in object storage and need branch/commit semantics for large-scale datasets without copying everything. Great for isolation, collaboration, and audit trails across teams.

• Great for: data engineering + ML teams sharing large datasets; regulated audit trails; multiple concurrent experiments.
• Watch out for: operational setup and governance decisions (who can branch, merge rules, retention).

Option C — Delta Lake / Apache Iceberg (table versioning & time travel)

Best when your “dataset” is really a set of tables in a lakehouse and you need ACID guarantees, schema evolution, and time travel. Ideal for structured data pipelines feeding feature generation.

• Great for: analytics-grade tables, ETL/ELT pipelines, consistent feature computation.
• Watch out for: mixing unstructured training files and table snapshots—define your dataset ID strategy early.

Option D — Experiment artifact systems (W&B / MLflow artifacts) for “what trained what”

Best for recording the exact dataset reference (URI, commit, hash) used by each run, plus metrics, confusion matrices, and evaluation reports. This isn’t a replacement for data lake versioning—but it closes the loop between data and models.

Core components

• Versioned storage: object store or lakehouse where snapshots/commits live.
• Metadata registry: dataset versions, owners, descriptions, lineage, retention policy.
• Pipeline engine: orchestrates ingestion → validation → transformation → snapshot release.
• Quality gates: schema checks, leakage checks, drift checks, and fairness/coverage checks.
• Experiment tracking: every run logs the dataset version ID + code version + parameters.
• Release gates: promotion rules to move from “candidate data” to “blessed training data.”

A practical “single source of truth” rule

Never train from “latest”. Train from an explicit dataset version (tag/commit/hash). If you want “latest”, define it as a tag that moves only through an approval pipeline.

1) Initialize Git + DVC

git init
dvc init

2) Add your dataset (raw or curated)

dvc add data/train
git add data/train.dvc .gitignore
git commit -m "Track training dataset with DVC"

3) Configure a remote storage (example: S3)

dvc remote add -d storage s3://YOUR-BUCKET/path/to/dvc-cache
git add .dvc/config
git commit -m "Configure DVC remote"

4) Push the data to remote

dvc push

5) Tag a dataset release (recommended)

Use semantic versions if your team likes them (v1.0.0), or a timestamped release (data-2026-01-14). The important part: make it stable.

git tag -a data-v1.0.0 -m "Blessed training dataset v1.0.0"
git push --tags

6) Reproduce the dataset on any machine

git clone YOUR_REPO_URL
cd YOUR_REPO
git checkout data-v1.0.0
dvc pull

7) Branch to experiment safely

git checkout -b exp/new-cleaning
# adjust cleaning pipeline / filters
dvc add data/train
git add data/train.dvc
git commit -m "New cleaning rules for training data"
dvc push

What to log for every training run

• dataset_version_id (Git tag, DVC data hash, lakeFS commit, Delta/Iceberg snapshot ID)
• dataset_uri (storage path / table name / branch)
• code_version (Git commit SHA)
• feature_pipeline_version (if features are computed separately)
• evaluation_report_version (metrics artifact reference)

Minimal example (pseudo-code for your training script)

# read from your CI job, config, or environment
DATASET_VERSION = os.getenv("DATASET_VERSION_ID")
CODE_VERSION = os.getenv("GIT_SHA")

# log to your experiment tracker (MLflow/W&B/Neptune/etc.)
log_param("dataset_version_id", DATASET_VERSION)
log_param("code_version", CODE_VERSION)

Deployment rule that prevents “mystery models”

Only allow promotion to production when a model artifact includes: dataset_version_id + code_version + evaluation report. This is a simple control that dramatically improves governance and debugging.

Recommended quality gates before creating a new dataset version

• Schema validation: types, required fields, category constraints.
• Leakage checks: look-ahead fields, target leakage proxies, duplicate rows across splits.
• Coverage checks: class balance, segment coverage, missingness by segment.
• Statistical drift checks: compare new data vs last “blessed” version (PSI/KL/KS, embeddings for text).
• PII & security checks: detect sensitive fields, enforce access rules, log usage.

Audit trail essentials

• Who approved the dataset release (and when).
• What changed (diff summary, affected tables/partitions/files).
• Which models were trained on it (reverse lineage).
• How long the version is retained (retention policy).

If you operate in regulated environments or want “auditability by design”, see our Compliance & Legal Tech services.

Practical retention strategy

• Keep forever: versions used in production + versions tied to externally audited decisions.
• Keep 30–90 days: experimental branches and intermediate builds.
• Expire automatically: orphaned versions not referenced by any active model or branch.
• Compress where possible: parquet, image compression, column pruning, partitioning.

Tip: budget for “quality, not just storage”

The biggest hidden costs usually come from rework (debugging, reruns, broken pipelines) and incident response, not raw storage. Strong versioning + validation reduces those expensive loops.

• Pitfall: training from “latest” → Fix: require an explicit dataset version ID in every training job.
• Pitfall: “data changed” but no one knows why → Fix: add change summaries + approvals for blessed releases.
• Pitfall: labels drift silently → Fix: version labels + guidelines; run label distribution checks.
• Pitfall: splits are regenerated differently → Fix: pin split files or deterministic seeds per version.
• Pitfall: quality checks exist but are optional → Fix: CI gate that blocks version creation when checks fail.
• Pitfall: too many tools, unclear ownership → Fix: define one dataset ID standard and a single “release” pipeline.

1. Define your dataset version ID (tag/commit/hash) and where it will be stored.
2. Select tooling based on data shape (files vs tables) and team collaboration needs.
3. Standardize dataset structure (raw/processed/splits/labels) with a clear convention.
4. Automate dataset build (ingestion → transforms → snapshot) with reproducible steps.
5. Add quality gates (schema, leakage, drift, coverage) before blessing a version.
6. Log dataset version everywhere (experiment tracker, model registry, deployment metadata).
7. Enforce promotion rules so models without dataset IDs can’t reach production.
8. Set retention policies for branches/versions to control cost.
9. Document the dataset (purpose, sources, known limits, PII handling) for safer reuse.
10. Review monthly (what versions exist, what’s expired, what caused incidents, what to improve).