AI/ML pipeline architecture — data, features, training, registry, serving, monitoring

Why architecture matters here

ML fails on operations, not on algorithms. The model that got 0.94 F1 in the notebook now serves at 0.72 because the feature store diverged from the training pipeline. The model that shipped is now hallucinating because label drift went unnoticed. The regulator asks for lineage and no one can produce it.

The architecture matters because every one of these failures is preventable with the right substrate. A feature store enforces training/serving parity. A model registry provides lineage. Monitoring catches drift. Governance produces audit trail.

With the pieces in place, the model is one small thing inside a large, reliable system.

Why an ML pipeline is not just a data pipeline

A data pipeline has one honest failure mode: it throws. An ML pipeline has that mode too, plus one no exception handler will ever see - every stage succeeds and the output is wrong. Training completes, evaluation passes, serving returns 200s at a p99 of 14 ms, and the model is simply worse than it was last month.

Three structural properties produce this. The pipeline emits two artifacts, not one: a dataset, and a learned function whose correctness is statistical rather than logical, so no unit test fails cleanly on it. It contains a feedback loop - the model's own predictions shape the traffic and the labels, so yesterday's deployment is an input to today's training data. And quality signal arrives late: a chargeback label lands 60 to 90 days after the prediction it grades, so the metric that would catch a bad promotion does not exist at promotion time. The stages below are conventional; the contracts between them are what make the thing survivable.

Ingestion and validation - the first gate

Most silent degradation is upstream data that changed shape without telling anyone. A mobile release stops populating device_locale, the column goes from 4% null to 61% null, and the model keeps scoring and keeps being wrong - nothing errors, because null is a legal value. The defence is a validation stage that runs before anything downstream may read, treating the schema as a contract rather than a description:

• Schema - column presence, dtype, enum domain. New columns warn; removed or retyped columns hard-fail.
• Volume - row count inside a band of the trailing 7-day median. A 40% drop is a broken upstream job, not a quiet Tuesday.
• Distribution - null rate, cardinality, and quantiles per numeric column against the last accepted partition.
• Referential - join keys resolve at the expected rate; 99.5% falling to 88% means a stale dimension table.

The policy per check matters more than the check. Quarantine and proceed on the last good partition is usually right for training data - a stale sample beats a poisoned model. Hard-fail is usually right for feature materialisation, where serving stale features to a live model is worse than an error you can see.

Dataset versioning - what exactly did this model see

Audit, postmortems, and anyone reproducing your number ask the same question: which rows trained this model? "The last 90 days of events" is not an answer, because that table has been rewritten by late-arriving records since.

What makes it answerable is an immutable, addressable dataset identity. Iceberg and Delta expose a snapshot ID per commit, so a run recording events@snapshot=8842119 can re-read the exact bytes months later; where the table format cannot help, a content hash over the sorted input manifest costs one pass and buys the same guarantee.

Lineage is the graph over those identities: raw snapshot -> validated partition -> feature materialisation -> training sample -> model version -> deployment. Record each edge in the stage that creates it rather than reconstructing it later with a crawler, because reconstruction is where lineage systems quietly lie. Make it queryable in both directions - given a model, which data; given a suspect feed, which live models are downstream. When a supplier admits a fortnight of corruption, that reverse query is the difference between rolling back four models and rolling back everything.

The architecture: every piece explained

The top strip is the data-to-model flow. Data lake holds raw and curated data (Iceberg or Delta commonly). Feature store exposes both offline (batch, historical) and online (low-latency) feature views; a shared definition ensures training and serving see the same values. Training runs distributed with experiment tracking; every run logs code, data snapshot, hyperparameters, and metrics. Model registry stores versioned model artifacts with metadata and approval workflow.

The middle row is the serving and observation loop. Serving exposes online (real-time) and batch (offline scoring) inference. Shadow / canary splits traffic to compare new vs current model without user impact. Monitoring tracks input drift, prediction distribution, latency, and business quality (when labels arrive). Feedback loop writes labels back to the data lake for the next training cycle.

The lower rows are the substrate. Governance holds lineage (which data → which model), approval workflow, and audit. Compute plane allocates GPU and CPU with quotas per team. Metadata + orchestration — MLflow, Kubeflow, Airflow — orchestrates DAGs and stores metadata so every model in production is traceable back to its raw data.

Feature computation and the skew problem

Training reads features from a batch store over history; serving computes them from a request payload and an online store in single-digit milliseconds. Two code paths, one definition - and every divergence between them is invisible offline and lethal online.

Skew has three flavours worth naming separately. Definition skew: the batch job counts over calendar days, the online path over a trailing 24-hour window. Time-travel skew, which is label leakage: the training join takes the feature value as of now instead of as of the prediction timestamp, so the model trains on information it will never have at inference. Freshness skew: definitions match, but the online store materialises hourly while training assumed per-event updates.

A feature store collapses the first by making one registered definition serve both paths and the second by making point-in-time joins the default - feature store architecture has the internals. At pipeline altitude there is a cheaper move needing no product: log the feature vector serving actually used next to the prediction, then recompute those rows through the batch path. The distribution of the difference is your skew monitor, and it catches what a shared definition does not.

Training as a pipeline stage - reproducibility in practice

Treated as a stage rather than a notebook, training is a pure function of a manifest. Anything that influences the weights and is not in the manifest is a reproducibility hole.

run:
  code_version:   git:4f2a91c          # commit, not branch
  env_digest:     sha256:9c1e...       # image digest, not :latest
  dataset:        events@snapshot=8842119
  feature_view:   txn_velocity_v3      # registered definition + version
  sample:         2026-04-01..2026-06-30, point_in_time=true
  seed:           1337
  hyperparams:    {depth: 8, lr: 0.05, rounds: 1200}
  hardware:       4x a100-80g          # affects reduction order

Bit-exact determinism is usually not worth its price: non-deterministic GPU reductions and non-associative float addition make two runs of one manifest differ in the last decimal, and deterministic kernels cost throughput. The honest target is metric-level reproducibility - a rerun lands inside a noise band you have measured, by running one manifest three times before trusting any comparison. A 0.3-point offline delta means nothing when seed variance is 0.5. Add periodic checkpoints to object storage and the stage survives preemption; the training pipeline covers the distributed mechanics.

Promotion gates - the registry as a checkpoint

A trained artifact is a candidate, not a model. The registry's value is not storage but the set of predicates that must hold before a version changes stage. Encode them as executable gates rather than a review checklist, so they cannot be skipped by someone in a hurry:

• Offline metric clears the incumbent on the frozen holdout by more than the measured noise band.
• No regression beyond tolerance on any protected slice - region, device class, customer tier. Aggregate lift routinely hides a segment getting worse.
• Prediction distribution sane: calibration error in bound, no collapse to a constant, positive rate near the incumbent's.
• Provenance complete: manifest present, snapshot resolvable, image digest pinned, upstream validation green for every partition in the sample.
• Operational shape passes: artifact loads in the serving runtime, latency and memory inside budget on target hardware.

Slice regression and calibration are the two that catch real incidents, and both are cheap. Stage transitions (candidate -> staging -> production -> archived), approvals, and audit trail belong to the model registry; evaluation design to evaluation frameworks and model calibration.

End-to-end flow

End-to-end: a fraud team registers a new feature — transaction velocity per user last 24h. The feature store materializes it in both offline (nightly batch to Iceberg) and online (streaming to Redis) modes with a shared definition. A training run reads offline features, trains a gradient-boosted model, and logs run metadata. The model artifact is registered as v42 with metrics. Approval workflow requires a data-science lead to sign off. Deployment routes 5% shadow traffic to v42; monitoring shows fraud recall +3% with acceptable latency. Canary promotes to 50%. Monitoring holds. Full promotion to 100%. Two weeks in, monitoring detects input drift on velocity; the feature engineer investigates; a retraining is triggered from the same DAG. Every step has lineage.

Offline evaluation buys a candidate, not confidence

Offline evaluation measures on a fixed dataset; production measures under a distribution the model itself perturbs. The gap is structural, not noise: a recommender scored on logged impressions only ever sees items the old policy chose to show, and a fraud model's offline recall rests on labels that exist because the old model flagged those transactions for review.

So the pipeline needs an online arm, in order. Shadow - mirror live requests to the candidate, discard its output, compare predictions and latency at zero user exposure. Canary - a small real slice watched on guardrails. Then ramp. Blue-green fits swaps where the versions cannot coexist (changed feature schema, different runtime) and buys instant rollback, but for models it is the weakest of the three because it gives no comparison period.

Two things here are model-specific rather than service-generic. Rollback is cheap in artifact terms and expensive in data terms - the reverted model's predictions are already in the feedback loop and will appear in the next training set. And the ramp schedule is bounded below by label latency: going to 100% in an hour on a model whose quality metric materialises in six weeks is deploying blind. See shadow deployment, champion/challenger, ML A/B testing, and model serving.

Monitoring and what should trigger a retrain

Model monitoring has three tiers and teams routinely build only the first. Operational - latency, error rate, feature-fetch timeouts - is what every service already has. Input - per-feature distribution against the training reference, null rates, unseen categoricals, plus prediction-distribution drift as a cheap label-free proxy. Outcome - the real quality metric, available only when labels land, which is the tier that matters and the tier that is always late.

Separate the two drifts, because the remedy differs. Data drift is a shift in the input distribution - new market, new client version, a campaign changing the traffic mix - and the feature-to-label mapping may be intact, in which case retraining is optional. Concept drift is a shift in that mapping, as when fraud tactics adapt, and no volume of fresh inputs helps until you retrain on fresh labels; input monitors catch it only by luck. Detector mechanics live in drift detection.

Triggers come in four kinds - scheduled, drift-threshold, performance-threshold, data-volume - and the pragmatic default is a schedule with drift as early escalation, since schedules are predictable and testable while pure drift triggers fire in storms. Whatever fires it, the retrain must traverse the same gates as a human-initiated release; automating training while automating away the gates is how a bad batch reaches production at 3am. See automated retraining.

Orchestration - DAGs, idempotency, and backfills

Idempotency means a stage keyed on (logical_date, code_version, input_dataset_version, params_hash) writes the same artifact to the same address every time, so a retry is free and a re-run is a cache hit. What it prevents is the append-mode double-write: a task commits rows, times out before reporting success, the scheduler retries, and the training set now holds a duplicated day that nothing will ever flag. Write to a staging location and publish atomically, or upsert on a key - never blind-append.

Backfill is the same machinery aimed at the past, and it is where ML correctness usually breaks. Recomputing January's features with today's code is valid only if the computation is a pure function of data that existed in January. The moment a feature reads a mutable dimension or a "current" aggregate, the backfilled value is one the model would never have seen at serving time, and you have manufactured leakage at scale. Bound backfill concurrency too - fanning 400 days across a shared cluster evicts every online workload on it.

Then the ordinary hygiene: backoff on transient failures only, validation failures classified non-retryable, per-stage resource requests so GPU training does not queue behind a CPU aggregation, and a DAG shaped so a failed branch cannot silently skip the gates in another.

Failure modes that never raise an exception

A good audit walks the things that will not page you and asks which stage is supposed to notice.

None of these turn a DAG red on their own. Each becomes visible only because some stage was handed an explicit expectation to check - which is the whole argument for the substrate. Team practice around it is covered in MLOps architecture.