flowchart TD
PF["Problem Framing and Metrics"]
DC["Data Collection and Labeling"]
EDA["Exploratory Analysis"]
FR["Feature and Representation"]
MD["Model Development"]
EV["Evaluation and Validation"]
DEP["Deployment"]
MON["Monitoring and Drift Detection"]
PF --> DC
DC --> EDA
EDA --> FR
FR --> MD
MD --> EV
EV --> DEP
DEP --> MON
DC -.->|revise framing| PF
EDA -.->|revise data| DC
FR -.->|revise framing| PF
EV -.->|revise framing| PF
MON -.->|revise framing| PF
6 The AI Development Lifecycle
6.1 1. Introduction
Building a machine learning system is rarely a single act of training a model. It is a sustained engineering and organizational process that begins long before any algorithm is selected and continues long after a model first reaches production. This process is what practitioners call the AI development lifecycle: the sequence of activities that turns a business or scientific question into a working, monitored, and continuously improved intelligent system.
Newcomers often imagine that the hard part of applied machine learning is the modeling, the choice of architecture, the hyperparameter search, the clever loss function. The empirical record says otherwise. Industry surveys and post-mortems of failed deployments consistently point to two earlier stages as the dominant sources of failure: the framing of the problem and the data that feeds it. A model can be technically excellent and still useless if it optimizes the wrong objective, or if the data it learned from does not resemble the data it will eventually see. This chapter therefore treats the lifecycle as a whole, with deliberate emphasis on the stages that matter most and fail most often.
A second theme runs throughout: the lifecycle is iterative, not linear. The clean left-to-right diagrams in textbooks are pedagogical conveniences. Real projects loop backward constantly. Evaluation reveals a labeling error that sends the team back to data collection. Monitoring detects drift that reopens the problem-framing conversation. Understanding these feedback loops is the difference between a project that converges and one that thrashes.
6.2 2. A Map of the Lifecycle
Before examining each stage, it helps to see the whole. The diagram below shows the canonical stages and, crucially, the backward arrows that make the process iterative rather than a one-way pipeline.
Notice that almost every stage can send work back to an earlier one. The arrows from monitoring back to problem framing and from evaluation back to data are not exotic edge cases. They are the normal rhythm of mature practice.
6.3 3. Problem Framing and Success Metrics
6.3.1 3.1 Translating a Goal into a Learnable Task
The first and most consequential stage is also the least technical. Someone, a clinician, a product manager, a policy analyst, has a goal stated in human terms: reduce hospital readmissions, surface relevant search results, detect fraudulent transactions. The task of framing is to convert this goal into a machine learning problem with a precise input, output, and objective.
This translation is where many projects quietly go wrong. Consider a request to “predict which customers will churn.” Framing forces uncomfortable but essential questions. What counts as churn, and over what horizon? Is the prediction actionable, that is, can the business actually intervene? Should the model output a probability, a ranked list, or a binary flag? A model that predicts churn perfectly but too late to retain anyone has solved the wrong problem.
6.3.2 3.2 Choosing Metrics That Reflect Value
Every model optimizes something, and what it optimizes is almost never identical to what the organization values. Bridging that gap requires distinguishing two kinds of metrics. Offline metrics (accuracy, area under the ROC curve, F1, mean average precision) are computed on held-out data and guide model selection. Online or business metrics (revenue, retention, click-through, clinician trust) are what actually matter and are measured in the real system, often through controlled experiments.
The danger is optimizing an offline proxy that diverges from the online goal. Accuracy is famously misleading under class imbalance: a fraud detector that labels everything “not fraud” can be 99.9 percent accurate and worthless. Good framing selects metrics that are robust to the data’s structure, weighs the asymmetric costs of false positives against false negatives, and, where stakes are high, considers fairness and calibration alongside raw discrimination. Goodhart’s law looms over this entire stage: when a measure becomes a target, it ceases to be a good measure, so teams must watch for the model gaming its proxy.
6.4 4. Data Collection and Labeling
6.4.1 4.1 Why Data Dominates Outcomes
If problem framing sets the direction, data determines how far the project can travel. The prevailing wisdom that more data beats cleverer algorithms is only half right. What beats cleverer algorithms is better data: representative, correctly labeled, and matched to deployment conditions. The movement often called data-centric AI, associated with Andrew Ng among others, argues that for many mature problems the highest-leverage work is improving data quality rather than tuning models.
6.4.2 4.2 Sourcing and Representativeness
Data may come from logs, sensors, surveys, public corpora, or third-party vendors. The central risk at this stage is sampling bias: the collected data does not represent the population the model will serve. A facial analysis system trained mostly on one demographic will fail on others. A demand forecaster trained on pre-pandemic data will mislead during a shock. Representativeness is not a property you can recover later through clever modeling; it must be designed into collection.
6.4.3 4.3 Labeling and Annotation
Supervised learning needs labels, and labels are expensive, noisy, and political. Who decides what a “toxic” comment is? How do you reconcile disagreeing annotators? Practical labeling programs invest in clear annotation guidelines, measure inter-annotator agreement (for example with Cohen’s kappa), and treat ambiguous cases as signal about the problem rather than noise to be averaged away. Label leakage, where information unavailable at prediction time sneaks into the features through the labeling process, is a subtle and recurring defect that inflates offline scores and collapses in production. Techniques such as weak supervision and programmatic labeling, exemplified by systems like Snorkel, let teams scale annotation while keeping humans focused on the hard cases.
6.5 5. Exploratory Data Analysis
Before modeling, the data must be understood. Exploratory data analysis (EDA), a discipline traced to John Tukey, is the practice of summarizing, visualizing, and interrogating data to form hypotheses and catch problems. EDA reveals missing values, outliers, skewed distributions, unexpected correlations, and the telltale signatures of data-collection errors.
This stage is diagnostic. A column that is empty for ninety percent of records, a timestamp in the future, a target that correlates suspiciously perfectly with one feature (a hint of leakage): these are the findings that send the team back to earlier stages. Skipping EDA to rush into modeling is a false economy, because the model will faithfully learn whatever pathologies the data contains. Good EDA also informs the next stage by suggesting which transformations and representations the data demands.
6.6 6. Feature and Representation Choices
How data is represented to a model often matters more than which model is chosen. In classical machine learning this stage is feature engineering: constructing informative inputs such as ratios, aggregations, time-since-last-event, or domain-specific encodings. Thoughtful features can let a simple, interpretable model outperform a complex one.
The deep learning era shifted the balance toward representation learning, in which the model learns its own features from raw inputs such as pixels, audio, or text. This does not eliminate the stage so much as relocate it: choices about tokenization, embeddings, normalization, and architecture are representation choices. A critical and easily overlooked concern is training-serving skew, where the feature computation in training differs from that at serving time. A feature scaled with statistics from the training set but recomputed differently in production will silently degrade predictions. Feature stores emerged in part to enforce consistency between the two environments.
6.7 7. Model Development
Only now, well into the lifecycle, does modeling proper begin. The guiding principle here is parsimony: start with a strong, simple baseline (a linear model, a gradient-boosted tree, a heuristic rule) before reaching for anything elaborate. A baseline establishes whether the problem is learnable at all and gives every later, more complex model something to beat.
Model development is itself a loop of training, examining errors, and adjusting. Practitioners manage the bias-variance tradeoff: underfitting models that are too simple to capture the signal, overfitting models that memorize noise. Regularization, cross-validation, and disciplined hyperparameter search are the tools of this stage. Because experiments multiply quickly, experiment tracking, recording each run’s data version, code, parameters, and metrics, is essential for reproducibility. The reproducibility crisis that has touched machine learning research stems largely from neglecting this discipline. Importantly, modeling is the stage practitioners most enjoy and most overinvest in, which is precisely why this chapter places it in proportion: it is one stage among many, and rarely the one that decides success.
6.8 8. Evaluation and Validation
6.8.1 8.1 Honest Estimates of Generalization
Evaluation asks whether the model will perform on data it has never seen. The cardinal rule is strict separation of training, validation, and test data, with the test set touched only once, at the end. Violations of this discipline, tuning on the test set, leaking future information, evaluating on data that overlaps with training, produce optimistic numbers that evaporate in deployment.
Special care is needed for structured data. With time series, random shuffling leaks the future into the past, so evaluation must respect temporal order. With grouped data (multiple records per patient, per user), splits must keep groups intact to avoid leakage. The validation scheme must mirror how the model will actually be used.
6.8.2 8.2 Beyond a Single Aggregate Number
A single headline metric hides as much as it reveals. Mature evaluation slices performance across subgroups to expose disparities, examines the confusion matrix to understand error types, checks calibration so that predicted probabilities mean what they claim, and performs error analysis on individual failures to find patterns. Increasingly, evaluation also includes robustness checks against distribution shift and adversarial inputs, and fairness audits across protected groups. The goal is not a number to celebrate but a defensible judgment about whether the system is safe and useful to deploy.
6.9 9. Deployment
Deployment moves a validated model from a notebook into a system that serves real predictions under real constraints. This is a software and operations problem as much as a machine learning one, and it is where many academically successful models stall.
Deployment patterns vary by need. Batch prediction scores data on a schedule and stores results. Online serving exposes the model behind an API for low-latency requests. Edge deployment pushes the model onto devices for privacy or connectivity reasons. Each pattern imposes constraints on latency, throughput, and resource use that may force a return to model development, for instance to distill or quantize a model that is too slow. Safe rollout practices, shadow deployment (running the new model alongside the old without acting on it), canary releases, and A/B tests, let teams measure real online metrics and contain the blast radius of failures. Reproducibility and versioning of data, code, and model artifacts make rollbacks possible when something goes wrong.
6.10 10. Monitoring and Drift Detection
A deployed model is not finished; it is now exposed to a world that changes. Monitoring is the continuous observation of a live system to ensure it remains healthy. It spans operational signals (latency, error rates, throughput) and, harder but more important, predictive quality.
The central long-run threat is drift. Data drift (also called covariate shift) occurs when the distribution of inputs changes: a new customer segment, a new sensor, a seasonal effect. Concept drift occurs when the relationship between inputs and the target changes: fraud tactics evolve, user preferences shift, an economic regime breaks. Drift is insidious because the model keeps producing confident predictions while silently growing less correct.
Detecting drift is complicated by delayed labels. When ground truth arrives slowly (did this loan default? did this patient recover?), teams cannot wait for it to notice a problem. They therefore monitor proxies: statistical distance between recent and reference input distributions (using tests such as the Kolmogorov-Smirnov test or population stability index), shifts in prediction distributions, and any early-arriving outcome signals. When drift crosses a threshold, the monitoring stage triggers a return to data collection and retraining, closing the largest loop in the lifecycle.
6.11 11. Iteration
The arrows pointing backward in the diagram of Section 2 are the heart of the matter. Iteration is not failure; it is how the system improves. Each pass through the loop should be cheap enough to run often, which is why automation of data pipelines, retraining, evaluation, and deployment pays for itself. The objective is a tight feedback loop in which production reality continuously informs the next version of the model, and in which the team’s understanding of the problem itself matures over time. Systems decay if left static; the only stable state is continuous, disciplined iteration.
6.12 12. Frameworks for the Lifecycle
Several formal frameworks describe this process, each emphasizing different concerns. Understanding their lineage clarifies why modern practice looks the way it does.
6.12.1 12.1 CRISP-DM
The Cross-Industry Standard Process for Data Mining (CRISP-DM), published in 1999, was the first widely adopted lifecycle model. Its six phases (business understanding, data understanding, data preparation, modeling, evaluation, deployment) map closely onto the stages above, and its insistence on starting from business understanding remains sound. CRISP-DM explicitly depicts iteration with arrows between phases. Its limitation is that it predates the realities of continuously running, learning systems: it treats deployment as nearly the final step and says little about monitoring, drift, or automated retraining.
6.12.2 12.2 The ML Lifecycle
What is loosely called “the ML lifecycle” extends CRISP-DM to reflect that machine learning models are living artifacts. It adds explicit emphasis on data versioning, experiment tracking, model validation, and, decisively, the monitoring and feedback loop after deployment. Where CRISP-DM ends with a deployed model, the ML lifecycle treats deployment as the beginning of an operational phase that feeds back into the next iteration.
6.12.3 12.3 MLOps
MLOps applies the principles of DevOps, automation, continuous integration and continuous delivery, infrastructure as code, observability, to machine learning systems. It is less a description of stages than a discipline for executing them reliably and repeatedly at scale. MLOps contributes continuous training pipelines, automated testing of data and models, feature stores to prevent training-serving skew, model registries, and production monitoring. Google’s influential analysis of technical debt in machine learning systems made the case that the model code is a small fraction of a real system, surrounded by configuration, data dependencies, and serving infrastructure that dominate maintenance cost. MLOps is the engineering response to that observation. The three frameworks are best understood as complementary layers: CRISP-DM names the phases, the ML lifecycle adds the post-deployment loop, and MLOps provides the automation and tooling that make the loop sustainable.
6.13 13. Where Projects Actually Fail
It is worth stating plainly, because it contradicts the instincts of many newcomers. The most common causes of failure in applied machine learning are not in the modeling stage. They cluster at the beginning and at the data.
Projects fail because the problem was framed to optimize a metric that did not correspond to real value. They fail because the training data was unrepresentative, mislabeled, or contaminated by leakage that inflated offline scores. They fail because no one monitored for drift and the model silently decayed. They fail organizationally when a brilliant prototype never crosses the gap into a maintainable production system. Surveys of the field repeatedly report that a large share of models built never reach production at all, and of those that do, many degrade for want of monitoring. The modeling stage, by contrast, is comparatively well-tooled and forgiving: a competent baseline often captures most of the available value.
The practical implication for a graduate practitioner is a reallocation of attention. Spend disproportionate effort on framing the problem correctly and on the data, the collection, the labeling, the exploratory understanding, the representation. Treat modeling as important but bounded. And build, from the start, the monitoring and iteration machinery that lets the system survive contact with a changing world. The lifecycle is a loop, and the loop never truly closes.
6.14 References
Wirth, R., and Hipp, J. (2000). CRISP-DM: Towards a Standard Process Model for Data Mining. Proceedings of the 4th International Conference on the Practical Applications of Knowledge Discovery and Data Mining. https://www.cs.unibo.it/~danilo.montesi/CBD/Beatriz/10.1.1.198.5133.pdf
Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., Chaudhary, V., Young, M., Crespo, J., and Dennison, D. (2015). Hidden Technical Debt in Machine Learning Systems. Advances in Neural Information Processing Systems (NeurIPS) 28. https://papers.nips.cc/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html
Ng, A. (2021). A Chat with Andrew on MLOps: From Model-centric to Data-centric AI. DeepLearning.AI. https://www.deeplearning.ai/the-batch/the-data-centric-ai-movement/
Tukey, J. W. (1977). Exploratory Data Analysis. Addison-Wesley.
Ratner, A., Bach, S. H., Ehrenberg, H., Fries, J., Wu, S., and Re, C. (2017). Snorkel: Rapid Training Data Creation with Weak Supervision. Proceedings of the VLDB Endowment, 11(3). https://www.vldb.org/pvldb/vol11/p269-ratner.pdf
Google Cloud. (2020). MLOps: Continuous Delivery and Automation Pipelines in Machine Learning. https://cloud.google.com/architecture/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning
Breck, E., Cai, S., Nielsen, E., Salib, M., and Sculley, D. (2017). The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction. IEEE International Conference on Big Data. https://research.google/pubs/pub46555/
Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., and Zimmermann, T. (2019). Software Engineering for Machine Learning: A Case Study. IEEE/ACM 41st International Conference on Software Engineering (ICSE-SEIP). https://www.microsoft.com/en-us/research/publication/software-engineering-for-machine-learning-a-case-study/
Gama, J., Zliobaite, I., Bifet, A., Pechenizkiy, M., and Bouchachia, A. (2014). A Survey on Concept Drift Adaptation. ACM Computing Surveys, 46(4). https://dl.acm.org/doi/10.1145/2523813
Paleyes, A., Urma, R.-G., and Lawrence, N. D. (2022). Challenges in Deploying Machine Learning: A Survey of Case Studies. ACM Computing Surveys, 55(6). https://dl.acm.org/doi/10.1145/3533378