COMPEL Certification Body of Knowledge — Module 2.4: Execution Management and Delivery Excellence
Article 3 of 10
Every AI transformation roadmap is ultimately realized — or not — through the delivery of individual AI use cases. The use case is where strategy becomes tangible: a fraud detection model that changes how risk decisions are made, a demand forecasting system that reshapes inventory planning, a document processing pipeline that transforms back-office operations. The COMPEL Certified Specialist (EATP) may not write the code or train the models, but they must understand and manage the use case delivery lifecycle with enough depth to maintain credible oversight, identify risks before they become crises, and ensure that technical delivery aligns with the broader transformation objectives.
This article addresses how individual AI use cases move from concept to production within the Produce stage of the COMPEL lifecycle. It builds on the use case portfolio management principles established in Module 2.3, Article 2: Gap Analysis and Initiative Identification and the broader execution architecture described in Article 1: From Roadmap to Reality — The Execution Challenge.
The Use Case Delivery Lifecycle
The delivery of an AI use case follows a lifecycle that is distinct from traditional software delivery, though it shares surface similarities. Understanding this lifecycle — its stages, its decision points, and its characteristic failure modes — is essential for the EATP.
Stage 1: Use Case Scoping and Requirements
Before development begins, the use case must be precisely scoped. The strategic rationale was established during roadmap design, but the technical and business requirements must be elaborated to a level that enables development to proceed.
Business requirements define the outcome. What decision will this AI system support or automate? What is the current baseline performance? What improvement constitutes success? Who are the end users, and how will the AI output integrate into their workflow? These questions must be answered with specificity. "Improve customer retention" is a strategic objective, not a business requirement. "Predict the probability of customer churn within the next 90 days with at least 75 percent accuracy, and present the prediction with the top three contributing factors to the retention team through their existing Customer Relationship Management (CRM) dashboard" is a business requirement.
Data requirements define the fuel. What data is needed to train and operate the model? Is it available, accessible, and of sufficient quality? What data governance constraints apply — privacy regulations, data residency requirements, consent limitations? Data requirements analysis frequently reveals show-stopping issues: the required data does not exist, exists but is not accessible due to organizational silos, or exists but is of insufficient quality for model training. Identifying these issues during scoping rather than during development saves significant time and resources. The data governance considerations introduced in Module 1.5, Article 7: Data Governance for AI are directly relevant here.
Technical requirements define the approach. What type of model or Artificial Intelligence (AI) approach is appropriate? What are the performance requirements — latency, throughput, accuracy thresholds? What infrastructure is needed for training, serving, and monitoring? What integration points exist with enterprise systems? The EATP does not define these requirements alone but ensures that the technical team's proposed approach is consistent with the organization's infrastructure capabilities, governance requirements, and strategic direction.
Success criteria define the gate. Before any development begins, the EATP ensures that clear, measurable success criteria are documented and agreed upon by both the technical team and the business sponsor. These criteria become the basis for the quality gates described in Article 8: Quality Assurance and Delivery Standards. Without them, "done" becomes a matter of opinion rather than evidence.
Stage 2: Data Preparation and Feature Engineering
Data preparation typically consumes the largest share of effort in AI use case delivery — often 60 to 80 percent of total development time. The EATP must understand this reality and plan accordingly.
Data acquisition and integration. Raw data must be extracted from source systems, integrated across sources where necessary, and loaded into the development environment. This often requires collaboration with data engineering teams, database administrators, and data owners — groups that may have competing priorities and limited availability. The EATP anticipates these resource dependencies during sprint planning and escalates access issues early.
Data quality assessment and remediation. The data must be assessed for completeness, accuracy, consistency, and timeliness. Missing values, duplicate records, inconsistent formats, and stale data are common findings. Remediation may be straightforward (imputation of missing values, standardization of formats) or may reveal fundamental data quality issues that require upstream process changes — a scope expansion that the EATP must manage carefully through the change control process.
Feature engineering. The transformation of raw data into features that models can learn from is both a technical and a domain expertise challenge. Feature engineering benefits from close collaboration between data scientists and business domain experts — the people who understand what the data means, what patterns are meaningful, and what relationships are spurious. The EATP facilitates this collaboration, ensuring that domain experts are available and engaged during the feature engineering phase.
Stage 3: Model Development and Experimentation
Model development is inherently experimental. Unlike traditional software development, where the path from requirements to implementation is largely predictable, Machine Learning (ML) model development involves hypothesis testing, iterative refinement, and genuine uncertainty about whether the required performance level is achievable.
Experimentation management. The technical team typically explores multiple modeling approaches, hyperparameter configurations, and feature sets before converging on a candidate model. The EATP does not manage this experimentation at the task level — that is the technical lead's responsibility — but monitors it at the milestone level. Key questions for the EATP include: Is the team converging toward a viable solution, or are they stuck in an exploration loop? Are the experiments being tracked systematically (through experiment tracking tools and version control), or are results being lost? Is the experimentation timeline consistent with the sprint plan, or does the scope of experimentation suggest that the original timeline was unrealistic?
Performance evaluation. Model performance is evaluated against the success criteria defined during scoping. The EATP must understand the key metrics — accuracy, precision, recall, F1 score, area under the curve, mean absolute error, or whatever metrics are appropriate for the use case — well enough to interpret results and ask informed questions. This does not require deep statistical expertise. It requires understanding what the metrics mean in business terms: "A precision of 0.85 means that 85 percent of the customers flagged as likely to churn will actually churn. The remaining 15 percent will be false positives who will receive retention outreach unnecessarily."
Bias and fairness assessment. Before a model advances beyond development, it must be assessed for bias and fairness — particularly for use cases that affect individuals or groups. This assessment is both a technical exercise (statistical analysis of model performance across demographic groups) and a governance requirement (alignment with the organization's AI ethics framework). The EATP ensures this assessment occurs and that its results are documented for governance review. The ethical foundations established in Module 1.1, Article 10: Ethical Foundations of Enterprise AI and the operationalized ethics practices from Module 1.5, Article 6: AI Ethics Operationalized inform this assessment.
Stage 4: Integration and System Testing
A model that performs well in a development environment must be integrated into the production technology ecosystem and tested in conditions that approximate real-world operation.
Integration development. The model must be connected to production data sources, deployed to the serving infrastructure, and integrated with the consuming application or workflow. This integration work is frequently underestimated in both complexity and duration. Application Programming Interface (API) contracts must be defined. Data pipelines must be built or adapted. Error handling and fallback logic must be implemented. The EATP ensures that integration development is explicitly planned in the sprint schedule, not treated as a minor step between "model is ready" and "model is deployed."
System testing. End-to-end testing verifies that the model performs correctly within the production environment — with real data flows, real latency constraints, and real integration points. System testing should include: functional testing (does the system produce correct outputs?), performance testing (does it meet latency and throughput requirements?), integration testing (does it interact correctly with upstream and downstream systems?), and failure mode testing (what happens when data is missing, when the model service is unavailable, or when outputs fall outside expected ranges?).
User acceptance testing. End users exercise the system in a controlled environment, providing feedback on usability, output interpretability, and workflow integration. User acceptance testing is not a formality — it is a critical validation that the system is fit for purpose from the perspective of the people who will use it daily. The EATP ensures that user acceptance testing is scheduled with sufficient time for feedback incorporation, not compressed into the final days before a deployment deadline.
Stage 5: Deployment and Operationalization
Deployment is not the finish line — it is the transition from project delivery to operational management. The EATP manages this transition carefully, ensuring that the deployed use case is fully operationalized.
Deployment execution. The model is deployed to the production environment following the organization's deployment procedures — which may include change advisory board approval, deployment window scheduling, and rollback planning. The Machine Learning Operations (MLOps) practices introduced in Module 1.4, Article 7: MLOps — From Model to Production and the model governance lifecycle from Module 1.5, Article 8: Model Governance and Lifecycle Management provide the foundational framework.
Monitoring activation. Production monitoring must be activated simultaneously with deployment. This includes model performance monitoring (is the model's accuracy degrading?), data drift monitoring (is the input data distribution changing?), operational monitoring (is the system meeting latency and availability requirements?), and business outcome monitoring (is the model delivering the expected business impact?).
Operational handover. The development team hands operational responsibility to the operations team — whether that is a dedicated MLOps team, the IT operations group, or the CoE. The handover includes runbooks (procedures for common operational scenarios), escalation procedures, and retraining protocols. An incomplete handover is a common source of operational failures, as the operations team encounters scenarios that were never documented or anticipated.
Technical Delivery Management for the Non-Technical EATP
The EATP is not expected to be a data scientist, ML engineer, or data engineer. But they are expected to provide meaningful oversight of technical delivery. This requires a specific set of capabilities.
Understanding Without Doing
The EATP must be able to:
- Read a technical status report and identify whether progress is genuine or whether technical jargon is obscuring a lack of progress
- Ask informed questions about model performance, data quality, and integration complexity without needing to interpret the raw technical details
- Recognize warning signs — extended experimentation without convergence, repeated data quality delays, integration complexity that keeps expanding, testing phases that are compressed to meet deployment deadlines
- Translate technical issues into business language for stakeholders who need to understand impact and make decisions without understanding the technical details
The EATP's Technical Oversight Questions
At each stage of the use case delivery lifecycle, the EATP should be asking:
- Is the team on track to meet the sprint deliverables? If not, what is the root cause, and is it addressable within the sprint or does it require a replanning conversation?
- Are the success criteria still achievable, or has development revealed that the original targets were unrealistic?
- Are there dependency risks — data access delays, infrastructure availability, integration partner responsiveness — that need to be escalated?
- Is the quality of the work product consistent with the standards established for this program, or is quality being sacrificed for speed?
- Are governance requirements being addressed in parallel with development, or are they being deferred to "later"?
Quality Gates and Acceptance Criteria
Quality gates are formal decision points where the EATP and relevant stakeholders assess whether a use case is ready to advance to the next delivery stage. They are the mechanism that prevents incomplete or substandard work from propagating through the pipeline.
Gate Structure
Each quality gate has three components:
Entry criteria define what must be in place before the gate review occurs. For example, the entry criteria for a deployment readiness gate might include: model performance validated against success criteria, integration testing complete with all tests passing, governance risk assessment approved, user acceptance testing complete with sign-off, and operational runbooks documented.
Review process defines who participates in the gate review, what evidence is examined, and how the decision is made. Gate reviews should include the technical lead, the business sponsor, the governance representative, and the EATP. The decision is typically one of three outcomes: pass (proceed to the next stage), conditional pass (proceed with specific remediation actions), or fail (return to the current stage for additional work).
Exit criteria define what is true after the gate is passed — essentially, the commitments and conditions under which the use case advances. Exit criteria often include documented assumptions, known limitations, and accepted risks.
Common Quality Gates in AI Use Case Delivery
- Data readiness gate: Data is acquired, quality-assessed, and deemed sufficient for model development
- Model viability gate: Model has demonstrated acceptable performance on validation data and has passed bias and fairness assessments
- Integration readiness gate: Model is integrated with production systems and has passed system testing
- Deployment readiness gate: All four pillar requirements are satisfied — technology (model and infrastructure), governance (risk assessment and approval), people (training and change management), and process (workflow documentation and procedures)
- Operational readiness gate: Model is deployed, monitoring is active, and operational handover is complete
The relationship between these gates and the broader COMPEL stage gate framework is detailed in Article 8: Quality Assurance and Delivery Standards.
Managing the Portfolio During Execution
During the Produce stage, the EATP manages not a single use case but a portfolio of use cases at various stages of delivery. Some may be in data preparation while others are in deployment. This portfolio perspective requires the EATP to:
- Allocate attention proportionally to risk, not proportionally to activity. A use case that is proceeding smoothly needs less EATP attention than one that is encountering blockers, regardless of its strategic importance.
- Manage resource contention across use cases. Data engineers, ML engineers, and domain experts are shared resources. When multiple use cases need the same resources simultaneously, the EATP must arbitrate based on roadmap priorities and overall program impact.
- Maintain portfolio-level visibility through a status dashboard that shows each use case's current lifecycle stage, key risks, and projected completion date. This dashboard is the EATP's primary tool for portfolio-level decision-making and for communicating program status to the Steering Committee.
Looking Ahead
Article 4, Change Execution — Operationalizing the People Pillar, shifts from the technical delivery lifecycle to the human dimension of execution. While this article addressed how AI use cases are delivered technically, Article 4 addresses how the organizational changes required to make those use cases effective are executed — the training programs, communication campaigns, and resistance management activities that determine whether technically deployed AI actually transforms the organization.
© FlowRidge.io — COMPEL AI Transformation Methodology. All rights reserved.