An AI feature becomes a production system when people depend on its output to complete real work. At that point, the important questions concern the entire operating path: what enters the system, how the answer is evaluated, what happens when confidence is low, who reviews exceptions, and how the team knows the service is creating value.
A strong rollout makes those answers explicit before usage expands. The goal is a service that produces a defined result, exposes its operating condition, and gives the team direct control over releases and recovery.
Define the unit of work
Begin with one action the system will help someone complete. Examples include classifying an incoming request, drafting a response from approved source material, extracting fields from a document, or recommending the next step in a known workflow.
Write down five details:
- The input the system receives.
- The output it must produce.
- The person or software that uses the output.
- The decision or action that follows.
- The condition that marks the work as complete.
This creates a measurable unit of work. “Generate a customer reply that an agent can approve and send” can be evaluated. “Use AI for customer service” is too broad to evaluate.
The definition should also identify the boundaries of the feature. A system that drafts an answer requires controls around source use, factual accuracy, and reviewer approval. A system that sends an answer requires additional controls around authorization, delivery, and recovery. The release design should match the authority the software receives.
Build a representative evaluation set
Quality needs a stable reference point. Collect examples that represent the work the system will encounter, including routine cases, incomplete inputs, ambiguous requests, and cases that require escalation.
For each example, record the expected result and the reason it is acceptable. A useful evaluation rubric can include:
- Factual accuracy against the approved source.
- Completeness of required information.
- Correct classification or routing.
- Compliance with business rules.
- Appropriate escalation when the system lacks enough information.
- A format the next person or system can use.
Separate release-blocking failures from issues that can be corrected during normal iteration. An invented policy, a missed safety condition, or an incorrect financial value may block a release. A wording preference may belong in the improvement backlog.
Run the same evaluation set against every material change to the model, instructions, retrieval source, tools, or output format. Store the results with the release. The stored evidence makes quality review a repeatable engineering control.
Design the complete operating path
The production architecture should cover every step between a submitted request and a completed result. That usually includes input validation, source retrieval, model execution, output validation, business-rule checks, storage, and delivery to the next person or system.
Plan explicitly for conditions that occur during normal operation:
- The request is missing required information.
- A source document is unavailable or outdated.
- The provider times out or returns an error.
- The output does not match the required structure.
- Confidence falls below the threshold for automated handling.
- A person rejects or corrects the result.
Each condition needs a defined response. The system can ask for missing information, retry a safe operation, route the item for review, or preserve the request for later processing. The important part is that the work remains visible and recoverable.
Use structured outputs when another system consumes the result. Validate every required field before the result advances. Preserve the input, source references, configuration version, output, validation result, and final disposition in an audit record. That record supports debugging, quality review, and future evaluation.
Release in controlled stages
A production release can expand in stages while the team gathers evidence from real use.
Start by running the system against live-shaped inputs without allowing the result to control the workflow. Compare its output with the current process and add newly discovered cases to the evaluation set.
Next, release the feature to a small internal group or a limited portion of traffic. Keep human approval in the path for decisions with material consequences. Record corrections and reasons so the team can distinguish isolated wording changes from recurring system failures.
Expand usage when the evidence meets the release criteria. The criteria should state the required quality score, the maximum blocking-error rate, acceptable latency, the expected review load, and the cost range for each completed unit of work.
Keep releases reversible. Version the instructions, retrieval configuration, tools, and model choice together. Use a feature flag or routing control that can return traffic to the previous known configuration. Confirm the rollback path during release testing so recovery is an exercised procedure.
Monitor the result and the system
Infrastructure metrics show whether the service is running. Outcome metrics show whether it is useful. Production monitoring needs both.
Track operating measures such as:
- Request volume and completion rate.
- Median and high-percentile latency.
- Provider, retrieval, validation, and application errors.
- Retry volume and queue depth.
- Human-review and escalation rates.
- Acceptance, correction, and rejection rates.
- Quality results from sampled completed work.
Break errors into categories that lead to action. Route provider timeouts, missing sources, malformed outputs, incorrect answers, and policy violations to the owner responsible for each condition. The category totals show the pattern the team needs to address.
Review the final business result as well. For an intake classifier, that may be correct routing and time to assignment. For a drafting tool, it may be approval rate and time saved per completed response. For document extraction, it may be field accuracy and the number of records completed without correction.
Measure cost per completed result
Token cost is one part of production cost. The complete calculation can include model usage, retrieval services, storage, infrastructure, monitoring, and human review.
Calculate the cost around the unit of work defined at the beginning:
Cost per useful outcome = total operating and review cost / accepted completed results
This measure exposes expensive retries, long inputs, unnecessary context, high rejection rates, and workflows that require extensive manual correction. It also connects technical changes to the business case. A lower model price has limited value when the resulting output creates additional review work.
Track the cost distribution and the overall mean. A small group of unusually expensive requests can identify oversized documents, repeated tool calls, retrieval problems, or abuse. Add budgets and limits where they protect the service while preserving the intended customer experience.
Assign operating ownership
A dependable system has named owners for the business result, the technical service, and the quality standard.
The business owner decides whether the feature is producing the intended value. The engineering owner handles reliability, security, releases, and incidents. A domain owner defines acceptable output and reviews cases that require subject knowledge. One person may hold several roles in a small organization. The responsibilities remain explicit in every team structure.
Set a review schedule before launch. Daily review may be appropriate during an early release. A stable service may move to weekly quality sampling and monthly cost review. Material changes should trigger the full evaluation and release process regardless of the regular schedule.
Production readiness review
Before expanding access, the team should be able to answer these questions with current evidence:
- What exact unit of work does the system complete?
- Which evaluation cases represent expected production use?
- Which failures block a release?
- Where do low-confidence and exception cases go?
- Which configuration is running in production?
- How can traffic return to the previous configuration?
- Which quality, reliability, and outcome measures are visible?
- What does each accepted result cost?
- Who owns business performance, technical operation, and quality review?
- When will the evidence be reviewed again?
Production readiness is the ability to operate the feature with evidence and control. A defined unit of work, repeatable evaluation, observable execution, reversible releases, visible costs, and named ownership give the team a system it can improve with confidence.