From prototype to production: don't skip any step
A large majority of AI projects stay at the prototype stage. The main reason: the move to production is underestimated. An AI assistant that works in a demo with 5 users and 100 documents is not ready for 500 users and 10,000 documents. It lacks non-regression tests, access rights control, cost monitoring, logging for audit, error handling and a recovery plan. This checklist covers the 25 essential points grouped into 5 categories: quality, security, monitoring, compliance and scalability. It's the guide we use at Digit-AI to validate the move to production of each AI assistant.
The problem
The prototype impresses the executive committee. The POC is validated. The team is enthusiastic. And then comes the question: "When do we deploy it for everyone?" That's where the trouble begins. The gap between a prototype and a production system is immense, and it is rarely anticipated.
The most frequent pitfalls of the move to production:
- The "it works in the demo" syndrome — The prototype was tested on 20 carefully chosen questions. In production, users ask thousands of unforeseen, poorly formulated, out-of-scope or provocative questions. Without exhaustive testing, surprises are guaranteed from the first week.
- The security oversight — The prototype uses a single index without access control. In production, the marketing department intern can access confidential financial data via the AI assistant. The company discovers the problem when an employee mentions information they shouldn't know.
- The absence of monitoring — No one monitors answer quality, API costs or latency. After 3 months, the bill has doubled and 30% of users have abandoned without the team noticing.
- Compliance neglected — The AI Act comes into force and the auditor asks for the traceability logs. The team realizes it has logged nothing. Retroactive compliance is 5 times more costly than an implementation from the start.
The AI solution
The Go Production checklist is organized into 5 categories. Each point is classified: blocking (B) — deployment forbidden without, important (I) — to implement within 4 weeks, recommended (R) — to plan within the quarter.
Quality and Testing (7 points)
[B] Golden Set of 200+ cases validated by the business. [B] Automated non-regression suite (faithfulness > 0.85, relevance > 0.80). [B] Handling of out-of-scope questions (refusal rate > 90%). [I] Weekly human evaluation (20 cases). [I] Daily evaluation pipeline. [R] A/B testing for prompt modifications. [R] User feedback with an improvement loop.
Security and Access (6 points)
[B] ACL filtering synchronized with the directory. [B] Prompt injection protection (input + prompt + output). [B] Authentication and authorization on the API. [I] Rate limiting per user. [I] Anomaly detection (suspicious queries). [R] Quarterly pentest of the AI assistant.
Monitoring, Compliance and Ops (12 points)
[B] Complete logging of each interaction. [B] Cost monitoring with overrun alerts. [B] Latency monitoring (p50/p95/p99) with SLA. [I] Quality dashboard. [I] Log pseudonymization (GDPR). [I] Documented retention policy. [I] LLM provider failover. [R] Auto-scaling. [R] Disaster recovery plan. [R] Operational runbook. [R] User training. [R] Escalation procedure.
Implementation
The move to production is done in three phases over 6 to 12 weeks. Each phase has a clear deliverable and a validation criterion.
Phase 1: Hardening (weeks 1-4)
Implement all the blocking points [B]. Non-regression tests: build the Golden Set, implement the assertions with DeepEval, integrate into the CI/CD. Security: ACL filtering, injection protection, authentication. Logging: structured logs, pseudonymization. Monitoring: costs and latency with alerts. Validation criterion: all tests pass, no flaw identified during the internal pentest, dashboard operational.
Phase 2: Progressive deployment (weeks 5-8)
Launch with 50 to 100 pilot users, a single use case. Collect the real metrics (quality, costs, latency, satisfaction). Identify the edge cases not covered by the Golden Set and add them. Implement the important points [I]: human evaluation, daily evaluation pipeline, rate limiting, anomaly detection. Fix the problems before expanding.
Phase 3: Scaling up (weeks 9-12)
Progressively open to all target users in waves of 100. Monitor stability: do the quality metrics remain stable? Does the latency p95 stay under the SLA? Do the costs follow the planned curve? Implement the recommended points [R]: auto-scaling, DR, runbook, training. Plan the quarterly review of the checklist.
Results
Results measured on the AI assistant in production projects that followed this checklist.
FAQ
How long does it take to move an AI assistant to production?
Starting from a working prototype, allow 6 to 12 weeks: 2-3 weeks for testing and evaluation, 2-3 weeks for security and compliance, 1-2 weeks for monitoring and observability, and 1-2 weeks for deployment and scaling. This timeline assumes the document base is already indexed and the cloud infrastructure is in place.
Can you deploy an AI assistant without going through the whole checklist?
Some points are non-negotiable: non-regression tests (minimum 100 cases), access rights filtering, protection against prompt injection, quality and cost monitoring, and interaction logging. The other points (cost optimization, scalability, DR) can be implemented in the 4 weeks following the launch if you start with a limited scope (100 users max).
What is the cost of an AI assistant in production?
For a RAG assistant serving 500 users with 200 queries/day: LLM API 200 to 500 euros/month, vector store 50 to 200 euros/month, infrastructure 100 to 300 euros/month, LLMOps tools 0 to 100 euros/month (open source self-hosted). Total: 350 to 1100 euros/month. The typical ROI is reached in 2 to 4 months.
Do you need a dedicated team to maintain an AI assistant in production?
Not necessarily a dedicated team, but clearly assigned responsibilities. At a minimum: an ML engineer (0.3 FTE) for monitoring and optimizations, a product owner (0.1 FTE) for the weekly quality review, and a security/compliance lead (0.1 FTE) for the quarterly audits. For an SMB, this represents about 0.5 FTE spread across 2 to 3 people.
For technical profiles
Go Production technical stack
The production stack is structured around 5 technical layers. Here is the reference architecture with the proven tools for each component.
Layer 1: Ingestion and Indexing
- Extraction: Unstructured.io (multi-format) or LlamaParse (complex PDFs)
- Cleaning: custom pipeline (regex + TF-IDF boilerplate detection)
- Chunking: LangChain RecursiveCharacterTextSplitter or LlamaIndex SentenceSplitter
- Embedding: OpenAI text-embedding-3-large or Cohere Embed v3
- Vector store: Qdrant (open source, self-hosted) or Pinecone (managed)
Layer 2: Retrieval and Generation
- Retrieval: hybrid search (vector + BM25) with reranking (Cohere Rerank or cross-encoder)
- LLM: GPT-4o or Claude Sonnet — with failover between the two
- Prompt: versioned in Git, mandatory structured JSON output
- Citations: NLI verification pipeline post-generation
Layer 3: Security — ACL metadata filtering, Guardrails AI, rate limiting, input sanitization
Layer 4: Observability — OpenTelemetry + Langfuse (tracing), Grafana (dashboards), Ragas/DeepEval (evaluation)
Layer 5: CI/CD — GitHub Actions, DeepEval pytest suite, canary deployment, automatic rollback
Comparison of deployment approaches
| Criterion | Full managed (SaaS) | Self-hosted (open source) | Hybrid |
|---|---|---|---|
| Time-to-production | 4-6 weeks | 10-16 weeks | 6-10 weeks |
| Monthly cost (500 users) | 800-2000 euros | 350-800 euros | 500-1200 euros |
| Data control | Limited | Total | Good |
| Maintenance | Minimal (0.1 FTE) | Significant (0.5 FTE) | Moderate (0.3 FTE) |
| Flexibility | Limited | Total | Good |