DCF Research

GenAI Proof of Concept: Scope, Cost, and Exit Criteria

By DCF ResearchEdited by Peter KorpakPublished · Updated

A GenAI proof of concept should answer one question: whether to fund this use case for production. To answer it, the proof of concept needs a bounded scope, real data access from week one, a test set and metrics agreed before the build, a cost-per-query estimate, and written go or no-go criteria. Without those it is a demo, and public evidence suggests that many demos never reach production. This guide is part of our AI consulting firms research.

What should a GenAI proof of concept prove?

A proof of concept should prove five things on your own data: quality meets the bar, data access works the way it would in production, unit cost fits the budget, the design can pass security and risk review, and the intended users can use it in their workflow. It should not try to build the product.

  1. Quality. Output meets agreed thresholds on a held-out test set built from real tasks.
  2. Data. The data is available, permitted for this use, and good enough, and the access path would work in production.
  3. Cost. Cost per query or transaction, multiplied by forecast volume, fits the budget.
  4. Risk. The design can pass security, privacy, and legal review, and known GenAI failure modes have been tested.
  5. Fit. The workflow, the users, and a named production owner exist. If you cannot name the owner or the use case yet, start with AI strategy or an assessment.

The terms are often blurred in proposals, so check which stage you are buying:

Proof of conceptPilotProduction build
QuestionCan this work on our data, at a cost and risk we accept?Does it work with real users in a live setting?Can we run it reliably at full scale?
UsersA few testers, or noneA defined group of real usersAll intended users
DataA representative sample, with the production access path testedLive data for the pilot scopeFull production data
OutputPrototype, evaluation report, production planMeasured user outcomes and operating dataSupported, monitored system
Decision it informsFund, re-scope, or stopExpand, adjust, or stopOperate and improve

If a firm calls a long, large engagement a proof of concept, check whether it is a pilot or a build; the cost and exit criteria should match the stage.

What should the scope and statement of work include?

Name the use case, users, data, environment, test set, metrics and thresholds, deliverables, schedule, and exit criteria. Anything left out becomes a change order or an argument at the end.

ElementWhat to write
Use case and usersOne workflow, one user group, what the system will and will not do
DataSources, owner, sensitivity class, sample versus full set, approvals, and the date access is granted
EnvironmentWhere it runs (your cloud or the firm's), which model providers, data residency, and what leaves your boundary
Test set and metricsA question or task set built from real work and labeled by your experts, with metrics and thresholds
BaselineHow your current process performs on the same set
Schedule and dependenciesMilestones, plus your obligations: data by week one, experts available for review
DeliverablesThe list below, each with an acceptance test
Exit criteriaWritten go and no-go rules, and the decision date
Cost and pass-throughFee, expenses, model and cloud usage, and who pays for each
IP and data termsOwnership of code, prompts, and evaluation sets; no training on your data; deletion on completion
Production optionA pre-agreed rate card and team for the follow-on build, with no obligation to buy it

What data access do I need before the proof of concept starts?

Get a named data owner, a sample extract, permission to use the data with the chosen model provider, and the security approval path confirmed in week one. Data problems rank among the leading causes of AI project failure in the public evidence, and an early audit finds them before build money is spent.

Gartner reported in February 2025 that 63% of organizations either do not have or are unsure whether they have the right data management practices for AI (a July 2024 survey of 1,203 data management leaders), and predicted that through 2026 organizations will abandon 60% of AI projects unsupported by AI-ready data. In RAND's 2024 interviews, 30 of 50 industry practitioners described persistent data quality problems.

Confirm before kickoff:

  • A named data owner and a sample extract, with fields, volume, and formats.
  • Classification and approvals: personal, confidential, or regulated data, and whether a model provider may process it and on what retention terms.
  • Where processing happens and whether data leaves your boundary.
  • A production access path (service accounts, network routes), tested early rather than assumed.
  • A split between data used to build and a held-out test set the firm does not tune against.
  • Subject-matter experts with scheduled time to label and review.

Make a data audit the first deliverable: what exists, how much, its quality on a sample, the gaps, and the access status.

How do I choose evaluation metrics for a GenAI proof of concept?

Choose metrics that match the task, set thresholds from the current process before building, and keep a held-out test set. Anthropic's evaluation guidance recommends evals that mirror your real task distribution, automated grading where possible, and more automatically graded questions over fewer hand-graded ones.

Use caseQuality metricsOperational metrics
Document question answering (see RAG implementation consulting)Context recall and context precision (did retrieval find the right passages and rank them well); faithfulness (the share of answer claims supported by the retrieved context); answer correctness against reference answers; citation accuracy; correct refusal when the corpus has no answerMedian and slow-tail latency, cost per query
Extraction or classificationPrecision, recall, and F1 per field or class against a labeled sampleThroughput, cost per document
Drafting or summarizationExpert rubric scores for accuracy, completeness, and tone; time saved against the baselineLatency, cost per output
Agents that call toolsTask success rate, tool-call error rate, unsafe-action rateSteps and tokens per task, cost per task

The retrieval and faithfulness definitions follow the Ragas documentation. Set thresholds from the baseline: measure how your people perform on the same test set today, then agree what good enough means before the build starts. Write them as testable statements, for example that experts judge answers correct on an agreed share of the test set and that unsupported claims stay below an agreed rate.

Put security and misuse tests in the same plan. The OWASP Top 10 for LLM Applications (2025) includes prompt injection, sensitive information disclosure, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation, and unbounded consumption. NIST's Generative AI Profile (AI 600-1) is a companion to its AI Risk Management Framework. Ask the firm to map its test plan to both.

Building the test set takes expert time. Estimate: 300 test questions × 10 minutes of expert time to write and verify = 3,000 minutes = 50 hours; at $100 an hour of internal cost, about $5,000.

How long should a GenAI proof of concept take?

Plan for about four to eight weeks: shorter for a narrow use case on clean data, longer for unstructured documents, several source systems, or strict reviews. This is a planning assumption, not a benchmark. Two AWS Marketplace listings for GenAI proofs of concept, from Colibri Digital and Flatiron Software, describe an 8-week engagement.

WeekActivityOutput
1Kickoff; confirm data access and approvals; build the baseline and test set; fix thresholdsSigned scope and thresholds; data audit started
2 to 3Data preparation; first end-to-end version; first evaluation runWorking baseline; first metrics
4 to 5Iterate on retrieval, prompts, and model choice; security tests; cost profilingMetrics against thresholds; risk findings; cost per query
6Final evaluation on the held-out set; readout; production planEvaluation report, recommendation, and plan

This is a 6-week plan. Compress it to 4 weeks by narrowing scope, or extend it to 8 for multi-source data or heavier review. Fix the decision date in the statement of work.

What does a GenAI proof of concept cost?

Estimate hours times a blended rate, then add usage. With the planning rates in AI consulting pricing, an engineering-led proof of concept is an estimated $48,000 to $72,000 for a lean 4-week scope, $90,000 to $135,000 for a standard 6-week scope, and $168,000 to $252,000 for an extended 8-week scope. These are estimates under stated assumptions, not quotes.

ScopeTeam assumptionHoursRate bandEstimated fee
Lean, 4 weeks2 people2 × 40 × 4 = 320$150 to $225 an hour$48,000 to $72,000
Standard, 6 weeks2.5 people (tech lead 0.5, AI engineer 1, data engineer 0.5, engagement lead 0.5)2.5 × 40 × 6 = 600$150 to $225 an hour$90,000 to $135,000
Extended, 8 weeks3.5 people3.5 × 40 × 8 = 1,120$150 to $225 an hour$168,000 to $252,000

The rate band is an estimate based on GSA CALC+ ceiling rates (the maximum hourly rates on US federal contracts) for data scientist and AI titles, retrieved 23 October 2026; commercial rates can be lower or higher.

In this example, model usage is small during the proof of concept and larger at scale. Estimate: 10,000 evaluation runs at 4,000 input and 500 output tokens, priced at $2 and $10 per million tokens, cost about 10,000 × $0.013 = $130. That is a mid-tier list price on the Anthropic and OpenAI pricing pages (as of October 2026); Google lists a mid-tier model at $2 input and $12 output. At an assumed 300,000 queries a month, the same profile costs about $3,900 a month. Ask for cost per query as a proof-of-concept output and compare it with the production budget.

Your own time is a cost too: data access, labeling, and review. Budget it, and ask whether vendor funding applies. The Colibri Digital and Flatiron Software listings say that completing the proof of concept may qualify for AWS funding, typically up to 10% of costs capped at $25,000. That is the sellers' statement, eligibility is not guaranteed, and funding ties the work to one platform.

Which deliverables should I require?

Require seven deliverables, each with an acceptance test:

  1. Scope and success criteria, signed in week one: use case, users, thresholds, baseline, exit criteria.
  2. Data audit report: sources, volume, quality on a sample, gaps, access status.
  3. Working prototype in your environment, with source code in your repository and setup and test instructions.
  4. Evaluation report: results against thresholds on the held-out set, error analysis, comparison with the baseline and with more than one model or design option, and cost per query.
  5. Security and risk assessment mapped to the OWASP and NIST references above.
  6. Production plan: architecture, team, timeline, run-cost model, named owner, open risks.
  7. Go or no-go recommendation with reasons, including what would change it.

What should the exit criteria be?

Write go and no-go rules before the build: thresholds on quality, a unit-cost ceiling, confirmed production data access, a passed or scheduled security review, and a named production owner with a budget. Set the decision meeting date in advance.

CriterionEvidence requiredProceed ifStop or re-scope if
QualityHeld-out test resultsThresholds are metThresholds are missed and no credible fix fits the budget
DataData audit and access testThe production access path is approved and coverage is adequateData is missing, unusable, or not permitted
Unit costCost per query times forecast volumeWithin the budget ceilingOver the ceiling even with a cheaper model or caching
RiskTest results and review statusNo open critical findings, or a dated remediation planAn unresolved critical finding
OwnershipNamed owner and funded budgetBoth existNo owner or no funding
Adoption signalFeedback from target users on real tasksUsers complete tasks and would use itUsers work around it

A clear stop is a successful proof of concept: it saved the production budget.

Why do GenAI pilots stall before production, and how do I avoid it?

Several public sources point to a gap between proof of concept and production, though most are predictions or preliminary research. The causes they cite are mostly data, risk, cost, and ownership rather than model capability. Each figure below has a different scope, so none is a benchmark for your project.

  • Gartner, July 2024. Predicted that at least 30% of generative AI projects would be abandoned after proof of concept by the end of 2025, due to poor data quality, inadequate risk controls, escalating costs, or unclear business value. This is an analyst prediction, not a measured rate.
  • Gartner, February 2025. Predicted that through 2026 organizations will abandon 60% of AI projects unsupported by AI-ready data.
  • MIT NANDA, July 2025. Preliminary findings from interviews with 52 organizations, 153 survey responses, and a review of more than 300 public initiatives. The executive summary says 95% of organizations are getting zero return from GenAI. For enterprise-grade, custom or vendor-sold tools, it says 60% of organizations evaluated them, 20% reached the pilot stage, and 5% reached production. The report calls its figures directional, based on interviews rather than company reporting, with success defined differently across organizations. In the same sample, external partnerships with customized tools reached deployment about 67% of the time against about 33% for internally built tools, a self-reported gap the authors say does not prove causation.
  • RAND, August 2024. Interviews with 65 experienced data scientists and engineers identified five leading root causes: stakeholders misunderstanding the problem, lack of data, focus on technology over the problem, inadequate infrastructure, and applying AI to problems that are too hard. RAND notes that by some estimates more than 80 percent of AI projects fail, a figure it cites rather than measures.

What helps, drawn from those causes:

  1. Pick a use case with a business owner, a baseline, and a budget line.
  2. Start data access, security review, and legal review in week one.
  3. Test on production-like data, not a curated sample.
  4. Model cost per query at forecast volume before the readout.
  5. Write stop criteria and a decision date into the statement of work.
  6. Negotiate the production option (rate card, team continuity) in the proof-of-concept contract, without obligation.
  7. Keep the evaluation harness and test set in your repository so you can re-run them when models, prompts, or data change.
  8. Plan the handoff and training so your team can debug and extend the system.

How was this guide researched?

DCF Research is an independent directory that tracks 50 data and AI consulting firms using public evidence, checked on 2026-10-23. Firms cannot pay to change their score or rank. We have not audited client projects, interviewed clients, or tested any firm's work. Statistics come from the third-party sources below, and the cost figures marked as estimates show their assumptions.

Sources

More in AI Consulting

← Back to Insights