Evaluating AI Tools Instead of Believing: How to Measure the Benefit

Why vendor numbers and gut feeling are not enough

It can be said plainly at this point: a glossy demo and a high benchmark score, meaning a good result in a standardized performance test, are a sales argument, not proof of effectiveness. Whoever takes them as proof decides based on someone else’s number, not their own. There are three documented reasons for this: benchmark scores are frequently “contaminated”, the vendor evaluates with a bias, and users misjudge their own gains.

“Contamination” means a model has already seen the test tasks, or very similar ones, in its training; the good score is then memorized and no evidence of ability.

On a widespread multiple-choice benchmark (MMLU), commercial models were able to correctly guess the missing answer option in a good half of the cases (ChatGPT 52 percent, GPT-4 57 percent): an indication that the test questions were part of the training material [6]. Even paraphrased or translated test tasks bypass the usual cleanup via text matching and still drive scores up; a 13-billion-parameter model reached a level on par with GPT-4 on a manipulated benchmark this way [8].

The evaluation itself is also a source of error. If you let an AI judge its own or similar work, it systematically favors its own output; this effect is documented as self-preference bias [7]. Transferred to a tool decision, this means: whoever evaluates success must not be a party at the same time.

Added to this is users’ self-assessment. In a controlled study, experienced open-source developers worked on average 19 percent slower with an AI tool, but themselves assumed they had been around 20 percent faster [22]. The perceived gain and the measured effect ran in opposite directions.

Two widespread misconceptions and what the data shows

Step 1: Define the success metric before you test

Before the first test, define what you will measure success by. A success metric determined only after the introduction provides a weak basis for comparison; established practice is to define a compact set of metrics, including data collection and a responsible person, before deployment [1].

The metric must be formulated measurably. “Processing time from 20 to under 2 minutes” is verifiable, “better service” is not [2]. Only the measurable formulation makes a tool evaluable at all.

It makes sense to connect three measurement levels: the quality of the output, operational stability in ongoing use, and the effect on the business [3]. Whoever only checks whether something works technically does not see whether it has business value.

To define before the test, then:

• The concrete, measurable success metric.
• The data source it is collected from.
• The person responsible for the measurement.
• The three measurement levels: output quality, operations and business impact.

Step 2: Compare against the tool-free state

Compare the tool on the same task with the same inputs against the state without the tool. This state is the baseline, the yardstick against which a change can be read [4][5]. The tool is thereby not tested against its own marketing, but against what happens without it.

A deliberately tool-free comparison group, the holdout group, shows the true overall effect. It is the part that continues to work without the tool for the entire test duration. Whoever only adds up individual improvements overestimates the benefit; a consistently tool-free group records what is actually different at the bottom line [4][5].

Step 3: Evaluate output quality objectively and blind

Evaluate results according to yes/no criteria defined in advance, and blind. “Blind” means the evaluator does not know which tool delivered an answer. A criteria- and rule-based evaluation with a yes/no verdict per criterion is common practice; it must be done blind and by a neutral party, because self-preference bias occurs here too [9].

The method of evaluation measurably influences the result. A structured, multi-dimensional approach lowered self-preference bias by an average of 31.5 percent in one study, without retraining the model [10]. Clear, separate criteria therefore beat a sweeping overall verdict.

For a reliable quality check, then:

• Define the review criteria before the evaluation.
• Judge each criterion binary, yes or no.
• Run the evaluation blind.
• Have a neutral party evaluate, not the vendor or the tool itself.

Step 4: Calculate the true costs, not just the price

Calculate total costs over the lifecycle, not just the license price. The license price is rarely the largest cost block; the main share is usually made up of integration, onboarding, maintenance and ongoing usage [11]. Between us: exactly these blocks appear on no quote a vendor ever sends, and are therefore overlooked first when calculating, until they suddenly dominate the bill in the second half of the project. Whoever lists them from the start does not calculate more pessimistically, but simply more completely.

As an order of magnitude, one industry estimate puts integration at 20 to 50 percent of the AI budget and total costs over three years at one and a half to two times the initial investment [12]. These values are estimates, not a measurement of the individual case. That costs run out of bounds without clean tracking is shown by a survey according to which 56 percent of companies missed their AI cost forecast by 11 to 25 percent and 24 percent by more than 50 percent [12].

For illustration, a calculation example with round numbers: if a company initially invests €10,000 for the introduction, total costs according to this estimate come to around €15,000 to €20,000 [12].

To capture, then, at minimum:

• License and usage fees.
• Integration into existing systems.
• Onboarding and training.
• Maintenance and ongoing operations.
• Cost per use in continuous operation.
• Speed (see below).

Speed is a cost factor

Speed is not a convenience, but part of the costs. Throughput, response time and compute costs cannot be optimized at the same time: higher speed at high volume is bought either with longer response times or with higher compute costs [13]. In chat and support scenarios, response times above roughly two to three seconds lead to abandonment [14]. Measurable here are the time to first output and the throughput per unit of time [15].

Step 5: Run a pilot first, then roll out fully

Introduce an AI tool first as a limited pilot and measure there against the success metric defined in advance. A pilot is a small, contained feasibility test before the full introduction.

The abandonment rates argue for this approach. Two Gartner forecasts show the magnitude. Both are forecasts, not a proven actual value:

• Around 30 percent of generative AI projects by the end of 2025: In 2024, Gartner forecast that around 30 percent of generative AI projects would be abandoned after the proof of concept by the end of 2025; this time horizon has now been reached, and no reliable actual evaluation is publicly available.
• Over 40 percent of agentic projects by the end of 2027: for so-called agentic projects, Gartner expects an abandonment rate of over 40 percent by the end of 2027 [16].

The cause usually does not lie in the technology. An MIT report concludes that around 95 percent of enterprise pilots with generative AI deliver no measurable return, primarily because of integration, governance and organizational change, not because of model quality [17]. Consistently, it shows: companies that scale AI successfully put about 70 percent of their effort into people and processes and only around 10 percent into the algorithms [18]. The evaluation of a tool must therefore include introduction and adoption, not just pure tool performance.

What research shows about the benefit of AI tools

Controlled studies document real productivity gains from AI tools, but they vary widely:

• Customer support: The number of cases resolved per hour rose by 14 percent on average and by 34 percent for entry-level staff (controlled field study, N=5,179) [19].
• Software development: Among software developers, the number of completed tasks increased by 26 percent in a controlled study (N=4,867) [20].
• Programming task: In a field experiment, a defined programming task was completed 55.8 percent faster (N=95) [21].

These values are documented magnitudes, not a guarantee for your own case. Across several studies it shows that the gains are highest for beginners and low-skilled activities and can tip towards zero or negative for experienced professionals [19][21][22]. How a tool works in your own operation can therefore only be determined by your own measurement in the concrete context.

Making your own benefit measurable

And therein lies the actually good news: as soon as the benefit of a tool becomes measurable, a question of belief turns into a question of numbers, and that one can be answered. “We have a good feeling” becomes a value you can defend, repeat and improve over time.

Whoever wants to judge the benefit of an AI tool reliably needs metrics defined in advance and a clean measurement setup. Exactly this tracking and analytics foundation is what allows marketing decisions, too, to be aligned with numbers instead of promises. Such a setup defines which metric is collected, where the data comes from and how the comparison against the initial state is made.

Dometrics aligns tracking, analytics and controlled measurement to make the contribution of individual measures visible. Whoever wants to check whether their own data foundation supports a reliable tool and marketing evaluation can have it reviewed in an analysis call.

More on the measurement and tracking foundation: Tracking & Data Analytics

Related topics: GEO & AI Search · Paid Advertising

Look up terms: Metric (KPI) · Incrementality (holdout measurement) · all terms in the Dometrics glossary

Frequently asked questions

How do I test an AI tool objectively before introducing it?

First define a measurable success metric, compare the tool on identical tasks and inputs against the current state, and evaluate the results blind according to criteria defined in advance. That way, the verdict hangs on measured values instead of first impressions.

How do I know whether an AI tool really delivers?

By the difference between a tool-free comparison group and the group with the tool, measured against the success metric defined in advance. Vendor numbers and subjective feeling are not enough, since measured and perceived effect can diverge.

Why should vendor benchmarks be treated with caution?

Because benchmark tasks were often already contained in the training data (contamination), and because an AI that evaluates its own work favors itself. A test with your own, fresh tasks from your own operation is more reliable.

What does an AI tool really cost?

More than the license price: integration, operations and ongoing usage usually make up the larger part; integration alone an estimated 20 to 50 percent of the budget. Speed is a cost factor too.

Is a pilot project worth it before a full rollout?

A pilot limits the risk, since a large share of AI projects is abandoned after the proof of concept and many pilots show no measurable return. The bottleneck usually lies in integration and adoption, not in model quality.

Sources

• [1] Google Cloud – "KPIs for gen AI: measuring AI success" (2024)
• [2] TechTarget – "How businesses can measure AI success with KPIs" (2024)
• [3] StackAI – "Measuring enterprise AI success" (2025)
• [4] Statsig – "Holdout groups in A/B testing"
• [5] Eppo – "Holdouts: measuring experiment impact accurately"
• [6] Deng, Zhao, Tang, Gerstein, Cohan – "Investigating Data Contamination in Modern Benchmarks for Large Language Models", NAACL 2024
• [7] "Self-Preference Bias in LLM-as-a-Judge", arXiv:2410.21819 (2024)
• [8] Yang, Chiang, Zheng, Gonzalez, Stoica – "Rethinking Benchmark and Contamination for Language Models with Rephrased Samples", arXiv:2311.04850 (2023)
• [9] "Self-Preference Bias in Rubric-Based Evaluation", arXiv:2604.06996 (2026)
• [10] Reducing self-preference bias through structured prompting, arXiv:2604.22891 (2026)
• [11] Glean – "Total cost of ownership of AI solutions"
• [12] Xenoss – "Total cost of ownership for enterprise AI" (2025)
• [13] DigitalOcean – "LLM inference tradeoffs" (2025)
• [14] Maxim AI – "Reduce LLM cost and latency" (2026)
• [15] Anyscale – "LLM serving benchmarking metrics"
• [16] Gartner – Forecasts on abandonment rates of generative and agentic AI projects (2024/2025)
• [17] MIT – "State of AI in Business 2025"
• [18] Astrafy – "Scaling AI from pilot purgatory" (2025)
• [19] Brynjolfsson, Li, Raymond – "Generative AI at Work", NBER w31161 (2023), publ. Quarterly Journal of Economics (2025)
• [20] Field study on developer productivity, Management Science (2025)
• [21] Peng et al. – GitHub Copilot field experiment, arXiv:2302.06590 (2023)
• [22] METR – "Measuring the impact of early-2025 AI on experienced open-source developer productivity", arXiv:2507.09089 (2025)