How does an organization know that its investment in artificial intelligence is working? Every business owner, public-sector director and institutional leader should be able to answer that question before approving the first investment. Yet many organizations still cannot.
The usual answers refer to user numbers, training hours, active licences, frequency of use or satisfaction with the tool. Sometimes even the opinion of the vendor that implemented the solution is accepted as proof of success.
None of those measures demonstrates effectiveness. They measure activity, adoption or perception.
Effectiveness begins to be demonstrated when we can objectively compare how a process worked before AI, what result we expected and what actually happened afterward. A serious implementation therefore begins with three decisions made before the technology is deployed.
1. Define success before starting
Before implementing anything, the expected result for each affected process must be recorded. It is not enough to say that AI should “increase productivity,” “reduce costs” or “improve efficiency.” Those expectations must become observable variables.
Depending on the case, the organization may need to measure cycle time, cost per transaction, throughput, error rate, rework, output quality, exceptions requiring human intervention, hours released, total technology cost and any new incidents or risks.
If a process currently takes six hours, costs 300 dollars and has a 4% error rate, there is a baseline. If after implementation it takes two hours, costs 160 dollars and maintains or improves quality, there is a measurable result. Only then can the discussion move to return.
This definition should not be left to one party. The vendor understands the solution’s capabilities and limits. Operations understands the real process, including exceptions and bottlenecks. Leadership knows what the improvement is worth and what investment and organizational change it is prepared to sustain. None of the three has the complete view alone.
2. Prevent the success criterion from changing after the result is known
An AI implementation can change tasks, redistribute responsibilities or remove manual work. Those involved may reasonably feel that the result affects their position.
The vendor wants to prove that the solution works. The executive who approved the investment may want to prove that the decision was sound. The operational team may fear changes to roles, structure or employment. Involving several parties is therefore not enough: a mechanism must prevent any of them from redefining success retrospectively.
The baseline, indicators, calculation method, comparison period and acceptance thresholds should be documented before the test begins. When project size or relevance justifies it, validation should include a control function independent from those who designed or executed the implementation: internal audit, a PMO, a transformation office, an investment committee or a designated third party.
This is not about adding bureaucracy. It is about making sure no one can move the goalposts after seeing the score.
3. Pilot before scaling
The third decision is to test the solution in a bounded environment before committing the full operation. An institution that applies AI to an entire core process without first testing a small, controllable and reversible portion is not yet carrying out an industrial implementation. It is making a bet.
A well-designed pilot answers questions more important than “does the tool work?” Does it work in our real process, with our data and exceptions? Does it preserve quality? How much human work does it still require? What does it cost to operate? What errors does it introduce? Can it scale without cost, supervision or risk rising at the same rate as production?
Baseline
Time, cost, quality and risk before the change.
Pilot
Bounded, reversible and representative use case.
Measurement
Same method, period and agreed thresholds.
Decision
Scale, correct or stop.
The subsequent decision should be explicit: scale, correct or stop. Not every pilot should become a full implementation. Cancelling an initiative that fails to demonstrate enough value may be exactly the right outcome of a sound evaluation.
Adoption is not effectiveness, and effectiveness is not return
An organization can achieve high AI adoption without improving any relevant outcome. It can increase productivity while reducing quality. It can improve productivity and quality at a technology cost that destroys the economic return.
Adoption ≠ productivity ≠ effectiveness ≠ return.
The four may be related, but they do not mean the same thing. The business objective should not be to make many people use AI. It should be to make selected processes deliver better results at an acceptable level of cost and risk.
The important decision occurs before the technology is purchased
The difference between an organization that learns to implement AI methodically and one that merely accumulates tools rarely lies only in the quality of the technology. It lies in whether someone required five questions to be answered before the investment:
- Which process do we want to change?
- How does it work today?
- What result do we want?
- How will we measure it?
- Who can objectively confirm that it has been achieved?
An organization that can answer them before starting has a project that can be measured, defended and eventually scaled. One that can respond only with expectations, licences, training hours or participant testimonials does not yet have a proven investment. It has acquired technology that still needs to demonstrate that it creates value.
Methodological note
The figures in the example are illustrative. Each use case requires its own indicators, comparison period and acceptance thresholds to be selected before the pilot. The method is intended to make the before-and-after comparison valid, not to impose one metric on different processes.
© 2026 Javier F. Pérez Bernabé. All rights reserved. Published by Quórum 12.
ONLINE SUBSCRIPTION
Receive every new analysis and invitation to open events.
This is a separate public category: it does not confer membership, admission or access to the private portal.