Guide · 17 min read

How to evaluate an AI automation agency before you hire one: a scorecard

Most agency evaluations score the pitch. This scorecard scores the evidence: ten weighted criteria, anchored examples of weak and strong answers, and a process that ends with a paid discovery and a contract you can walk away from.

How should we evaluate AI automation agencies before hiring?

Score each agency on weighted criteria backed by evidence, run a paid discovery with the leader, then sign with agreed ownership and exit terms.

Evaluate an AI automation agency by scoring what it can prove, not what it says in the pitch. Use ten weighted criteria, ask for one specific piece of evidence for each, and let a short paid discovery decide between the last two firms.

This guide is the scorecard. Our buyer's checklist for choosing an AI automation agency covers first-call questions and red flags, and how an AI implementation partner delivers walks through the stages of a good engagement. Read those to understand the market. Use this page to make the decision on paper.

Why the weights look the way they do

The weights follow the reasons AI projects stall between a promising pilot and daily use. Those reasons are well documented, and very few of them are about the model.

RAND's researchers interviewed 65 experienced data scientists and engineers for The Root Causes of Failure for Artificial Intelligence Projects, published in August 2024. The most common root cause they heard was leadership misunderstanding the problem, or how to set the project up to succeed. That is why diagnosis carries the heaviest weight below.

Gartner predicted in July 2024 that at least 30% of generative AI projects would be abandoned after proof of concept by the end of 2025, citing poor data quality, inadequate risk controls, escalating costs or unclear business value. In February 2025 Gartner went further on data alone, predicting that through 2026 organisations will abandon 60% of AI projects unsupported by AI-ready data.

The newest warning matters most to anyone hiring outside help. On 29 September 2026 Gartner predicted that by 2028, 70% of enterprises will abandon agentic AI built by vendor forward-deployed engineering, because costs climb and the client cannot evolve the system on its own. That is a buying failure, and a scorecard is where you catch it.

There is a counterweight. In The GenAI Divide: State of AI in Business 2025, published in August 2025, a study by MIT's NANDA initiative reported that about 95% of enterprise generative AI pilots produced no measurable return. In the same research, as Fortune reported, buying from specialised vendors and building partnerships succeeded about 67% of the time, and internal builds about a third as often. The study's method has been debated, so read it as a direction: outside help can raise the odds, if you choose it well.

The scorecard: ten weighted criteria and the evidence to ask for

Ten criteria worth 100 points. Diagnosis, delivery evidence, integration depth and ownership carry most weight, and each names the evidence that earns full marks.

Copy this table into a spreadsheet. Each member of your panel scores every criterion from 1 to 5. Multiply each score by its weight, divide by 5 and add the rows. A perfect agency scores 100 points.

Two design choices are deliberate. Price is not a criterion. Compare price only between firms that clear the quality bar, otherwise a cheap, weak proposal can outscore an expensive, strong one on arithmetic alone.

There is also no criterion called "AI expertise". Every firm will claim it. Criteria 1 to 3 test it the only way that counts: against your work, your systems and your data.

Ask for evidence to be shown, not sent. A slide that says "integrated with your CRM" is a claim. A run log with timestamps, retries and an exception queue, viewed live, is evidence. Score the difference.

What a 1, 3 and 5 look like on each criterion

Anchors stop scorers guessing. A 1 is an unbacked claim, a 3 is a credible description, and a 5 is something shown to you.

Without anchors, three people on a panel score the same answer 2, 3 and 4, and the total becomes a measure of mood. These anchors describe what you should actually see or hear at each level.

Score independently first, then compare. Where two people differ by two or more points on one criterion, that gap is the most useful thing in the whole exercise. It usually means one of you saw evidence the other missed, or one of you took a claim on trust.

Write one line of evidence next to every score. "Showed run log, three retries handled, exception queue owned by a named person" survives a challenge from your finance director. "Seemed strong" does not.

Where can we find an AI implementation partner for our company?

Longlist six to eight firms from referrals, vendor partner directories and published work, then cut to three using written questions before any sales call.

Build a longlist of six to eight firms from four places, and know what each source can and cannot tell you before you trust it.

Searches for the top rated AI implementation firms for mid-market companies in the UK mostly return directories and lists. Use them to find names, not to rank them. A rating tells you how pleased past clients felt. It does not tell you whether the firm can connect your CRM to your finance system without creating duplicates. The scorecard does.

You are not late to this. The Office for National Statistics, in its Business Insights and Conditions Survey published on 8 October 2026, found that around 30% of all UK businesses used at least one AI technology in September 2026, rising to 51% of businesses with 250 or more employees. In our experience, most mid-market teams are choosing their first implementation partner, not their third.

Cut the longlist with five written questions

Before any sales call, send each firm the same half-page brief and five questions, and ask for written answers within a week.

1. Describe a workflow you built that is still in daily use after six months. Which systems did it touch?
2. What would you need from our team in the first two weeks?
3. What do you do when our data is not good enough to build on?
4. Who owns the code, the workflows and the tool accounts when the engagement ends?
5. What would make you turn this work down?

The last question does the most work. A firm that can name the conditions under which it would say no has thought about fit. A firm that would take anything will also build anything, including the wrong thing.

How to run the evaluation, from shortlist to paid discovery

Pick one workflow, form a three-person panel, score the shortlist independently, then pay the leading firm for a fixed-fee discovery before committing to a build.

Start with one workflow and a baseline

Before you call anyone, write half a page: the workflow, who does it, how often, how long it takes today and which systems it touches. Without that page, every agency evaluates you instead of the other way round. Our guide to what to automate first with AI helps you pick it.

Form a panel of three

You need the budget holder, the person who runs the process day to day, and someone from IT or security. In discussion, give the most weight to the process owner's scores on diagnosis and adoption. They are the one who will live with the result.

Run every shortlist meeting the same way

Same brief, same length, same order of questions. Ask each firm to show one live workflow rather than describe it. Score within 24 hours, before memories blur into a general impression.

Use a paid discovery as the trial

A free proposal is written before anyone has looked at your data or systems, which is exactly the fixed-price-before-discovery red flag in our checklist. A short paid discovery is the cheapest way to see how a firm really works.

In our own work, a useful discovery has a fixed fee, a fixed length of two to four weeks and written outputs:

• A map of the workflow as it runs today, including the exceptions.
• A sample check of the data the workflow will read and write.
• A design for the first workflow, with permissions and approval points.
• A baseline and the measure you will review against.
• An estimate of build cost and monthly running costs.
• A list of risks, including anything that should be fixed before a build.

Agree before you start that the outputs are yours and usable by another firm. If a firm will not agree to that, score it 1 on ownership and stop there.

Watch three things during discovery. Who turns up: the people you met in the pitch, or someone new? Whether they speak to the people who do the work, or only to the sponsor. And whether they find something you did not already know. In our experience the last one is the best predictor of the build.

Which questions should you ask on reference calls?

Ask referees about a project older than six months: what broke after launch, who maintains it now, and whether they would hire again.

Ask each shortlisted firm for two referees: one whose workflow has been live for more than six months, and one whose scope changed during the project. Then ask to speak to the process owner, not only the sponsor who signed the contract.

1. What was the workflow, and is it still running today?
2. How long was it from signature to the workflow being used every day?
3. What broke in the first three months, and how quickly was it fixed?
4. Who maintains it now: your team, the firm, or nobody?
5. Did the people doing the work actually use it? What changed to make that happen?
6. Were running costs close to the estimate? Which line moved?
7. Did you receive the code, accounts and documentation you were promised?
8. What would you change in the contract if you signed it again?
9. Would you hire them for a different problem?

Listen for specifics. A referee who names the error that broke in week three and how it was fixed was close to the delivery. A referee who can only say the team was lovely to work with probably was not.

After the calls, revisit criteria 2, 6 and 7. References are the one point in the process where you hear about delivery, adoption and ownership from someone with no reason to sell to you.

Which contract and ownership terms should you agree before work starts?

Agree who owns the code and workflow logic, whose name the accounts sit in, what documentation and handover you receive, and how either side exits.

Gartner's September 2026 prediction came with advice that applies to any outside build. It recommends contractually securing deliverables, knowledge transfer, IP rights and transition or exit duties, and measuring success by your ability to manage the system yourself rather than by the build being finished.

For a structured cross-check, the UK government's Guidelines for AI procurement, first published in June 2020 for public-sector buyers, covers preparation, evaluation, award and contract management. Most of it transfers to a private-sector purchase of this size.

Agree these terms before the build begins. Once a workflow is live and the team depends on it, every one of them is harder to negotiate.

How to read the totals and make the decision

Above 80 is strong, 65 to 79 workable with conditions, below 65 a no. A 1 on diagnosis, governance or ownership vetoes any total.

These bands are our own working thresholds, not an industry standard. Adjust them if your risk appetite is different, but set them before you score, not after.

A worked example

Three illustrative firms, invented for this guide, show how the totals and the veto interact. Scores are the panel's agreed 1 to 5 for each criterion.

Firm A gave the most polished pitch and the best demo, and would have won a vote taken in the room. On the scorecard it led with a platform before understanding the work, and its ownership terms were vague. Firm C was strong almost everywhere, but wanted the workflows and tool accounts to stay in its own name. That single 1 ends the conversation.

Firm B goes to paid discovery. Its weaker delivery evidence becomes a condition: one more referee before the build contract is signed.

Close calls and the second scoring

If two firms finish within five points of each other, decide on diagnosis and ownership first, then on price. After the paid discovery, rescore criteria 1, 3, 4 and 9 with what you now know. If the total falls by more than ten points, do not proceed to the build. You have paid for the discovery, and the outputs are yours to take elsewhere.

Where The Sparked Group scores itself low, and when we are the wrong fit

We score low where buyers want published prices before a first call, an off-the-shelf chatbot, or a large programme before one workflow is proven.

We design, build and support AI workflows and internal tools for B2B teams, around the systems they already use. Score us like anyone else. Here is where we expect to lose points.

• Running cost transparency before a first conversation. We quote after an assessment and do not publish standard prices, so a process that needs a price before any call will score us low at that stage. Ownership, hosting, third-party software and AI usage costs are agreed before any build.
• Fit, if you want a generic AI workshop. We do not run them, and a training provider will serve you better.
• Fit, if you want an off-the-shelf chatbot with no systems integration. A product will cost less and launch faster.
• Fit, if you want a large transformation programme before a useful first workflow exists. We start with one workflow and let it earn the next.
• Adoption, if nobody on your side can own the workflow. Planning, implementation and team adoption sit in the same engagement, and adoption needs an owner inside your team. Without one, we would score our own likely outcome low and tell you so.

Where we expect to score well, test it rather than take our word for it. Ask us to show the evidence for criteria 1, 6 and 7. Our published case studies carry dated, measured figures, such as the energy technology scale-up where a CRM figure of 153 marketing leads was traced to 15 genuine hand-raises.

If you want to start with your own workflow rather than our pitch, find your starting point and bring that half page to the first call.

Frequently asked questions

How should we evaluate AI automation agencies before hiring?

Score each agency on weighted criteria such as problem diagnosis, delivery evidence, integration depth, governance, adoption and ownership terms, asking for evidence that is shown rather than described. Then run a short fixed-fee paid discovery with the leading firm, take reference calls on projects live for more than six months, and agree ownership and exit terms in writing before any build.

What criteria matter most when choosing an AI automation agency?

Problem diagnosis matters most, because misunderstanding the problem is the most common root cause of AI project failure in RAND's August 2024 research. Next come evidence of workflows running in production, depth of integration with your systems, and written ownership and exit terms that keep the code, accounts and documentation in your hands.

Should we pay for a discovery phase before hiring an AI agency?

Yes, for anything beyond a simple workflow. A free proposal is written before anyone has seen your data or systems. A fixed-fee discovery of two to four weeks, with written outputs you own, shows how the firm really works and gives you a plan another firm could use if you decide not to continue.

What questions should we ask an AI agency's references?

Ask about a project live for more than six months: whether it still runs, how long it took to reach daily use, what broke in the first three months, who maintains it now, whether running costs matched the estimate, whether the code and accounts were handed over, and whether they would hire the firm again for a different problem.

Where can we find a top rated AI implementation firm for a mid-market UK company?

Build a longlist from peer referrals, your CRM or platform vendor's partner directory, firms that publish their method and dated case studies, and review sites. Use ratings to find names rather than to rank them, then cut to three firms with written questions and compare them on a weighted, evidence-based scorecard.

What contract terms protect us when hiring an AI automation agency?

Agree that code and workflow logic are assigned to you, third-party accounts sit in your organisation's name, documentation and knowledge transfer are included, every running cost has a named payer and spend limit, support and change rules are clear, and exit duties cover handover and removal of access.